1. Databricks for machine learning

How to use this quest: work through the chapters in order, run the code in a disposable development workspace when available, and explain why each choice is appropriate rather than memorising product names. Treat the quizzes as retrieval practice; revisit every explanation you cannot confidently teach back.

Exam-ready workflow: for a scenario, identify the data source and governance boundary first; decide whether the task is preparation, training, evaluation or inference; choose the relevant Databricks capability; then verify reproducibility, permissions, monitoring and cost. Keep the exam objectives and product terminology in the current official study guide as the source of truth because platform features and certification blueprints can change.

Databricks Machine Learning runs on the lakehouse. A lakehouse keeps data in open table formats on cloud object storage and adds warehouse-style management on top, so the same governed tables serve data engineering, analytics and machine learning. For ML this means training data does not have to be exported to a separate system. You read it where it already lives, under the same permissions, and you track the work on the same platform.

Layer	What it provides	Why ML cares
Data platform	Delta tables, files in volumes, structured and unstructured data	One governed, versioned copy of the training data
Governance	Unity Catalog: access control, lineage, discovery, auditing, sharing	Data, features, models and notebooks sit under one permission model
Data processing	Spark, SQL and Lakeflow for ingestion, ETL and streaming	Feature pipelines and streaming inputs
ML development	Notebooks with the ML runtime, MLflow, AutoML, Genie Code	Build, track, tune and compare models
Serving and monitoring	Model Serving, Feature Serving, Vector Search, monitoring	Batch and real-time use of models in production

Databricks Runtime for ML

A cluster runtime is the image that provides Spark, the Python environment and the libraries on a cluster. The Databricks Runtime for ML (often written DBR ML) is the variant built for machine learning. Its advantages:

• Common ML libraries are pre-installed at versions that are tested to work together, so you avoid most dependency conflicts.
• MLflow is pre-installed, so experiment tracking and model logging work without setup.
• Distributed ML and GPU-enabled variants are available for workloads that need them.
• You start modelling faster, because environment setup is not your job.

Serverless compute removes cluster sizing decisions altogether. The notebook workflow stays the same either way.

Collaborative notebooks

• Multi-language. One notebook can mix Python, SQL, Scala and R. A language magic at the top of a cell switches the language for that cell, for example %sql or %md.
• Shell and file commands. %sh runs shell commands on the driver, %fs runs file system commands, and %md renders markdown.
• Version history. Revisions are tracked automatically. Git folders connect a workspace folder to a Git repository for proper version control.
• Real-time co-editing. Several people can edit and comment on the same notebook.
• Widgets. Input widgets parameterise a notebook, for example a date range or a model name, so the same notebook can run with different inputs.
• Built-in charts and data profiles. Results can be charted with a few clicks. A data profile summarises a table's columns, distributions and missing values with one button.

Managing libraries and code

• %pip install package installs a Python library for the notebook session only. Other users on the same cluster are not affected.
• Cluster-level libraries affect everyone who attaches to that cluster, so they should be kept for shared dependencies.
• With Git folders, local Python modules in the repository can be imported, and arbitrary files can sit beside the notebook.

Genie Code in the notebook

Genie Code is the AI assistant built into the Databricks workspace. In a notebook it suggests code, explains errors and helps with debugging. It is covered in more depth in the Agentic Machine Learning realm.

2. Delta Lake and Unity Catalog

Two foundations sit under every ML workflow on Databricks: Delta Lake stores the data reliably, and Unity Catalog names, governs and shares it.

Delta Lake

• Delta Lake is an open-source storage layer. It is the default table format on Databricks, so a table created with a plain CREATE TABLE is a Delta table.
• ACID transactions. Atomicity (a write fully succeeds or fully fails), consistency (tables stay valid), isolation (concurrent readers and writers do not see partial work) and durability (committed writes survive).
• Scalable metadata handling for very large tables with many files.
• Unified batch and streaming. The same table can be written by a batch job and read by a stream, or the reverse.
• Automatic schema validation. Writes that do not match the table schema are rejected unless schema evolution is enabled.
• Schema evolution adds columns without rewriting the existing data files.
• Upserts with MERGE INTO update matching rows and insert new ones in one statement.
• Change Data Feed records row-level inserts, updates and deletes so downstream consumers can process only what changed.
• Time travel reads an earlier version of a table by version number or by timestamp.
• Performance from data skipping (statistics let the engine skip files) and liquid clustering (adaptive data layout on chosen columns).
• Supported from Python, Scala, SQL and R.

Task	Python	SQL
Read a table	df = spark.read.table("main.ml.churn")	SELECT * FROM main.ml.churn
Write a new table	df.write.saveAsTable("main.ml.churn")	CREATE TABLE main.ml.churn AS SELECT ...
Overwrite a table	df.write.mode("overwrite").saveAsTable("main.ml.churn")	INSERT OVERWRITE main.ml.churn SELECT ...
Read an old version	spark.read.option("versionAsOf", 3).table("main.ml.churn")	SELECT * FROM main.ml.churn VERSION AS OF 3
Read by timestamp	spark.read.option("timestampAsOf", "2026-01-01").table(...)	SELECT * FROM main.ml.churn TIMESTAMP AS OF '2026-01-01'
List versions	spark.sql("DESCRIBE HISTORY main.ml.churn")	DESCRIBE HISTORY main.ml.churn
Roll back	spark.sql("RESTORE TABLE main.ml.churn TO VERSION AS OF 3")	RESTORE TABLE main.ml.churn TO VERSION AS OF 3

Unity Catalog

Unity Catalog is the governance layer for the whole platform. It provides:

• A three-level namespace. Every object is addressed as catalog.schema.object.
• Objects. Tables, views, volumes (governed storage for files such as CSV, images or model artifacts), functions and registered models all live in the namespace.
• A single permission model. Grants on catalogs, schemas and tables control who can read, write or manage each object.
• Discovery. Search finds tables and models by name, description and tags.
• Lineage. Automatic tracking shows which tables, notebooks and models produced or used a given asset.
• Auditing and monitoring. Access is logged, so you can answer who used which data.
• Sharing. Data and models can be shared with other organisations through open sharing.

-- Grant read access to a team, at the schema level
GRANT USE CATALOG ON CATALOG main TO `data-scientists`;
GRANT USE SCHEMA ON SCHEMA main.ml TO `data-scientists`;
GRANT SELECT ON SCHEMA main.ml TO `data-scientists`;

Files that are not tables go into volumes, referenced by a path such as /Volumes/main/ml/raw/customers.csv. A CSV in a volume can be read with spark.read.csv(path, header=True, inferSchema=True) and then saved as a managed Delta table.

3. Exploring and cleaning data

Before any model is trained, the data needs to be understood. Exploratory data analysis (EDA) answers four questions: what is in each column, how is the target distributed, where are values missing or invalid, and how do the features relate to the target.

Summary statistics

• Spark DataFrame.summary() returns count, mean, standard deviation, min, the 25th, 50th and 75th percentiles, and max for numeric columns. It can also compute other statistics. Pass the statistic names to choose them, for example df.summary("count", "min", "max").
• df.describe() returns a smaller set: count, mean, stddev, min and max. It does not return percentiles.
• dbutils.data.summarize(df) produces an interactive profile in a Databricks notebook, with histograms and missing counts per column.
• Count missing values per column with F.count(F.when(F.col(c).isNull(), c)) or the equivalent isNull() filter. Null and NaN are different in Spark, so check both where floats are involved.

Cleaning types and invalid values

• Coerce types first. Cast strings that hold numbers or dates to the right type with df.withColumn("age", F.col("age").cast("double")). Non-numeric text becomes null after the cast, so count it before and after.
• Invalid values. Negative ages, impossible dates and sentinel values such as -999 are errors. Replace or remove them before imputation, otherwise the imputed value is computed from garbage.
• Outliers are values far from the rest of the distribution. They may be errors or real extremes, so check the domain before removing them.

Outlier rules

Method	Rule	Good when
IQR	Keep values between Q1 minus 1.5 times IQR and Q3 plus 1.5 times IQR, where IQR = Q3 minus Q1	Skewed data. Quartiles are not affected by the outliers themselves.
Standard deviation	Keep values within mean plus or minus k times the standard deviation, usually k = 3	Roughly symmetric, near-normal data

The IQR method uses approxQuantile in Spark to get the quartiles. The standard deviation method uses mean and stddev from the summary. Both are computed on the training data only in a real project.

Visualising features

• Continuous feature: histogram for the shape, box plot for spread and outliers, scatter plot against the target for the relationship.
• Categorical feature: bar chart of category counts, and a bar chart of the target rate per category.
• Target distribution: a bar chart of class counts reveals imbalance at a glance. A regression target is better shown as a histogram.
• The built-in notebook charts and display() cover these with a few clicks. Matplotlib and other libraries work too.

Comparing features

Comparison	Method	Notes
Two continuous features	Pearson correlation for a linear relationship; Spearman rank correlation for a monotonic one	Correlation near 0 does not rule out a nonlinear link
Two categorical features	Chi-square test of independence on a contingency table	Expected counts should not be tiny
Continuous feature against a categorical target	Compare group means or medians, box plots per class, ANOVA for several groups	Check the spread, not just the centre

from pyspark.sql import functions as F

# Summary statistics on selected columns
df.summary("count", "mean", "stddev", "min", "50%", "max").show()

# IQR bounds for one column
q1, q3 = df.approxQuantile("income", [0.25, 0.75], 0.01)
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
df_clean = df.filter((F.col("income") >= lower) & (F.col("income") <= upper))

4. Splitting data and avoiding leakage

The model must be judged on data it has never seen. Splitting the dataset into separate sets, and using each set for one purpose only, is what makes the evaluation honest.

The three roles

Set	Purpose	Influence on the model
Training	Fit the model's internal parameters	Directly sets the parameters, for example the coefficients or tree splits
Validation	Tune hyperparameters and compare candidate models	Indirect: choices made from this set shape the final model
Test	Estimate performance on unseen data once, at the end	None. It must not influence training or selection

The validation set is the holdout used to find the best configuration. The test set is the holdout kept for the final verdict. If the test set is used to pick a model, its score becomes optimistic and the estimate is no longer trustworthy.

Typical ratios

• Large datasets: 70 percent training, 15 percent validation, 15 percent test, or 60, 20, 20.
• Smaller datasets: a two-way split such as 80 percent training and 20 percent test, with cross-validation inside the training part instead of a fixed validation set (see the Tuning realm).
• Always fix a random_state so the split can be reproduced.

Sampling methods for the split

Method	How it works	Use when
Random sampling	Each row has an equal chance of selection	Rows are independent and the classes are balanced
Stratified sampling	Divide into subgroups (strata) and sample each in proportion	Classification with imbalanced classes. Keeps the minority class in both sets
Cluster sampling	Randomly pick whole clusters and keep every row inside them	Data arrives in natural groups, such as stores or devices
Time-based split	Train on earlier periods, test on later ones, with no shuffling	Time series and any data where the future must not leak into training

from sklearn.model_selection import train_test_split

# Random split, 80 percent train and 20 percent test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

# Stratified split keeps the class ratio the same in both sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

Data leakage

Leakage happens when information from outside the training data reaches the model during training, so its scores look better than they will be in production. Typical causes:

• Fitting a scaler, imputer or encoder on the full dataset before splitting. The statistics (means, medians, categories) then include test rows.
• Including a feature that is only known after the event being predicted, such as a refund flag in a churn model.
• Random shuffling of time-ordered data, so the model is trained on the future to predict the past.
• Duplicate or near-duplicate rows that appear on both sides of the split.

The rule that prevents most leakage: split first, then fit every preprocessing step on the training set only, and apply the fitted step to validation and test data.

Cross-validation as a split strategy

Cross-validation (CV) repeatedly partitions the training data so every row is used for validation once. It gives a more stable performance estimate than a single validation set, and it is the default when data is limited.

• K-fold: split into k folds, train on k minus 1, validate on the remaining fold, and repeat k times.
• Stratified k-fold: each fold keeps the class ratio. Use it for classification with imbalance.
• Shuffle split: repeated random train and validation splits, with no requirement that the folds partition the data.
• Nested cross-validation: an outer loop estimates performance and an inner loop tunes hyperparameters, so tuning does not leak into the estimate.

5. Handling missing data

Most real datasets have gaps. Missing values reduce model quality in several ways: they lower performance, they bias inferences when the gaps are not random, they under-represent some groups, and they add complexity to the code that must handle them. Good data preparation makes an explicit, justified choice for each column.

Options for handling missing data

Option	What happens	Trade-off
Drop rows	Remove any row with a missing value, or with missing values in chosen columns	Simple, but loses data and can bias the sample if the gaps are not random
Drop columns	Remove a feature with a high share of missing values	Appropriate when the feature is mostly empty and adds little
Treat as a category	Encode missing as its own category, such as "Unknown" or -1	Keeps the information that the value was missing. Good for categorical features
Mean, median or mode	Replace missing values with a statistic of the column	Fast and simple, but shrinks variance and ignores relationships between columns
K-nearest neighbours (KNN)	Replace a value with the average of the k most similar rows	Uses other features, but is slow on large data and sensitive to scale
Regression imputation	Predict the missing value from other columns with a regression model	Uses relationships between columns. Understates uncertainty if used alone
Multiple imputation	Create several plausible completed datasets and combine the results	Best statistical treatment of uncertainty, but the most effort

Choosing mean, median or mode

• Mean for continuous features that are roughly symmetric, with few outliers.
• Median for continuous features that are skewed or contain outliers, because the median is not pulled by extreme values.
• Mode for categorical features, and the only one of the three that makes sense for them. Using mode() on a continuous column is a poor choice, because most continuous values are unique and the mode is arbitrary.
• Examine the distribution first. The statistic that fits the distribution is the defensible one.

Factors that decide the method

• Nature of the data: continuous, ordinal or categorical. Each type has different sensible statistics.
• Amount missing: a few percent can be imputed simply. When a large share is missing, KNN or multiple imputation can be better, and dropping the feature may be the honest answer.
• Distribution: the median works better than the mean for skewed data.
• Mechanism of missingness: data missing completely at random (MCAR), missing at random given other observed columns (MAR), or missing because of the value itself (MNAR). Simple imputation can hide MNAR patterns.
• Effect on the analysis: imputation must improve the dataset without adding bias. Domain knowledge decides whether an imputed value is plausible.

Mark the imputed values

Keep a companion indicator column for every imputed feature. The indicator tells the model, and the analyst, which values were filled in. It supports model evaluation, data quality checks, transparency and error tracing.

ID	Name	Age	Age_imputed
1	Alice	25.0	0
2	Bob	30.0	0
3	Charlie	26.0	1

from pyspark.ml.feature import Imputer
from pyspark.sql import functions as F

# Flag first, so the indicator reflects the original nulls
df = df.withColumn("Age_imputed", F.col("Age").isNull().cast("int"))

# Median imputation, fitted on training data only
imputer = Imputer(inputCols=["Age"], outputCols=["Age_filled"], strategy="median")
imputer_model = imputer.fit(train_df)
train_df = imputer_model.transform(train_df)
test_df = imputer_model.transform(test_df)

6. Encoding categorical features

Most learning algorithms accept only numbers. Categorical features, such as city names or product types, must be converted to numeric form, and the method chosen affects accuracy, efficiency and interpretability.

One-hot encoding

One-hot encoding creates one binary column (0 or 1) for each distinct category. Each row has a 1 in the column for its own category and 0 elsewhere.

ID	City	New York	Paris	London
1	New York	1	0	0
2	Paris	0	1	0
3	London	0	0	1

Drawbacks: it creates sparse, high-dimensional data; it limits the split options of tree-based models, because each indicator can only be split as 0 or 1; it becomes inefficient with high-cardinality features (10,000 categories means 10,000 columns); it spreads the importance of one feature across many columns, obscuring it; and it increases computation.

One-hot encoding is appropriate for linear models, neural networks and k-nearest neighbours with few categories. It is generally poor for high-cardinality features.

from pyspark.ml.feature import StringIndexer, OneHotEncoder

indexer = StringIndexer(inputCol="city", outputCol="city_idx", handleInvalid="keep")
encoder = OneHotEncoder(inputCols=["city_idx"], outputCols=["city_ohe"])
# The encoder is a transformer; the indexer is an estimator that is fitted first.

Label and ordinal encoding

Label encoding maps each category to an integer. Ordinal encoding does the same when the categories have a meaningful order, such as Freshman, Sophomore, Junior, Senior, and assigns the integers in that order.

• Use ordinal encoding only when the order is real and the distances between levels are roughly meaningful.
• Label encoding of unordered categories creates a false order (for example, London is "greater" than Paris), which misleads distance-based and linear models.
• Label encoding is acceptable for tree-based models, which split on thresholds and can tolerate the arbitrary order.
• Spark's StringIndexer performs label encoding. Its labels are assigned by frequency by default, so the mapping is not alphabetical.

Target encoding

Target encoding replaces each category with the mean of the target for that category. In the example, New York has clicks [1, 0] and gets a value of 0.5, while London has [1] and gets 1.

• Strengths: one numeric column, no dimensional explosion, and it captures the signal directly.
• Overfitting risk: a category seen once with a positive target gets value 1, which is overconfident. Smoothing toward the global mean or cross-validated encoding reduces this.
• New categories: a category absent from training has no mean and needs a fallback such as the global mean.
• Leakage: the means must be computed on training data only.
• Small datasets: the estimates are noisy, so target encoding is risky.

High-cardinality features

A feature with many unique values causes the high-cardinality problem. The common fixes are:

• Group rare categories: any category below a frequency threshold becomes "Rare" or "Other".
• Top-N categories: keep the N most frequent categories and group the rest into "Other".
• Target or frequency encoding to turn each category into one number.
• Embeddings (below) for very large vocabularies.

Missing categories

Categorical gaps are handled by imputing the mode or by treating missing as a category of its own. Treating missing as a category keeps the signal that the value was absent, which is often predictive.

Embeddings

An embedding is a dense, low-dimensional vector that represents a category or item. Similar items sit close together in the vector space. Embeddings can be learned by a neural network in an embedding layer, or taken from a pre-trained model such as Word2Vec or FastText.

The process: convert each category to an integer index, pass the index through an embedding layer (a learned lookup table), and let the model assign each category a vector learned from the data.

Aspect	One-hot	Embeddings
Representation	Sparse binary, one column per category	Dense numeric vectors of a fixed size
Semantic relationships	None. Every category is equally far from every other	Captures similarity, so related categories are close
High cardinality	Inefficient	Scales well
Best models	Simple models such as decision trees, linear models	Neural networks and deep learning, and recommendation or search systems

7. Standardisation and log transforms

Once the data is clean and encoded, two transformations often help the model: standardisation, which puts features on a common scale, and log transformation, which compresses long right tails. Feature selection, removing columns that add noise or duplicate other columns, comes first.

Feature selection by dropping columns and rows

• Irrelevant or redundant features contribute little to the prediction or are highly correlated with another feature. Drop them to simplify the model.
• Columns or rows with heavy missingness can be dropped when imputation would add more noise than signal.
• Computational efficiency: a smaller dataset trains faster.

Data standardisation

Standardisation transforms data from different sources into one consistent format and scale. Features are typically scaled with a z-score:

z = (x - mean) / standard deviation

The result has mean 0 and standard deviation 1. Standardisation is fitted on training data only, then applied to validation and test data.

Why standardise

• Interpretability: coefficients of features on the same scale can be compared directly.
• Preventing model bias: a feature measured in large units does not dominate just because of its unit.
• Faster convergence: gradient-based optimisers reach the optimum more quickly.
• Effective regularisation: L1 and L2 penalties treat all coefficients fairly only when the features share a scale.
• Comparing coefficients across features with different units.

Which models care about scale

Sensitive to feature scale	Robust to feature scale
Linear models, including regularised linear and logistic regression	Decision trees
Support vector machines (SVM)	Random forests
K-nearest neighbours (distance-based)	Gradient-boosted and other tree ensembles
Neural networks
Principal component analysis (PCA)

Drawbacks of standardising

• The original units are lost, so predictions and coefficients are in standard deviations, not dollars or years.
• Interaction terms become harder to interpret after scaling.
• Result interpretation is harder for stakeholders.
• The z-score assumes roughly normal features. Heavy skew or outliers distort the mean and standard deviation.

Log transformation

A log transform replaces a positive, right-skewed variable with its logarithm, so that large values are compressed and the distribution is closer to symmetric. It is appropriate when:

• The variable is strictly positive and spans several orders of magnitude, such as incomes, prices or counts.
• The relationship with the target is multiplicative or proportional, not additive.
• The residuals of a linear model are heavily skewed or fan out with the size of the prediction.

Use np.log1p(x) when zeros are possible, because it computes log(1 + x) and avoids log of zero.

import numpy as np

# Train on the log of the target
y_train_log = np.log1p(y_train)
model.fit(X_train, y_train_log)

# Predict, then return to the original units before scoring
pred_log = model.predict(X_test)
pred = np.expm1(pred_log)
mae = mean_absolute_error(y_test, pred)

Putting it in order

1. Assess the distributions and decide whether scaling or a log transform is needed.
2. Split the data into train and test (and validation) first.
3. Fit imputers, scalers and encoders on the training data.
4. Transform the validation and test data with those fitted objects.
5. Save the fitted pipeline so the same steps run at inference time.

8. Feature Store fundamentals

Features are the inputs a model learns from. In a churn model they could be trailing seven-day purchase totals, a customer's demographics joined to their transactions, or an estimated lifetime value. The work of turning raw data into these values is feature engineering. A feature store manages those features so they are computed once, documented, governed and reused.

Why a feature store

Problem	What the feature store provides
Discovery: several teams build the same feature with different definitions	A shared catalogue where features can be found and reused
Lineage: governance needs to know how a feature is computed and who owns it	Feature tables are Unity Catalog tables, so lineage is recorded automatically
Skew: training and production compute features differently, so predictions drift	The same feature definition is used for training and for inference lookups
Online serving: exploration code does not scale to production	Production-grade tables, with an online copy for low-latency lookups

Feature tables in Unity Catalog

• A feature table is a Delta table in Unity Catalog with a primary key. The primary key identifies each entity (for example customer_id) so features can be looked up by key.
• Any suitable Delta table can become a feature table. The table's schema is the feature schema and its comment is the feature description.
• The table is governed by the Unity Catalog permission model, so grants and auditing apply to features exactly as they do to other tables.
• Lineage from feature table to model is recorded, so you can see which features a model depends on.
• Three-level naming is used: catalog.schema.feature_table.

Creating and writing a feature table

from databricks.feature_engineering import FeatureEngineeringClient

fe = FeatureEngineeringClient()

# Create the feature table with a primary key, and load initial data
fe.create_table(
    name="main.ml.customer_features",
    primary_keys=["customer_id"],
    df=customer_features_df,
    description="Per-customer purchase features"
)

# Later, add or update rows. "merge" upserts by primary key.
fe.write_table(
    name="main.ml.customer_features",
    df=new_customer_features_df,
    mode="merge"
)

# Read the current feature values
features = fe.read_table(name="main.ml.customer_features")

Creating an empty table is also possible: define the schema and primary keys first, then write data to it. In SQL, a Delta table becomes usable as a feature table once its primary key is defined as a constraint on NOT NULL columns, for example:

ALTER TABLE main.ml.customer_features ALTER COLUMN customer_id SET NOT NULL;
ALTER TABLE main.ml.customer_features ADD CONSTRAINT customer_pk PRIMARY KEY (customer_id);

Adding features to an existing table

New feature columns can be added to a Delta table with ALTER TABLE ... ADD COLUMNS, then written with a merge by primary key. Existing rows keep their keys, so downstream training sets that do not use the new columns are not affected.

Offline and online feature tables

Aspect	Offline feature table	Online feature table
Storage	Delta tables in Unity Catalog	A low-latency copy synced from the offline table
Typical use	Building training sets, batch scoring, analysis	Real-time inference that looks up features per request
Latency	Batch-oriented	Designed for millisecond key lookups
Data volume	Full history if needed	The current values needed for serving

9. Training and scoring with feature tables

A feature table earns its place when a model uses it. The model is trained from a set of labelled rows that carry only keys and the label. The feature store attaches the feature values at training time, and records which features were used so that later scoring can look them up again automatically.

The end-to-end pattern

1. Create the feature table with its primary key.
2. Write feature values for every entity, keyed by the primary key.
3. Build a training set: a DataFrame of keys and labels, plus FeatureLookup definitions that name the table and the features to retrieve.
4. Train the model on the training set's DataFrame.
5. Log the model with the feature client, so the model keeps the lookup metadata.
6. Score in batch with score_batch, or in real time through a serving endpoint that performs the lookups.

from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
from sklearn.ensemble import RandomForestClassifier
import mlflow

fe = FeatureEngineeringClient()

# Labelled rows carry only the key and the label
label_df = spark.table("main.ml.churn_labels")   # customer_id, churned

feature_lookups = [
    FeatureLookup(
        table_name="main.ml.customer_features",
        feature_names=["purchases_7d", "avg_basket"],
        lookup_key="customer_id"
    )
]

training_set = fe.create_training_set(
    df=label_df,
    feature_lookups=feature_lookups,
    label="churned",
    exclude_columns=["customer_id"]
)
train_pdf = training_set.load_df().toPandas()

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(train_pdf.drop(columns=["churned"]), train_pdf["churned"])

# Log the model with the training set, so feature metadata is stored with it
fe.log_model(
    model=model,
    artifact_path="churn_model",
    flavor=mlflow.sklearn,
    training_set=training_set,
    registered_model_name="main.ml.churn_model"
)

Scoring with feature lookups

For batch scoring, pass a DataFrame that contains only the keys, and the model's logged metadata tells the client which features to fetch:

predictions = fe.score_batch(
    model_uri="models:/main.ml.churn_model@champion",
    df=spark.table("main.ml.customers_to_score")   # customer_id only
)

For real-time inference, the endpoint serving the model retrieves the features from the online table at request time, so the caller sends the key rather than the whole feature vector.

Training-serving consistency

• Because the same feature table and lookup definitions are recorded with the model, the training features and the inference features come from one definition. This removes a common source of skew.
• Point-in-time correctness matters for features that change over time. Training sets built from time-aware lookups should not use values from after the label was observed.
• Updating a feature table changes what later lookups return. Keep a versioned history, through Delta time travel, if you must reproduce an old training run.

Writing, reading and deleting

Action	How
Create a table from a DataFrame	fe.create_table(name, primary_keys, df)
Add or update rows	fe.write_table(name, df, mode="merge")
Read the table	fe.read_table(name)
Build training data	fe.create_training_set(df, feature_lookups, label)
Score a batch	fe.score_batch(model_uri, df)
Remove the feature table	Drop the underlying Unity Catalog table, which is a Delta table

Upgrading a workspace feature table

A workspace feature table can be upgraded to a Unity Catalog feature table. Migrate the data into a Unity Catalog Delta table, define the primary key, and create the training and scoring code against the new table name. Move the consumers across before retiring the old table.

10. Paradigms and algorithm choice

Machine learning covers several paradigms, and the first decision in any project is which one matches the data and the question.

Types of machine learning

Paradigm	Data	Goal	Typical model family
Supervised	Inputs with known labels	Predict the label for unseen inputs. Models the conditional probability P(Y given X1 ... Xn)	Regression and classification
Unsupervised	Inputs with no labels	Find structure. Models the joint probability P(X1, X2, ..., Xn)	Clustering, dimensionality reduction
Semi-supervised	A small labelled set plus many unlabelled rows	Use the few labels to infer labels for the larger set	Hybrid methods
Self-supervised	Unlabelled data, with labels made from the data itself	Learn patterns in a large dataset that transfer to another task	Deep networks, pretrained encoders
Reinforcement	States, actions and rewards from interaction	Learn a policy that maximises cumulative reward	Neural networks and deep learning

Supervised tasks

• Regression predicts a continuous output, such as sales, the number of viewers or a price.
• Classification predicts a discrete category, such as yes or no, an image class or the presence of a disease.
• Forecasting predicts future values from historical data, such as demand. The order of time matters, so it is a special case that needs time-aware splitting.

Unsupervised tasks

• Clustering groups records with similar features. K-means is the common algorithm. It needs a number of clusters k and assigns each record to the nearest centroid.
• Dimensionality reduction reduces the number of features while keeping most of the structure. Principal component analysis (PCA) is the standard method, and it projects data onto directions of greatest variance.
• Embeddings turn items such as text, categories or images into dense vectors. Clustering the embeddings groups items by meaning rather than by raw column values.

Choosing a model family

Model	Strengths	Watch for
Linear regression	Interpretable coefficients, fast, a good baseline	Assumes linear effects. Sensitive to scale and outliers
Logistic regression	Probabilistic output, interpretable, fast	Linear decision boundary unless features are engineered
Decision tree	Captures nonlinearity and interactions. No scaling needed. Readable rules	Overfits deep trees. Unstable with small changes in data
Random forest	Strong accuracy with little tuning. Averages many trees	Less interpretable. Larger models
Gradient-boosted trees	Often the best accuracy on tabular data	Needs careful tuning to avoid overfitting
SVM	Effective in high-dimensional spaces with clear margins	Needs scaled features. Slow on very large data
K-nearest neighbours	Simple and non-parametric	Slow at prediction time. Needs scaled features
Neural networks	Learn complex patterns, text, images, sequences	Data hungry. Needs tuning and more compute

Estimators and transformers

Spark ML and scikit-learn both separate learning from transforming.

• Transformer: an object with transform() that maps one DataFrame to another. A fitted model is a transformer, and so is a one-hot encoder.
• Estimator: an object with fit() that learns from data and returns a fitted model. Logistic regression, a StringIndexer and an Imputer are estimators.
• In scikit-learn, the same idea appears as fit and transform or predict methods on the same objects.

from pyspark.ml.feature import StringIndexer, OneHotEncoder
from pyspark.ml.classification import LogisticRegression

indexer = StringIndexer(inputCol="city", outputCol="city_idx")   # Estimator
indexer_model = indexer.fit(train_df)                             # returns a Transformer
train_idx = indexer_model.transform(train_df)                     # Transformer

encoder = OneHotEncoder(inputCols=["city_idx"], outputCols=["city_vec"])  # Estimator
lr = LogisticRegression(featuresCol="features", labelCol="label")         # Estimator

Machine learning foundations to choose the algorithm

• Classification or regression, decided by whether the target is categorical or continuous.
• Labelled or unlabelled data, decided by whether a target column exists.
• Data size and feature count, which affect training time and overfitting risk.
• Interpretability requirements, which may force a linear model or a shallow tree.
• Feature types, which decide encoding and scaling needs.

11. Pipelines, AutoML and imbalance

Building a model is a sequence of stages: prepare the data, split it, fit the preprocessing on training data, train, evaluate and iterate. Pipelines and AutoML make that sequence repeatable.

Training pipelines

A pipeline chains transformers and a final estimator, so one fit call trains every stage in order on the training data, and one transform or predict call runs the same steps on new data. It removes manual steps that differ between training and scoring, and it prevents leakage, because each stage is fitted on training rows only.

# scikit-learn
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000))
])
pipe.fit(X_train, y_train)          # scaler learns the training mean and sd
pipe.predict(X_test)                # the same scaling is reused

# Spark ML
from pyspark.ml import Pipeline as SparkPipeline
spark_pipe = SparkPipeline(stages=[indexer, encoder, assembler, lr])
pipeline_model = spark_pipe.fit(train_df)     # returns a PipelineModel
predictions = pipeline_model.transform(test_df)

Machine learning runtime and environment

The ML runtime gives a consistent set of libraries for training. Use the same runtime for training and scoring, so package versions do not change results. The model's environment is recorded with it when it is logged, which supports reproducibility.

AutoML

AutoML takes a dataset and a target, then automatically profiles the data, applies preprocessing, selects features and algorithms, tunes hyperparameters and ranks the resulting models. For classification, regression and forecasting it produces leaderboards and generated notebooks with the code for each candidate.

Advantage	Explanation
Speed	A strong baseline in minutes rather than days
Best practice by default	Standard splits, sensible preprocessing and tuning are applied consistently
Transparency	Generated notebooks show exactly what was done, so the work can be audited and edited
Tracking	Every candidate is logged to MLflow with its parameters and metrics
Domain input	Analysts can edit the generated code to add knowledge AutoML does not have

AutoML does not replace problem framing, metric choice or checks for leakage. Review the generated notebook and the metric before trusting the top model.

Data imbalance

When one class is rare, a model can score high accuracy by predicting the majority class every time. Methods that counter imbalance:

• Cost-sensitive learning: assign a higher misclassification cost to the minority class during training. The model is penalised more for missing a rare case. In scikit-learn this is class_weight="balanced" or custom weights.
• Oversampling the minority class, by duplicating rows or synthesising new ones (for example SMOTE). Apply it to training folds only.
• Undersampling the majority class, which is fast but discards information.
• Stratified splitting and stratified cross-validation, so every split keeps the class ratio.
• Metric choice: use precision, recall, F1, PR-AUC or ROC-AUC rather than accuracy.
• Threshold tuning: move the decision threshold on predicted probabilities to trade precision against recall.

12. Metrics and the bias-variance trade-off

A metric turns a model's behaviour into a number that can be compared and optimised. Choosing the right metric is as important as choosing the model, because each metric rewards a different kind of success.

Regression metrics

Metric	What it measures	Range and direction	Notes
R-squared (R²)	Proportion of the variance in the target explained by the model	Up to 1. Higher is better. Can be negative for a model worse than predicting the mean	Scale-free, but not a measure of error size
Mean absolute error (MAE)	Average absolute difference between actual and predicted	Lower is better, in the target's units	Less sensitive to outliers
Mean squared error (MSE)	Average of squared differences	Lower is better, in squared units	Penalises large errors more heavily. Common for optimisation
Root mean squared error (RMSE)	Square root of MSE	Lower is better, in the target's units	Interpretable like MAE, but still penalises large errors

from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

r2 = r2_score(y_true, y_pred)
mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = mean_squared_error(y_true, y_pred) ** 0.5   # or root_mean_squared_error in recent scikit-learn

Classification metrics

Almost every classification metric starts from the confusion matrix. For a binary problem, with positive as the class of interest:

	Predicted positive	Predicted negative
Actual positive	True positive (TP)	False negative (FN)
Actual negative	False positive (FP)	True negative (TN)

Metric	Formula	Meaning and when it matters
Accuracy	(TP + TN) / total	Proportion correct. Misleading with class imbalance
Precision	TP / (TP + FP)	Of the predicted positives, how many are right. Matters when false positives are costly, such as flagging legitimate email as spam
Recall (sensitivity)	TP / (TP + FN)	Of the actual positives, how many are found. Matters when false negatives are costly, such as missing a disease
F1 score	2 times precision times recall / (precision + recall)	Harmonic mean of precision and recall. A balance when both errors matter
Log loss (cross-entropy)	Average of negative log of the probability given to the true class	Uses predicted probabilities. Penalises confident wrong predictions heavily. Lower is better
ROC-AUC	Area under the curve of true positive rate against false positive rate, across all thresholds	Measures how well the model ranks positives above negatives. 0.5 is random, 1 is perfect

from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             f1_score, log_loss, roc_auc_score, confusion_matrix)

accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)
cm = confusion_matrix(y_true, y_pred)

# Probability-based metrics need scores, not hard labels
ll = log_loss(y_true, y_prob)
auc = roc_auc_score(y_true, y_prob)

Clustering metrics

• Silhouette score compares, for each point, the distance to its own cluster with the distance to the nearest other cluster. Values near 1 mean well-separated clusters, values near 0 mean overlap. It is computed with silhouette_score(X, labels).
• Elbow method plots the within-cluster sum of squares (inertia) against the number of clusters k. Choose the k where adding clusters stops giving a large improvement, the elbow.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

labels = KMeans(n_clusters=2, random_state=42, n_init=10).fit_predict(X)
score = silhouette_score(X, labels)

inertias = [KMeans(n_clusters=k, random_state=42, n_init=10).fit(X).inertia_
            for k in range(1, 7)]

Metrics on transformed targets

If the target or a feature was log-transformed, exponentiate the predictions before scoring, as covered in the Data Preparation realm. Metrics computed on the log scale answer a different question.

Bias, variance and model complexity

• Bias is error from overly simple assumptions. A high-bias model underfits: it is wrong on training data and on new data.
• Variance is error from sensitivity to the training sample. A high-variance model overfits: it does well on training data and worse on new data.
• Increasing complexity (deeper trees, more features, more neurons) lowers bias and raises variance. Reducing complexity does the opposite.
• The goal is the complexity at which total error on unseen data is lowest. Regularisation, pruning, more data and early stopping move the balance toward lower variance.

Symptom	Training score	Validation score	Likely cause and fix
Underfitting	Low	Low, close to training	Too simple. Add features or capacity, reduce regularisation
Overfitting	High	Much lower	Too complex or too little data. Regularise, simplify, add data, use cross-validation
Good fit	High	Similar and high	Balanced complexity

13. Hyperparameters and search methods

A model has two kinds of numbers. Parameters are learned from the data, such as regression coefficients or tree split thresholds. Hyperparameters are set before training and control how learning happens. Hyperparameter tuning, also called model tuning, searches for the hyperparameter values that give the best performance on validation data.

Common hyperparameters

• Random forest: maximum depth of the trees, number of trees, number of features considered at each split.
• Linear models: regularisation strength, and whether to normalise features.
• Gradient boosting: learning rate, number of boosting rounds, tree depth.
• SVM: the penalty C, the kernel, and the kernel width gamma.
• Neural networks: the activation function, the number of neurons in each hidden layer, learning rate.

Search methods

Method	How it works	Strengths	Weaknesses
Grid search	Defines a grid of values and tries every combination	Systematic. Finds the best combination within the grid	Cost grows multiplicatively with each hyperparameter
Random search	Samples combinations from distributions	More efficient in high dimensions. Often finds good values faster	No guarantee of the best combination, and it does not cover the grid
Bayesian optimisation	Builds a probability model of the objective and uses past results to pick promising values next	Uses fewer trials on expensive models. Often the most efficient	More complex to implement. Overhead on very cheap models

Tree-structured Parzen Estimator (TPE) is a popular Bayesian method. It models the good and the bad trials separately and proposes values that are likely to be in the good region. It is the default sampler in Optuna and is the algorithm behind Hyperopt's tpe.suggest.

Cross-validation as the scoring step

Tuning uses validation data, and a single validation split can be noisy. Cross-validation scores each combination over k folds and averages the results, so the choice is less sensitive to one lucky split.

Train-validation split	K-fold cross-validation
One fit per combination	k fits per combination
Cheap	k times the compute
Estimate can be noisy with small data	Average over folds is more stable
Uses a fixed portion of the data for validation	Every row is used for validation once

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    "C": [0.1, 1, 10],
    "kernel": ["linear", "rbf"],
    "gamma": [0.01, 0.1, 1]
}
search = GridSearchCV(SVC(), param_grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)

Parallelising tuning

• Single-node models can be tuned in parallel with n_jobs=-1 in scikit-learn search classes, which uses all available cores.
• Trials of a single search are independent, so they parallelise well. Each trial is a separate model fit.
• Spark-based parallelism distributes trials across the cluster. Hyperopt's SparkTrials does this, and Optuna can run trials across processes.
• Parallel trials cannot learn from each other within the same round, so Bayesian methods trade a little efficiency for parallel speed.

Hyperopt and fmin

Hyperopt is a Bayesian optimisation library that provides the fmin function. You define an objective function that returns a loss to minimise, a search space, an algorithm and a budget of evaluations.

from hyperopt import fmin, tpe, hp, Trials, STATUS_OK
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score

space = {
    "max_depth": hp.quniform("max_depth", 2, 12, 1),
    "n_estimators": hp.quniform("n_estimators", 50, 300, 10),
}

def objective(params):
    model = RandomForestRegressor(
        max_depth=int(params["max_depth"]),
        n_estimators=int(params["n_estimators"]),
        random_state=42)
    rmse = -cross_val_score(model, X_train, y_train, cv=3,
                            scoring="neg_root_mean_squared_error").mean()
    return {"loss": rmse, "status": STATUS_OK}

trials = Trials()
best = fmin(fn=objective, space=space, algo=tpe.suggest,
            max_evals=50, trials=trials)

Ray Tune

Ray Tune is another scalable tuning library. It offers grid and advanced search algorithms and runs trials across a cluster. It is a good choice when tuning must scale beyond one machine.

14. Tuning with Optuna

Optuna is an open-source hyperparameter optimisation framework. It automates the search, adapts it using earlier results, and records every trial. It is written in Python with few dependencies, and it works with any framework, including scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow and Keras, because it only needs an objective function that returns a number.

Terminology

Term	Meaning
Objective function	A Python function that takes a trial, builds a model with the suggested values, and returns the metric to optimise
Trial	One call of the objective function, with one set of suggested values
Study	An optimisation session: a collection of trials and the settings used to choose the next ones
Parameter (in Optuna)	A variable being optimised, usually the loss or the accuracy
Search space	The ranges and choices suggested for each parameter, defined inside the objective
Sampler	The strategy that proposes the next values. The default is TPE
Pruner	Stops unpromising trials early, using intermediate results. Beyond the core exam scope

Defining the search space

Inside the objective, trial suggests values. Each suggest call names a parameter and gives its range or choices:

Method	Suggests
trial.suggest_int(name, low, high)	Integers in a range
trial.suggest_float(name, low, high)	Floats in a range. Use log=True for values spread over orders of magnitude, and step= for a grid of floats
trial.suggest_categorical(name, choices)	One value from a list
trial.suggest_uniform and trial.suggest_discrete_uniform	Older names. Deprecated in favour of suggest_float with the matching options

A complete study

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 300),
        "max_depth": trial.suggest_int("max_depth", 2, 16),
        "max_features": trial.suggest_categorical("max_features", ["sqrt", "log2"]),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 10),
    }
    model = RandomForestClassifier(random_state=42, **params)
    score = cross_val_score(model, X_train, y_train, cv=3, scoring="f1").mean()
    return score

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50, n_jobs=-1)

print(study.best_params)     # dictionary of the best values
print(study.best_value)      # the best objective value
print(study.best_trial)      # the full best trial

Reading the results

• study.best_params, a dictionary of the best parameter values.
• study.best_value, the objective value of the best trial.
• study.best_trial, the best trial object, including its parameters and value.
• study.best_trials, the set of Pareto-optimal trials, used for multi-objective studies. For a single objective it is not the usual choice.
• study.trials_dataframe(), a pandas table of every trial, useful for plotting progress.

How Optuna chooses the next trial

The sampler looks at finished trials, models which regions gave good values, and proposes new values from the promising regions. The default TPE sampler does this with a Bayesian model. Grid and random samplers are also available, and the sampler choice does not change the objective or the search space.

Tracking trials with MLflow

Optuna's MLflowCallback logs each trial's parameters and metrics to MLflow automatically. Each trial becomes a child run of a parent run, so all the trials of one study stay grouped together in the experiment.

from optuna.integration.mlflow import MLflowCallback

mlflow_callback = MLflowCallback(
    tracking_uri="databricks",
    create_experiment=False,
    mlflow_kwargs={
        "experiment_id": experiment_id,
        "nested": True
    }
)

study.optimize(objective, n_trials=50, callbacks=[mlflow_callback])

• tracking_uri="databricks" sends the runs to the Databricks workspace's MLflow.
• create_experiment=False uses the experiment you name, rather than creating a new one.
• nested: True groups the trial runs under a parent run.
• Callbacks run after each trial, so logging happens as the study progresses.

Retrieving the best model

The study records the best parameters, but the model itself must be refitted with those parameters on the full training data and then logged or registered. Pulling the best run from MLflow is an alternative when the trials logged their models: query the experiment for the best run by its metric and load that run's model.

import mlflow

best_run = mlflow.search_runs(
    experiment_ids=[experiment_id],
    order_by=["metrics.f1 DESC"],
    max_results=1
).iloc[0]

best_model = mlflow.sklearn.load_model(f"runs:/{best_run.run_id}/model")

15. Serving modes: batch, streaming, real-time and edge

Model deployment is the step that moves a trained model out of the development notebook and into a production environment, where end users or other systems can receive predictions or insights. A model creates value only once something calls it: a scheduled job, a stream of events, or an application making requests. The right deployment method depends on how quickly a prediction is needed, how many records must be scored, and how often the input data changes.

The full machine learning lifecycle runs from a business problem and success criteria, through data collection, preprocessing and feature engineering, and model training and evaluation on static historical data. Deployment and production is where the model meets continuously changing data. Monitoring then tells you when performance drifts, which triggers retraining and redeployment.

Four deployment methods

Method	Throughput	Latency	Typical example
Batch	High	High (hours to days)	Periodic customer churn scores
Streaming (pipeline)	Moderate	Moderate (seconds to minutes)	Scoring a continuous feed of listening or click events
Real-time (request and response)	Lower per endpoint	Low (milliseconds)	Fraud check during a payment, chatbot reply
Edge or embedded	Limited by the device	Low, on device	Sensor on a farm reading humidity and classifying locally

Batch deployment

Batch inference generates predictions on a regular schedule. A job reads the new records, scores them with the model, and writes the results to persistent storage, where downstream consumers such as BI dashboards pick them up. It is the most common deployment strategy.

• Use it when immediate predictions are not necessary, predictions can be made in bulk, the number of new records is large, and the input changes or arrives at intervals of roughly 30 minutes or more.
• Advantages: the cheapest method, easy to implement, efficient per record, and able to handle very high volume.
• Limitations: high latency, predictions go stale between runs, and it is a poor fit for rapidly changing data or streaming requirements.

Streaming (pipeline) deployment

Streaming deployment scores records continuously as they arrive rather than waiting for a batch window. A pipeline reads an incoming stream, applies the model in its final step, and writes predictions to a table, so downstream systems see results within seconds to minutes. It is event driven, so systems can act sooner than with batch.

• Advantages: lower latency than batch, and predictions arrive while they can still be acted on.
• Limitations: costlier than batch, more complicated to develop, maintain and monitor, lower throughput than batch on the same hardware, and resource intensive because compute has to keep running to process the feed.

Real-time deployment

Real-time deployment produces a prediction in response to each request, usually through a REST API. Typical uses are fraud detection during a transaction, a chatbot answering a user, or a recommendation displayed while a page loads. Requirements usually include low latency, immediate action, 24/7 availability and continuous monitoring. Large language models are increasingly served this way.

Building real-time serving yourself means fast, scalable serving infrastructure that is costly to build and maintain, plus tooling for packaging, scaling, security and monitoring. Model Serving on Databricks is the managed option covered in the last chapter of this realm.

Edge and embedded deployment

Edge deployment runs inference on local hardware such as a phone, gateway or sensor. Choose it when connectivity is offline or unreliable, when latency must be very low, or when data has to stay on the device for security or compliance. The model is exported to a portable runtime such as ONNX Runtime or TensorFlow Lite, as described in the next chapter.

16. Packaging models with MLflow for deployment

Before a model can be deployed anywhere, it has to be packaged so that any target can load it. MLflow provides that packaging through the MLflow Model format and the Model Registry. MLflow has four components: Tracking records experiments (code, data, configuration and results), Projects packages code for reproducible runs, Models defines a general model format that supports many deployment tools, and the Model Registry manages the model lifecycle centrally.

Packaging with MLflow keeps the deployment environment consistent with the training environment, because the dependencies and configuration travel with the model. The same packaged model can be served by a local server, in containers on Kubernetes or a cloud service, or by Databricks Model Serving.

Anatomy of an MLflow Model

An MLflow Model is a directory that holds arbitrary files plus an MLmodel file. The MLmodel file lists the flavors the model can be viewed in and can also carry metadata such as a signature and an input example. Saving a scikit-learn model with mlflow.sklearn.save_model(model, "my_model") writes a directory like this:

my_model/
├── MLmodel
├── model.pkl
├── conda.yaml
├── python_env.yaml
└── requirements.txt

A trimmed MLmodel file looks like this:

sklearn:
  sklearn_version: 0.19.1
  pickled_model: model.pkl
python_function:
  loader_module: mlflow.sklearn

Flavors and the python_function interface

A flavor is one way of viewing the same saved model. The python_function flavor, exposed as mlflow.pyfunc, is the generic interface. A deployment tool that knows only pyfunc can load any MLflow Python model and call predict, which is why pyfunc is the default route for batch jobs, Spark and serving. The pyfunc API includes log_model, save_model, load_model and predict.

Built-in flavors cover scikit-learn, Spark MLlib, PyTorch, TensorFlow, ONNX, Hugging Face models and LLM libraries such as LangChain and OpenAI. You can also define custom flavors.

Signatures and input examples

A signature records the input and output schema of a model. An input example is a sample payload. Both are stored in the MLmodel file, and they let pyfunc and serving check that incoming columns and types match what the model expects.

from mlflow.models import infer_signature

signature = infer_signature(X_train, model.predict(X_train))
mlflow.sklearn.log_model(
    model, "model", signature=signature, input_example=X_train.head(3)
)

Registering in Unity Catalog and loading by alias

import mlflow
from mlflow import MlflowClient

mlflow.set_registry_uri("databricks-uc")
mlflow.register_model(
    model_uri=f"runs:/{run_id}/model",   # run_id comes from the training run
    name="catalog.schema.churn_model",   # three-level name required in Unity Catalog
)

client = MlflowClient()
client.set_registered_model_alias(
    name="catalog.schema.churn_model", alias="champion", version=3
)

model = mlflow.pyfunc.load_model("models:/catalog.schema.churn_model@champion")

The Unity Catalog registry stores models with three-level names and gives centralized governance, permissions, lineage, versioning, tags and annotations. Aliases such as champion and challenger mark which version is in use. Code that loads by alias picks up a promoted version the next time it runs, without a code change. A serving endpoint is different: each served version is pinned, so promoting a new champion means updating the endpoint configuration.

Custom models with PythonModel

When a model needs logic beyond a single estimator, such as preprocessing, a business rule or several artifacts, wrap it in a subclass of mlflow.pyfunc.PythonModel. load_context runs once when the model is loaded and can read the artifacts logged with it. predict runs on each request or batch of rows.

import joblib
import mlflow.pyfunc

class ChurnWrapper(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        # base_model.pkl was written with joblib.dump before logging
        self.model = joblib.load(context.artifacts["base_model"])

    def predict(self, context, model_input, params=None):
        proba = self.model.predict_proba(model_input)[:, 1]
        return (proba >= 0.5).astype(int)

with mlflow.start_run():
    mlflow.pyfunc.log_model(
        artifact_path="model",   # MLflow 3 uses name= in place of artifact_path
        python_model=ChurnWrapper(),
        artifacts={"base_model": "base_model.pkl"},
        signature=signature,
    )

The wrapper class must be importable wherever the model is loaded, so keep it in a Python file or in the logging notebook, and make sure its dependencies are captured in the logged environment.

Exporting for edge runtimes

When the target cannot run the Python environment, such as a phone, a gateway or specialized hardware, export the model to ONNX and score it with ONNX Runtime. The ONNX file is logged as its own flavor, and the consumer reads it without MLflow.

import numpy as np
import mlflow
import onnxmltools
import onnxruntime

onnx_model = onnxmltools.convert_keras(keras_model)
mlflow.onnx.log_model(onnx_model, "onnx_model")

session = onnxruntime.InferenceSession(onnx_model.SerializeToString())
input_name = session.get_inputs()[0].name
predictions = session.run(None, {input_name: data_np.astype(np.float32)})[0]

Target	Format	How it is loaded
Databricks, Spark or Python service	MLflow pyfunc	mlflow.pyfunc.load_model or spark_udf
Edge or embedded device	ONNX or TensorFlow Lite	ONNX Runtime or TensorFlow Lite interpreter

17. Batch and streaming inference in pipelines

Batch and pipeline inference run on Spark, Delta tables and Lakeflow Declarative Pipelines. Choosing the right tool depends on whether the data fits on one machine, whether features come from a feature table, and whether the scoring must run continuously.

Batch inference with pandas

Pandas batch inference loads the model on the driver with mlflow.pyfunc.load_model and calls predict on a pandas DataFrame. It suits data that fits comfortably in driver memory, such as a few hundred megabytes. Write the results back to Delta with Spark.

import mlflow.pyfunc

model = mlflow.pyfunc.load_model("models:/catalog.schema.churn_model@champion")
feature_cols = ["tenure", "monthly_charges", "support_calls"]

features_pdf = spark.table("catalog.schema.customer_features").toPandas()
features_pdf["churn_score"] = model.predict(features_pdf[feature_cols])

spark.createDataFrame(features_pdf).write.mode("overwrite") \
    .saveAsTable("catalog.schema.churn_predictions")

Distributed batch inference with a Spark UDF

mlflow.pyfunc.spark_udf turns a pyfunc model into a Spark UDF. The model is loaded on the executors and each executor scores its share of the rows, usually in vectorized batches passed as pandas Series. Pass the feature columns in the order the model signature expects, and set result_type to match the model output.

import mlflow.pyfunc
from pyspark.sql.functions import col

predict_udf = mlflow.pyfunc.spark_udf(
    spark,
    model_uri="models:/catalog.schema.churn_model@champion",
    result_type="double",
)

feature_cols = ["tenure", "monthly_charges", "support_calls"]
scored = (
    spark.table("catalog.schema.customer_features")
    .withColumn("churn_score", predict_udf(*[col(c) for c in feature_cols]))
)
scored.write.mode("overwrite").saveAsTable("catalog.schema.churn_predictions")

Batch scoring with Feature Store

If the model was trained with features from an offline feature table, the batch job only needs the primary keys. score_batch joins the features from the feature table, applies the model and returns the predictions. The model must have been logged with fe.log_model, which stores the feature lookup metadata with it. The legacy Feature Store client exposes the same method as FeatureStoreClient.score_batch.

from databricks.feature_engineering import FeatureEngineeringClient

fe = FeatureEngineeringClient()
new_customers = spark.table("catalog.schema.customers_to_score").select("customer_id")

predictions = fe.score_batch(
    model_uri="models:/catalog.schema.churn_model@champion",
    df=new_customers,
)
predictions.write.mode("append").saveAsTable("catalog.schema.churn_predictions")

An offline feature table is a Delta table with a primary key. It suits batch inference and training, but it is not optimized for real-time latency.

Delta layout and maintenance for batch tables

• Liquid clustering replaces table partitioning and ZORDER. Enable it with CLUSTER BY (Databricks Runtime 13.3 or later). Databricks recommends it for all new Delta tables, and it manages the layout for you.
• OPTIMIZE compacts many small files into larger ones, which speeds up reads. On tables that do not use liquid clustering, OPTIMIZE table ZORDER BY (col) also co-locates related values.
• Predictive optimization finds tables that would benefit from maintenance and runs OPTIMIZE and VACUUM for you. Enable it at the account level first, then per catalog or schema.
• VACUUM deletes data files that the table no longer references, once they are older than the retention period (7 days by default).

CREATE TABLE catalog.schema.customer_features (customer_id BIGINT, tenure INT)
USING DELTA
CLUSTER BY (customer_id);

ALTER CATALOG catalog1 ENABLE PREDICTIVE OPTIMIZATION;

Streaming inference with Lakeflow Declarative Pipelines

Delta Live Tables (DLT) is now branded Lakeflow Declarative Pipelines, and the older name, Delta Live Tables, is still common in documentation and existing code. You declare tables in SQL or Python, and the pipeline works out the dependency graph, runs updates in order, retries failures, scales compute and enforces data quality. Two kinds of table are common. A streaming table processes new rows incrementally and is the usual target for ingestion and scoring. A materialized view stores the result of a query and refreshes it. Older DLT syntax used CREATE LIVE TABLE and CREATE STREAMING LIVE TABLE.

CREATE OR REFRESH STREAMING TABLE bronze_events
AS SELECT *
FROM cloud_files("/Volumes/catalog/schema/raw_events", "json");

CREATE OR REFRESH MATERIALIZED VIEW clean_events
AS SELECT user_id, event_type, event_ts
FROM bronze_events;

Auto Loader, exposed in pipelines through cloud_files, picks up new files as they land in cloud storage and processes each one once. It can infer the schema, accept schema hints, and evolve the schema automatically. It reads JSON, CSV, Avro and Parquet.

The inference step is the last table in the pipeline. The model is applied as a Spark UDF, so each executor scores its rows and pipeline autoscaling sizes the compute:

from pyspark import pipelines as dp
from pyspark.sql.functions import col
import mlflow.pyfunc

predict_udf = mlflow.pyfunc.spark_udf(
    spark, "models:/catalog.schema.churn_model@champion", result_type="double"
)

@dp.table(name="scored_events")
def scored_events():
    return (
        spark.readStream.table("clean_events")
        .withColumn("score", predict_udf(col("feature_a"), col("feature_b")))
    )

Data quality in the pipeline

Expectations state a rule that every row should satisfy. The ON VIOLATION clause sets the action. DROP ROW removes the failing rows, FAIL UPDATE stops the update, and with no clause the violation is recorded in the pipeline metrics and the row is kept. All of these are tracked in pipeline metrics and the event log, which also provide a lineage diagram and row-level logging. Notifications can be sent with Databricks SQL alerts.

CREATE OR REFRESH STREAMING TABLE clean_events (
  CONSTRAINT valid_user EXPECT (user_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_ts EXPECT (event_ts IS NOT NULL) ON VIOLATION FAIL UPDATE
)
AS SELECT * FROM STREAM bronze_events;

18. Real-time inference with Model Serving

Real-time deployment returns a prediction immediately in response to a request. Databricks Model Serving hosts models behind REST endpoints, so applications can call them without managing servers. Model Serving covers three kinds of served content: custom models (your MLflow models), Foundation Model APIs (curated foundation models behind simple APIs), and external models (third-party model APIs governed through the same layer).

What a serving endpoint provides

• Deploys any MLflow model from Unity Catalog as a REST API on serverless compute.
• CPU and GPU support, with GPUs for deploying large language models.
• Autoscaling within a selected workload size range, with compute kept warm for fast scale-up.
• Scale to zero for predictable traffic that is not 24/7, which saves cost at the price of a cold start after idle periods.
• Low overhead latency and high throughput are the stated goals, and availability is published as 99.5%. Check current documentation for the exact figures.
• Security through PrivateLink and IP allowlists, and governance through Unity Catalog.
• Monitoring through a ready-made metrics dashboard, an export API for external observability platforms, and alerts through Databricks SQL.
• Integration with Feature Store lookups and Vector Search, so the endpoint can fetch features or vectors itself.

Endpoint-centric workflow

Each endpoint has a stable scoring URI, so callers never change when the models behind it change. One endpoint can host several models, and the same model can sit behind several endpoints. Each deployed model version is a served entity with its own workload size, scale-to-zero setting and traffic share. Staged rollout and A/B testing both come from changing those shares.

Deploying a custom model

The steps are: log the model and register a version in Unity Catalog, create the endpoint with that version, and then query it. You can create the endpoint in the Serving UI, or through the REST API, the Python SDK, the Go SDK, Terraform, or a Lakeflow Job that automates the deployment in a CI/CD process.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import EndpointCoreConfigInput, ServedEntityInput

w = WorkspaceClient()
w.serving_endpoints.create_and_wait(
    name="churn-endpoint",
    config=EndpointCoreConfigInput(
        served_entities=[
            ServedEntityInput(
                entity_name="catalog.schema.churn_model",
                entity_version="3",
                workload_size="Small",
                scale_to_zero_enabled=True,
            )
        ]
    ),
)

The REST equivalent is a POST to /api/2.0/serving-endpoints with the same configuration in the body.

Querying the endpoint

Requests go to the endpoint's invocations route. The body holds the rows as dataframe_records, a list of objects, or as dataframe_split, with separate columns and data arrays. The response contains a predictions list.

curl -X POST "https://<workspace-url>/serving-endpoints/churn-endpoint/invocations" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"dataframe_records": [{"tenure": 12, "monthly_charges": 70.5, "support_calls": 2}]}'

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
response = w.serving_endpoints.query(
    name="churn-endpoint",
    dataframe_records=[{"tenure": 12, "monthly_charges": 70.5, "support_calls": 2}],
)
print(response.predictions)

Splitting traffic between versions

Traffic splitting sends a share of requests to each served entity behind one endpoint. Use it for a canary release, staged rollout or A/B test. The percentages must add up to 100, and every route names a served entity. The configuration below is updated with a PUT to /api/2.0/serving-endpoints/{name}/config.

{
  "served_entities": [
    {"name": "churn-v3", "entity_name": "catalog.schema.churn_model",
     "entity_version": "3", "workload_size": "Small", "scale_to_zero_enabled": true},
    {"name": "churn-v4", "entity_name": "catalog.schema.churn_model",
     "entity_version": "4", "workload_size": "Small", "scale_to_zero_enabled": true}
  ],
  "traffic_config": {
    "routes": [
      {"served_entity_name": "churn-v3", "traffic_percentage": 90},
      {"served_entity_name": "churn-v4", "traffic_percentage": 10}
    ]
  }
}

Splitting inside one endpoint keeps a single stable URI for callers. If you need traffic divided between separate endpoints, the calling application does the routing. Once the new version proves itself, move its share to 100 percent and remove the old served entity.

Real-time inference with online features

Online feature serving removes feature logic from the caller. Features are stored in an offline feature table (a Delta table with a primary key), registered as lookups when the model is logged, and synced to an online table. At inference time the endpoint fetches the features automatically, so the request carries only the key.

from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup

fe = FeatureEngineeringClient()
training_set = fe.create_training_set(
    df=labels_df,  # columns: customer_id, churned
    feature_lookups=[
        FeatureLookup(table_name="catalog.schema.customer_features",
                      lookup_key="customer_id")
    ],
    label="churned",
)
fe.log_model(
    model=model,
    artifact_path="model",
    flavor=mlflow.sklearn,
    training_set=training_set,
    registered_model_name="catalog.schema.churn_model",
)

A request for this model needs only the key: {"dataframe_records": [{"customer_id": 1042}]}.

Feature functions compute on-demand features at inference time. They are Python UDFs governed by Unity Catalog, and they take inputs from the request and bindings defined during training. They support every type that Feature Store supports except MapType and ArrayType. Online feature stores offer low latency and high throughput, serverless autoscaling, point-in-time correctness for training data, and scheduled updates.

19. MLflow Tracking Fundamentals

MLflow is an open-source platform for the machine learning lifecycle. It is co-developed by Databricks and the wider community and is pre-installed on Databricks Runtime for ML. It has four components: Tracking, which records and compares experiments; Evaluation, which automates metric computation for common supervised tasks; Models, a unified packaging format that works across frameworks; and the Model Registry, which provides version control and managed deployment. MLflow can be used from Python, R, Java, the command line and the REST API.

Experiments

An experiment is the top-level container for a set of related runs. Teams create one to explore different configurations, parameters or algorithms for the same problem. There are two kinds:

• Notebook experiment: tied to the notebook that creates the runs. It is created automatically when training runs in a notebook that has no experiment set.
• Workspace experiment: stored at a path you choose, such as /Users/name@example.com/churn. Several notebooks can write runs into the same workspace experiment.

import mlflow
mlflow.set_experiment("/Users/name@example.com/churn")

Runs

A run is one execution of data science code. Each run belongs to exactly one experiment, and a run can be nested under a parent run. The table lists what a run holds.

Element	What it holds	Typical logging call
Parameters	Key-value inputs such as hyperparameters, stored as strings	log_param, log_params
Metrics	Numeric results, optionally with a step number to build a history	log_metric, log_metrics
Tags	Mutable key-value labels	set_tag, set_tags
Artifacts	Any files or folders: plots, data samples, JSON, text	log_artifact, log_artifacts, log_figure, log_text, log_dict
Model	Model folder with MLmodel, conda.yaml or requirements, and the serialized model	mlflow.sklearn.log_model and other flavors
Metadata	Run ID, experiment ID, status, start and end times, user and source	Recorded automatically

Manual logging inside a run looks like this. Used as a context manager, the run ends automatically when the block exits, even if an error is raised.

import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score

with mlflow.start_run(run_name="rf-baseline") as run:
    mlflow.log_params({"n_estimators": 200, "max_depth": 8})
    model = RandomForestClassifier(n_estimators=200, max_depth=8)
    model.fit(X_train, y_train)
    preds = model.predict(X_test)

    mlflow.log_metric("f1", f1_score(y_test, preds))
    mlflow.log_dict({"features": list(X_train.columns)}, "features.json")
    mlflow.sklearn.log_model(model, name="random_forest_model")
    run_id = run.info.run_id

Nested runs group child runs, for example one child per hyperparameter setting, under a parent run:

with mlflow.start_run(run_name="depth-search") as parent:
    for depth in [4, 8, 16]:
        with mlflow.start_run(run_name=f"depth-{depth}", nested=True):
            mlflow.log_param("max_depth", depth)

Artifacts

Artifacts are stored under the run and can be downloaded later. log_artifact copies one local file, log_artifacts copies a whole folder, log_figure saves a plot figure, log_text saves a string, and log_dict saves a dictionary as JSON. The logged model is also an artifact, stored with its MLmodel descriptor and environment files.

Autologging and custom logging

Autologging records parameters, metrics, the trained model and related details for supported libraries without extra code. Call mlflow.autolog() to enable it for every supported library, or a framework-specific form such as mlflow.sklearn.autolog(). Databricks Autologging is enabled by default in notebooks on Databricks Runtime for ML and can be turned off with mlflow.autolog(disable=True).

Custom logging covers what autologging cannot know: a business-specific score, a decision threshold, a plot, or a configuration file. Custom calls can be added inside the same active run as autologging.

mlflow.autolog()
with mlflow.start_run():
    model.fit(X_train, y_train)
    mlflow.log_metric("super_score", custom_score(y_test, model.predict(X_test)))

The MLflow Client API

MlflowClient gives programmatic access to experiments and runs, which is how selection and promotion decisions are automated. Common methods include search_experiments, get_experiment_by_name, search_runs, get_run, get_metric_history and list_artifacts.

search_runs takes experiment_ids (a list), filter_string (a SQL-like condition), order_by (a list such as ["metrics.rmse ASC"]) and max_results. Filters refer to metrics., params., tags. and attributes. keys. Parameter and tag values must be quoted strings, while metric values are compared as numbers.

from mlflow import MlflowClient

client = MlflowClient()
runs = client.search_runs(
    experiment_ids=["1234"],
    filter_string="params.model = 'random_forest' AND metrics.rmse < 10",
    order_by=["metrics.rmse ASC"],
    max_results=5,
)
if runs:
    best = runs[0]
    print(best.info.run_id, best.data.metrics["rmse"])

Identifying the best run

Choose the sort direction from what the metric means. Accuracy, F1, R-squared and ROC AUC are better when larger, so sort DESC. RMSE, MAE, MSE and log loss are better when smaller, so sort ASC. Always check that the result list is not empty before indexing it.

What the MLflow UI shows

The experiment page lists its runs in a table. Columns include run name, creation time, duration, user, source, and each logged parameter and metric. You can filter and search runs, select several to compare side by side, and plot metrics. Opening a run shows its overview (run ID, status, start and end times, duration, user and source notebook), its parameters, metrics with their history, tags, the artifact tree including the logged model and its signature, and a control to register the model. The UI suits exploration; the Client API suits automation.

20. Unity Catalog Model Registry

The Unity Catalog model registry stores registered models in the same three-level namespace used for tables: catalog.schema.model. Because models live in Unity Catalog, they are governed by its privileges, can be shared across every workspace attached to the same metastore, and link back to the data used to train them.

Core objects

Object	Meaning	Example
Registered model	Named container for all versions of one model	ml_prod.churn.churn_classifier
Model version	Integer-numbered snapshot created by each registration. Its artifacts cannot be changed afterwards	Version 4
Alias	Movable named pointer to one version of a registered model	champion, challenger
Tag	Key-value label on the registered model or on one version	validation=passed

Registering needs the CREATE MODEL privilege on the target schema, plus USE CATALOG and USE SCHEMA on its parents. Loading or serving a model needs EXECUTE on it.

Registering a model

Registration copies a logged model from a run into the registry under a three-level name. If the name does not exist, it is created; each later registration adds a new version. Set the registry to Unity Catalog first, then register from the run URI.

import mlflow
from mlflow import MlflowClient

mlflow.set_registry_uri("databricks-uc")
mv = mlflow.register_model(
    model_uri=f"runs:/{run_id}/random_forest_model",
    name="ml_prod.churn.churn_classifier",
)
print(mv.name, mv.version)

client = MlflowClient()

A logged model can also be registered at logging time with the registered_model_name argument of the flavor's log_model function. Registration can complete asynchronously, so a new version may briefly be pending before it is ready for use.

Setting and removing tags

Tags are key-value strings. Registered model tags describe the model as a whole, such as an owner or a use case. Version tags describe one version, such as whether it passed validation.

name = "ml_prod.churn.churn_classifier"

client.set_registered_model_tag(name, "owner", "growth-team")
client.set_model_version_tag(name, version="4", key="validation", value="passed")

client.delete_model_version_tag(name, version="4", key="validation")
client.delete_registered_model_tag(name, "owner")

Aliases and the champion and challenger pattern

An alias is a named pointer to exactly one version of a registered model. A version can carry several aliases, and an alias can be moved at any time. Unity Catalog uses aliases instead of stages such as Staging or Production, which do not exist in Unity Catalog.

client.set_registered_model_alias(name, "champion", 4)
client.set_registered_model_alias(name, "challenger", 5)

champion = client.get_model_version_by_alias(name, "champion")
client.delete_registered_model_alias(name, "challenger")

model = mlflow.pyfunc.load_model(f"models:/{name}@champion")

In the champion and challenger pattern, the champion is the version serving production and the challenger is a newly trained candidate. The usual sequence is:

1. Train the challenger, log its metrics, and register it as a new version.
2. Point the challenger alias at the new version and evaluate it on the same held-out data as the champion.
3. If it is better, point the champion alias at the challenger's version. Consumers that load models:/name@champion pick up the change with no code edit.
4. To roll back, point champion at the previous version. Nothing was deleted, so the rollback is one call.

challenger = client.get_model_version_by_alias(name, "challenger")
champion = client.get_model_version_by_alias(name, "champion")

new_f1 = client.get_run(challenger.run_id).data.metrics["f1"]
old_f1 = client.get_run(champion.run_id).data.metrics["f1"]

if new_f1 > old_f1:
    client.set_registered_model_alias(name, "champion", challenger.version)

Promoting code versus promoting models

	Promote the model	Promote the code (recommended)
What moves	The trained model version, copied through environments	The tested training and inference code; each environment trains its own model
Strengths	Deploys the exact artifact that was validated; no retraining needed; useful when training is costly or cannot be repeated	Models are trained on each environment's own data; the code is tested in staging; reproducible and auditable
Drawbacks	Inference and monitoring code need a separate promotion process; the environment must be able to load the artifact	More compute per environment; results can differ slightly; needs data access and pipelines in every environment
Choose when	Training data is restricted to one environment, retraining is impractical, or the exact artifact must be proven	Production must retrain on production data, models refresh regularly, or code review and testing gate every release

A model version can be copied across catalogs when a model must move from a development catalog to a production catalog. Registering from a models:/ URI creates a new version under the destination name.

mlflow.set_registry_uri("databricks-uc")
mlflow.register_model(
    model_uri="models:/dev_catalog.churn.churn_classifier@challenger",
    name="prod_catalog.churn.churn_classifier",
)

Why Unity Catalog beats the workspace registry

Capability	Workspace registry	Unity Catalog registry
Scope	One workspace	Every workspace attached to the same metastore
Names	Two-part, unqualified by catalog	Three-level catalog.schema.model
Access control	Workspace permissions	GRANT privileges on catalogs, schemas and models
Promotion markers	Stages such as Staging and Production	Aliases and tags
Governance	Limited to the workspace	Lineage to training data and centralized audit

21. MLOps Principles and Practice

MLOps is the set of practices, processes and technologies for managing data, code and models so that an ML system stays performant, stable and efficient over time. It combines three disciplines, each covering part of the work.

Discipline	Focus	Key practices
DataOps	Data processing and governance	Optimized processing, centralized discovery and governance, data quality checks, traceable lineage
DevOps	Delivering ML code	Machine learning is code, CI/CD, Git version control, production-grade scheduled jobs
ModelOps	The model lifecycle	Treating models as objects and as data, orchestration and automation of training and deployment

The ML lifecycle

A project moves from a business problem through success criteria, data collection, preprocessing and feature engineering, model training and evaluation, deployment, and monitoring. Development works with static historical data. Production works with continuously changing data, which is why monitoring and retraining belong to the lifecycle rather than being an afterthought.

Core challenges MLOps addresses

• Reproducibility and experiment tracking. Results differ between environments, and experiments are hard to document. Use tracked runs, versioned code and controlled environments.
• Model evaluation and comparison. Metric choices and evaluation designs vary, so the best model is hard to pick fairly. Fix the metric, hold out a test set and log every result.
• Deployment and standardization. Moving a model to production is complex, and each project does it differently. Use standard templates, automated pipelines and versioned, registered models.

Best practices for an MLOps strategy

• Use a data-centric approach. ML projects are made of data pipelines, which need storage, governance and orchestration.
• Keep one project code repository in Git and use it in every environment.
• Separate development, staging and production, and promote changes through them in order.
• Deploy code rather than model artifacts by default. Promote a model artifact only when the registry chapter's conditions apply.
• Automate with CI/CD and scheduled jobs rather than manual notebook runs.
• Set up monitoring, alerting and retraining triggers from the start.

Reference architecture

Infrastructure is set up once per project or team. It includes a Unity Catalog metastore, one or three workspaces, catalogs for dev, staging and prod, a Git repository, and a project template such as Databricks Asset Bundles or MLOps Stacks. Environments can be separated in two ways:

• Direct separation: a completed workspace per environment. Environments are simpler and the approach scales well to many projects.
• Indirect separation: one workspace with enforced separation. Infrastructure is simpler and needs fewer permissions, but each environment is more complex and the approach does not scale well.

Each stage has a clear owner and cadence:

Stage	Who and how often	Typical work
Development	Data scientist; initial build and each update	EDA, ML pipeline development, validation, changes committed to Git
Staging	ML engineer; every change and every model refresh	A pull request triggers CI. Unit, integration and stress tests run; merge to main only when they pass
Production	ML engineer; when changes pass tests or a model needs refreshing	A release branch triggers automatic deployment of code, jobs and compute; monitoring, drift detection, retraining

Automation and CI/CD tools

• Databricks Asset Bundles (recommended): infrastructure as code for jobs, models and pipelines, defined in YAML, versioned with the code and deployed to several workspaces with the CLI.
• Databricks CLI, SDKs, REST API and Terraform: scripted or infrastructure-driven deployment. SDKs exist for Python, Java, Go and R.
• Lakeflow Jobs: DAG-based orchestration of notebooks, Python scripts or wheels, SQL, and Spark Declarative Pipelines. Triggers include manual, cron schedule, API, file arrival, Delta table update and continuous.
• Third-party CI/CD: GitHub Actions, Azure DevOps and Jenkins are common choices. Airflow can also orchestrate Databricks work.
• Git folders: a visual Git client in the workspace for clone, checkout, commit, push, pull and branch management, authenticated with a personal access token or equivalent.

Advantages of ML runtimes

Databricks Runtime for ML is a cluster runtime with common machine learning libraries and MLflow pre-installed at tested, compatible versions. Its advantages are less setup time, fewer dependency conflicts because versions are tested together, Databricks Autologging enabled by default, GPU-capable variants for deep learning, and a consistent environment that makes results easier to reproduce across clusters. The exact package list depends on the runtime version.

AutoML

Databricks AutoML automates data preparation, feature engineering, model training and selection, and hyperparameter tuning for common problems such as classification, regression and forecasting. It generates editable notebooks, so a team gets a strong baseline quickly and can then add domain expertise.

Monitoring

Monitoring covers four areas: business requirements (the solution still meets its objectives), model performance (accuracy and efficiency over time), resource utilization (cost and SLA compliance), and traceability (audit trails and data lineage).

Drift type	What changes	Typical sign
Data drift	Statistical properties of the input data	Feature distributions shift, such as a new product line changing customer age profiles
Concept drift	The relationship between features and the target	The same inputs now lead to different outcomes, such as seasonal buying behaviour
Model quality drift	Predictive performance	Accuracy, precision, recall or F1 worsen over time
Bias drift	Outcomes across groups	Prediction accuracy diverges between groups, creating unfair treatment

Data drift and concept drift can occur together. Responses include collecting new data, retraining the model, and monitoring continuously. Retraining is triggered by a schedule (the recommended starting point), by data changes or detected data drift, by changes to model code or configuration, or by monitoring alerts. Lakehouse Monitoring can monitor tables, views and ML models, including fairness and bias. It computes metrics such as null counts, averages and quantiles over configurable time granularities and data slices, auto-generates dashboards, sends SQL alerts, and stores its results as Delta tables.

22. Genie Code Fundamentals

Genie Code is an autonomous AI partner for data teams on Databricks. It works across data engineering, data science, machine learning and dashboards. It is grounded in Unity Catalog metadata, semantics and governance, and it can plan and run multi-step workflows while monitoring and resolving issues along the way. For machine learning, it turns natural language requests into the code, tracked runs, registered models and serving endpoints that a manual workflow would build step by step.

Why a language-driven workflow

A traditional model build involves data preparation, feature engineering, training, tuning, evaluation, registration and deployment, each with its own code and tools. Genie Code offers a more flexible, language-driven alternative. The practitioner describes the goal, Genie Code generates and runs the code iteratively, and the practitioner reviews the results and steers the next step. Automated data preparation, feature engineering, training and model selection, and hyperparameter tuning produce notebooks and artifacts that can be explored and changed.

The five ML needs it covers

Tables are referenced in prompts with an @ mention, such as @flights.

Need	Example request	What Genie Code does
Data discovery	Help me find a table for investigating flight delays	Searches using lineage, descriptions and code samples as signals
Exploratory data analysis	Do EDA on @flights and identify 5 interesting insights	Analyzes the table and reports insights
Model training	Build a predictive model. Are we ready, or do we need more EDA?	Considers what robust ML needs, then builds and evaluates the model
Model deployment	Register and deploy the model on an endpoint	Registers the model, deploys it, iterates on errors and integrates with other services
Model monitoring	Create a dashboard for evaluating model performance from @tables	Creates datasets and charts automatically

Integrations

Genie Code is designed to work with the rest of the Databricks ML stack:

• Feature Store: the training dataset can be expanded using existing feature tables.
• MLflow Model Tracking: metrics and parameters of every trial run are tracked, so trials can be compared.
• Model Registry: generated models can be registered to Unity Catalog.
• Model Serving: generated models can be deployed to a serving endpoint for real-time inference.
• Surrounding services: collaborative multi-language notebooks, runtimes and environments, data versioning and monitoring, and both batch scoring (high throughput) and online serving (low latency).

The end-to-end lifecycle

1. Find the data with a discovery request, then ask for EDA on the chosen table.
2. Ask for a model. Genie Code builds it, tunes it, and logs every trial run to MLflow.
3. Compare the tracked runs and choose the candidate that best matches the business goal.
4. Register the chosen model in Unity Catalog and deploy it to a Model Serving endpoint.
5. Verify the endpoint returns predictions, then ask for a monitoring dashboard built from the tables.

Evaluating the best model from a session

Because every trial is tracked, the best model is identified by comparing the logged runs. Sort or filter by the metric that fits the problem, such as F1 for imbalanced classification or RMSE for regression. Check the parameters of the top runs, confirm the chosen run is the one registered, and confirm that the endpoint serves that version. Review the generated code too: check the train and test split, look for leakage from target-derived features, and confirm the metric matches the business objective.

23. Customizing Genie Code

Genie Code works from general knowledge by default. Customization adds task-specific context in three forms: MCP connections that give access to data and tools, instructions that set standing guidance, and skills that package reusable workflows. Each has a different scope, so the right choice depends on the kind of context being provided.

Mechanism	What it provides	When it applies	Best for
MCP connections	Access to external tools, data sources and metadata	When the agent needs that context during a task	Context that exists but is hard for an agent to reach
Instructions	Standing guidance written as markdown	Always on	Standards, libraries, style and language
Skills	On-demand bundle of workflow knowledge: markdown with optional scripts	Used in Agent mode when a task matches	A single repeatable task

MCP connections

MCP (Model Context Protocol) connects Genie Code to external tools and data sources. Use it when important context already exists but is difficult for an AI agent to access. Typical sources are documentation systems, tools and services exposed through APIs, structured data sources, and custom applications. Sources selectable in Databricks include Unity Catalog functions, Vector Search, Genie Spaces, Unity Catalog connections to external MCP servers, and Databricks Apps that serve as custom MCP servers.

Best practices: connect only the sources a workflow needs, choose the source that answers the question most directly, and record what each connection is for so that teammates know when to rely on it.

Instructions

Instructions are markdown files that give Genie Code always-on context: standards, preferred libraries, style and language. They can be configured at the workspace level, which suits team standards, or at the user level, which suits personal preferences.

Best practices: write clear, specific instructions; keep them under 20,000 characters; use headings and delimiters so the structure is easy to follow; keep the scope broadly relevant to all tasks; and provide context and references where they help.

# Team ML standards
## Language and libraries
- Use Python with scikit-learn, and PySpark for large tables.
## Experiment tracking
- Log every run to /Shared/churn-experiments.
## Registration
- Register final models to ml_prod.churn in Unity Catalog.
- Set the champion alias only after metrics are reviewed.

Skills

Skills are on-demand bundles of domain knowledge and workflow, written as markdown with optional scripts. They are used in Agent mode for task-specific procedures, examples and automation. Because they load only when relevant, they are the right place for detailed procedures that would clutter the always-on instructions.

Best practices: focus each skill on a single task or workflow; give it a clear, descriptive name; provide step-by-step examples; include only the context the task needs; iterate and refine it as the workflow changes; and separate guidance from automation, keeping the explanation in markdown and the executable steps in scripts.

# Register and promote a classifier
## When to use
Use after a training run has logged its metrics.
## Steps
1. Find the run with the highest F1 in the shared experiment.
2. Register it to ml_prod.churn.churn_classifier.
3. Point the challenger alias at the new version and report its metrics.

Choosing the right mechanism

• A standard that should always apply, such as a naming rule or a preferred library, belongs in instructions.
• A repeatable procedure that applies only to certain tasks, such as registering and promoting a model, belongs in a skill.
• Knowledge that lives in an external system, such as documentation or a governed table, belongs behind an MCP connection.

Choosing the wrong mechanism causes problems. A rule buried in a skill is missed when the task does not match. A long procedure in instructions consumes context on every task.

End-to-end use

1. Write the team's standards as workspace-level instructions.
2. Connect the MCP sources the team relies on, such as a Genie Space holding the data dictionary.
3. Add a skill for each repeatable workflow, such as training and registering a classifier.
4. Prompt Genie Code through the workflow in natural language: EDA, training, tuning, and logging trial runs.
5. Review the tracked runs, register the chosen model to Unity Catalog, and move the champion alias after review.
6. Deploy the model to a serving endpoint and verify that it returns predictions.