Databricks ML Associate Course Notes

Databricks machine learning work sits on the lakehouse pattern: data is stored in open Delta tables while teams use SQL, Spark, notebooks, MLflow, Feature Store, and governance tools around the same data. The exam often tests why this matters: fewer data copies, shared metadata, and a cleaner path from exploration to production.

• Reducing the lakehouse to lineage only, when the exam expects notebooks, clusters, MLflow, Feature Store, AutoML, Unity Catalog, and reproducibility to work together.

• If the scenario asks how teams share governed data and ML assets, think lakehouse plus Unity Catalog rather than a separate copied ML dataset.

The MLflow section focuses on recording and analyzing data associated with model training runs. Specifically, it details how to log input data using `mlflow.log_input()` to capture datasets and configuration parameters during experiments. This data is then tracked alongside model metrics, enabling reproducibility and performance analysis. The goal is to establish a comprehensive record of the data used in each run for debugging and comparison purposes.

• Calling every logged item an artifact; parameters, metrics, inputs, artifacts, and model outputs are tracked differently.

• If the scenario asks how to compare training attempts, look for MLflow experiments and runs.

Data processing starts by checking whether the dataset is usable for modeling. Descriptive statistics, missing-value counts, distributions, outliers, and class balance help reveal whether a column needs imputation, transformation, encoding, or removal before training begins.

• Using descriptive statistics as if they transform data; they diagnose distributions, missing values, and outliers before transformation.

• When missing numeric values are the issue, Spark ML Imputer is the clue.

Model development in Databricks usually starts with a training pipeline: load prepared data, transform features, fit an algorithm, evaluate results, and record the run. Spark ML pipelines are useful because they package preprocessing and model training into repeatable stages.

• Confusing Spark ML estimators and transformers; estimators learn from data, while transformers modify DataFrames or produce predictions.

• If the workflow chains preprocessing and training stages, think Spark ML Pipeline.

Model governance begins when a trained model becomes a managed asset instead of a file inside a notebook. Databricks model lifecycle management tracks model names, versions, aliases, metadata, permissions, and lineage so teams know which model is approved for each use case.

• Treating the model registry as plain file storage instead of a governed lifecycle record with versions, aliases, permissions, and lineage.

• If the question asks who may register, load, or manage models, look for Unity Catalog privileges.

Model deployment turns a trained model into a repeatable inference process. Databricks supports both batch inference, where many records are scored offline, and online inference, where applications send requests to a serving endpoint for low-latency predictions.

• Using online serving for scheduled bulk scoring when batch inference would be simpler and cheaper.

• If predictions are needed immediately by an application, choose online inference through a serving endpoint.