Databricks ML Associate Course Notes

Databricks machine learning work sits on the lakehouse pattern: data is stored in open Delta tables while teams use SQL, Spark, notebooks, MLflow, Feature Store, and governance tools around the same data. The exam often tests why this matters: fewer data copies, shared metadata, and a cleaner path from exploration to production.

The MLflow section focuses on recording and analyzing data associated with model training runs. Specifically, it details how to log input data using `mlflow.log_input()` to capture datasets and configuration parameters during experiments. This data is then tracked alongside model metrics, enabling reproducibility and performance analysis. The goal is to establish a comprehensive record of the data used in each run for debugging and comparison purposes.

Data processing starts by checking whether the dataset is usable for modeling. Descriptive statistics, missing-value counts, distributions, outliers, and class balance help reveal whether a column needs imputation, transformation, encoding, or removal before training begins.

Model development in Databricks usually starts with a training pipeline: load prepared data, transform features, fit an algorithm, evaluate results, and record the run. Spark ML pipelines are useful because they package preprocessing and model training into repeatable stages.

Model governance begins when a trained model becomes a managed asset instead of a file inside a notebook. Databricks model lifecycle management tracks model names, versions, aliases, metadata, permissions, and lineage so teams know which model is approved for each use case.

Model deployment turns a trained model into a repeatable inference process. Databricks supports both batch inference, where many records are scored offline, and online inference, where applications send requests to a serving endpoint for low-latency predictions.