Databricks Data Engineer Associate Course Notes

Architecture + compute selection. Separate control/compute/governance concerns and choose compute from workload constraints.

• Platform architecture, Delta Lake, and Unity Catalog: Do not describe Unity Catalog as the compute plane or Delta Lake as the control plane.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.

Classify ingestion pattern, then choose COPY INTO, Auto Loader, Lakeflow Connect, partner, or custom clients based on source, scale, frequency, and governance.

• Batch, streaming, incremental, and Lakeflow Connect ingestion: Do not equate incremental with continuous; incremental describes what changes are processed, not necessarily how continuously it runs.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.

Build trusted Silver/Gold data with explicit cleaning, DataFrame semantics, correct grain, measured tuning, fit-for-purpose Gold objects, and quality checks.

• Clean Bronze into trusted Silver: Job succeeded != data is correct.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.

Model a real DAG, use bounded retries/loops/branches, and choose triggers from the business condition.

• Retries, run-if conditions, branches, and loops: Do not retry the entire job when only one failed task needs a retry if successful work can be preserved.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.

Use Git folders for reviewed source development and Declarative Automation Bundles for reusable, target-aware deployment.

• Git folders: branches, commits, pushes, pulls, and PR workflow: Do not treat a local commit as a remote change.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.

Use history and stage-level evidence before tuning; understand liquid clustering/predictive optimization and classify startup/library/OOM failures.

• Job run history and performance baselines: Do not optimize the fastest task first just because it is easy.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.

Understand managed/external lifecycle, least-privilege hierarchy, fine-grained row/column controls, and ABAC at scale.

• Managed vs external tables and lifecycle: Do not say “external tables are not governed by Unity Catalog.” They are governed for metadata/access, but file lifecycle is external.

• Use the scenario requirement as the decision rule; eliminate choices that violate feature, scale, governance, or lifecycle constraints.