CI/CD on Databricks: Declarative Automation Bundles (DABs)
CI/CD on Databricks centers on automating the testing and deployment of data pipelines, notebooks, and jobs across environments using Declarative Automation Bundles (DABs) — formerly known as Databricks Asset Bundles — as the primary infrastructure-as-code tool, with everything version-controlled in Git. The rename (March 2026) reflects that bundles now cover full declarative automation workflows, not just “assets”; it’s non-breaking, so existing databricks.yml files and CLI commands keep working as-is.
- CI covers linting, unit testing (pytest or Nutter), and bundle validation on every code push.
- CD automates promotion from dev → staging → prod using platform tools like GitHub Actions or Azure DevOps.
Credentials are managed through service principals and Databricks Secrets — never hardcoded — and Unity Catalog provides clean environment separation at the data layer.
Key best practices
- Idempotent jobs
- Separate workspaces per environment
- All deploys run through the pipeline (never from a local machine)
- Post-deploy observability via job run history, Delta table history, and external monitoring tools
The recommended modern stack combines GitHub Actions + DABs + Unity Catalog + MLflow for end-to-end automated delivery of Databricks workloads.
Declarative Automation Bundles (DABs)
The recommended modern approach — replaces legacy databricks-cli deploys.
- Version-controlled as YAML in your repo
- Define everything in databricks.yml
- Supports multi-environment targets (dev, staging, prod)
- Deploy: databricks bundle deploy –target prod
- Run: databricks bundle run <job-name>
- Supports: Jobs, DLT/Lakeflow pipelines, notebooks, Python wheels, Model Serving endpoints
What’s new: the default bundle templates (e.g. default-python) now follow Spark Declarative Pipelines on Lakeflow conventions — defining pipelines as plain source files plus pyproject.toml, rather than notebooks. This pushes bundle-based projects further toward standard software engineering practice (testable modules, dependency management) and away from notebook-driven pipeline code.
Sources: