Loading video player...
Learn how to implement a complete CI/CD pipeline for Databricks using Databricks Asset Bundles (DABs) and Azure DevOps. This hands-on tutorial walks you through the entire workflow from local development to production deployment with automated validation and service principal authentication. šÆ What You'll Learn: ⢠Set up Azure DevOps pipelines for Databricks workflows ⢠Configure Databricks Asset Bundles for multi-environment deployments ⢠Implement OAuth service principal authentication ⢠Create automated dev and prod deployment workflows ⢠Validate bundles before production deployment ⢠Use pull requests for production promotion š Topics Covered: 0:00 - Introduction & Architecture Overview 1:30 - Azure DevOps Setup & Project Configuration 3:15 - Variable Groups & Environment Configuration 5:00 - Service Principal Creation & Authentication Setup 7:20 - Workspace Permissions & Prerequisites 8:45 - Repository Structure & Asset Bundle Configuration 11:30 - databricks.yml Configuration for Multi-Environment 14:00 - CLI Profile Setup for Dev & Prod 15:30 - Azure Pipelines YAML Walkthrough 18:00 - Validation Stage for Pull Requests 19:30 - Deploy to Dev Branch Demo 22:45 - Testing Bundle Deployment in Dev Workspace 25:00 - Making Changes & Validating Locally 27:30 - Automated Dev Deployment Trigger 29:00 - Creating Pull Request to Main Branch 31:15 - PR Validation & Approval Process 33:00 - Deploy to Production Workspace 35:30 - Verifying Production Deployment 37:00 - Recap & Key Takeaways š Key Features Demonstrated: ā Automated deployment on push to dev branch ā Bundle validation on pull requests ā Separate dev and production environments ā Service principal-based authentication ā No manual intervention required ā Version control with Git branching strategy š” Prerequisites: - Azure DevOps account (dev.azure.com) - Two Databricks workspaces (dev and prod) - Databricks CLI installed locally - Service principal with workspace access - Basic Git knowledge š Useful Resources: - Databricks Asset Bundles Documentation: https://docs.databricks.com/dev-tools/bundles/ - Azure DevOps Pipelines: https://learn.microsoft.com/en-us/azure/devops/pipelines/ - Databricks CLI: https://docs.databricks.com/dev-tools/cli/ - Service Principal Setup: https://docs.databricks.com/dev-tools/auth/oauth-m2m.html - Databricks Asset Bundle Reference: https://docs.databricks.com/dev-tools/bundles/settings.html š Sample Configuration Files: The demo includes complete examples of: - databricks.yml for multi-environment configuration - azure-pipelines.yml with validation and deployment stages - CLI profile setup for local development šļø Architecture Flow: 1. Local Development ā Dev Branch 2. Push to Dev ā Auto-deploy to Dev Workspace 3. Create PR ā Validation Stage 4. Merge to Main ā Auto-deploy to Prod Workspace š„ Perfect for: - Data Engineers implementing CI/CD for Databricks - DevOps teams managing Databricks deployments - Platform teams standardizing deployment processes - Anyone looking to automate Databricks workflows š¬ Questions or feedback? Drop them in the comments! š Subscribe for more Databricks tutorials, CI/CD best practices, and data engineering content! #Databricks #CICD #AzureDevOps #DataEngineering #AssetBundles #DevOps #DataPipelines #MLOps #Automation