Implement CI/CD with Azure DevOps
Introduction Imagine you are part of a team of data engineers who are collaborating on one or more notebooks within a development environment, using Azure Databricks. When you are ready to deploy your changes to production, you must coordinate with an operations team to copy the notebooks over to a production Azure Databricks workspace since company policy dictates that you and your team are not allowed to manually copy the changed files over. This process causes bottlenecks and extra work. Leadership has asked you to find an automated process that incorporates version control, automated testing capabilities, and controls for deployment approvals if needed. Automated testing and deployment is a common practice in software development. However, those same principles also apply to data engineering and data science. Data engineers and data scientists need to collaborate on parts of the system and be able to deploy to production without constantly relying on opera...