Best Practices for Development on Azure Databricks

Dustin Vannoy

The Databricks Data Intelligence Platform offers a suite of capabilities to empower data teams to build high-quality data pipelines and AI applications. While Databricks simplifies development with many easy-to-use tools, it’s still important to follow software engineering best practices. The practices covered in this session will help you ensure the data pipelines and AI-driven processes are reliable and accurate.

In this session, we’ll explore how Azure Databricks features support best practices like code reuse, version control, automated testing, and automated deployments. During our talk, we will delve into the new PySpark native test framework, Databricks Repos, Databricks Asset Bundles, the VS Code extension, Databricks Connect, and GitHub Actions/Azure DevOps.

We will demonstrate how to use these tools together to improve the development and testing of data pipelines and applications on Azure Databricks.