Building reliable ETL pipelines is always a challenging task for Data Engineers. And complex business scenarios make it difficult to maintain, test, and automate the ETL pipelines.
Delta Live Tables (DLT) is the new shiny framework in the Azure Databricks ecosystem. And it is built on top of Delta Lake. It can help you to build a series of separate Apache Spark tasks as a pipeline, each having its own business logic and data expectations defined. This not only helps in automating the pipeline, but maintains data quality all the way.
In this session, we will first see what Delta Lake is and how it works. Then we will discuss what Delta Live Tables are, and the components. Next, we will go through the process of building a multi-step, reliable ETL pipeline with Delta Live Tables. And finally, we will automate these pipelines.
So let’s get started!
Sign up to stay up to date with news, special announcements and educational content.
Redgate will only contact you about PASS Data Community Summit (in line with our Privacy Policy) unless you separately request emails about Redgate. You can unsubscribe from these updates at any time.
Summit video library