Testing your Data Factory

Benjamin Kettner, Frank Geisler

When building pipelines in Azure Data Factory or Synapse Analytics, the Data Factory UI has an excellent Git integration that enables developers to create branches, commits and pull requests directly from the development environment. Compared to the very basic version control capabilities of SSIS, this was a quantum leap, however it is not the end of the story. Implementing CI/CD processes will improve not only the time to delivery but also has the potential to significantly increase the quality of your solution and the confidence of your users in your deployment.
However, having version control in place is only the first part of the CI/CD story, if you want to build your cloud-borne data integration solution adhering to CI/CD principles, you will need to come up with a suitable testing strategy for your solution.
In our session we will demonstrate how to build a test environment for Azure Data Factory, what tools you can use to test your data integration logic and how you can integrate these tests into your pipelines and make the results visible. Participants should bring a basic understanding of Azure Data Factory. From this session you will take the basic concepts of CI/CD and test strategies as well as a working knowledge to start implementing these processes in your own environment.