Using External Tables in SQL Server Hybrid Cloud Deployments

Hugo Queiroz, Moiz Kohari

SQL Server 2022’s support of S3 protocol enables two key enterprise features, backup/restore and external tables residing in MinIO based object storage. In this presentation we will highlight how clients in key sectors like automotive, finance and government are ingesting large data sets into MinIO and then are able to run analytics on this newly ingested data. While there are many methods to populate data into a MinIO S3 bucket (such as Kafka, NiFi and Spark) this presentation will cover data movement using SFTP. FTP/SFTP provide barebones simplicity and are supported in many highly constrained environments making it a de facto data movement tool in legacy deployments.
In this presentation you will learn how to:

1. setup a secure ftp access to a MinIO bucket and move a parquet file to this bucket
2. Receive a lambda notification when a new parquet file has been uploaded to MinIO
3. Using SQL Server 2022’s external table functionality initiate queries to this newly populated parquet file and run different analytic jobs