Level Up Your Synapse Game - CI/CD Automation for a Faster, More Reliable ELT Process

Welcome to the April edition of our MVP series! This month, Ryan is joined by Kamil Nowinski, Data Platform MVP from the United Kingdom. Kamil will show us an end-to-end example of building CI/CD for Azure Synapse artifacts deployment.
00:08 Ryan's introduction
00:39 Kamil's introduction
01:26 Deploying a Synapse Workspace
15:55 Azure Data Factory pipelines
17:24 Using an existing Synapse Workspace
21:22 Wrap-Up
Synapse workspace deployment (Azure DevOps extension):
marketplace.visualstudio.com/...
Kamil's Twitter: / nowinskik
Kamil's Blog: azureplayer.net/
Kamil's TH-cam: / @kamilnowinski
More about #adftools:
azureplayer.net/adftools/
github.com/Azure-Player/azure...
marketplace.visualstudio.com/...

[@JaspreetKaur-bp4ru]

Is It possible to deploy selective files? For example there are so many folders and files in synapse workspace, and I only want to deploy specific pipeline, notebook present in a folder. Is it possible? if yes, where to mention this path for selective files that I want to deploy, in yml file. Kindly assist. Thanks in advance.

[@JaspreetKaur-bp4ru]

when we select source branch, is there any option to select only specific folder from that source branch to be deployed?

[@keen8five]

Why is the disabling and enabling of triggers not done automatically by the Synapse workspace deployment extension? What is the scenario you don't want this to happen automatically?

[@KamilNowinski]

This is very good question. Also IMO it should be configurable (on/off/definition) in one task. I will share the feedback with Product Group during MVP Summit this week.

[@anthonysalazar5143]

@@KamilNowinski Hello, excellent video. I have a question, why you don't call the workspace_publish branch templates directly, by pressing the Publish button in the Synapse environment ?

[@KamilNowinski]

@@anthonysalazar5143, clicking "publish" button is a manual deployment method, whereas here I wanted to present how to automate this process. In this video you see the deployment to the same (DEV) environment, but adding another stage you can do the same for higher envs like TEST, UAT, PROD.

[@Multidimensional.Analytics]

MVP should always equal = Minimum Viable Product. There's no "most valuable person". I developed a billion dollar company's pricing mainframe over 7 years. As a multidimensional ssas cube top tier Business Intelligence Analyst (SSIS, SSAS, SSRS, Power BI, synapse CI/CD or anything else the Mainstream establishment releases.), I never want to hear MVP in any other context.
Our industry is obscene. The amount of money we earn as consultants, developer etc is obscene. $110 per hour as a Dev minimum. Recalibrate the whole system.
Us as Analysts can recalibrate macro economics. We can legitimately cure hunger and housing globally in a instant.
We as analysts are powerful. There is no "most important person". If every Analyst, Data Analyst, Data Report Analyst, Business Intelligence Analyst, Business Analysts, Data Engineer, Software Dev, Web Dev et al globally said "lets correct poverty across the globe", we as analysts could functionally do it.

[@Multidimensional.Analytics]

Have to approve my comment lol. I am the only person out of billions that has this handle. Think about it. I'm Jo

[@Multidimensional.Analytics]

Most people can't think in 5, 6 dimensions at a time. I can

[@Multidimensional.Analytics]

5,6,7 etc. They all lead to the same variable outcome. We as a species are a closed system

[@Multidimensional.Analytics]

MVP should always equal = Minimum Viable Product. There's no "most valuable person". I developed a billion dollar company's pricing mainframe over 7 years. As a multidimensional ssas cube top tier Business Intelligence Analyst (SSIS, SSAS, SSRS, Power BI, synapse CI/CD or anything else the Mainstream establishment releases.), I never want to hear MVP in any other context.
Our industry is obscene. The amount of money we earn as consultants, developer etc is obscene. $110 per hour as a Dev minimum. Recalibrate the whole system.
Us as Analysts can recalibrate macro economics. We can legitimately cure hunger and housing globally in a instant.
We as analysts are powerful. There is no "most important person". If every Analyst, Data Analyst, Data Report Analyst, Business Intelligence Analyst, Business Analysts, Data Engineer, Software Dev, Web Dev et al globally said "lets correct poverty across the globe", we as analysts could functionally do it.