Part 2- Environment Setup - Integrating Azure DevOps GIT Repository to Azure Databricks for CI/CD

#cicd #azuredevops #devops #databricks #azuredatabricks
In this Part 2 video of CI/CD in Azure Databricks, I have discussed about the different environment setup required for building the CI/CD pipelines. Things like Integrating the Azure DevOps with Azure Databricks workspace, Branch protection and merging techniques are seen in this video with Demo.
The next parts will be uploaded soon, stay tuned!
Part #3. Creating CI Pipeline
Part #4. Creating CD Pipeline and Testing
Here’s what you’ll see in this video:
0:00:00 - Intro
0:02:10 - Creating Azure DevOps Git Repository
0:05:51 - Git Integration in Azure Databricks
0:11:23 - Branch Protection
0:20:13 - Merging the changes to Main
0:28:00 - Outro
- - - Book a Private One on One Meeting with me (1 Hour) - - -
www.buymeacoffee.com/mrktalks...
- - - "Express your encouragement by brewing up a cup of support for me" - - -
www.buymeacoffee.com/mrktalks...
- - - Other useful playlist: - - -
Azure Data Factory Playlist: • Azure Data Factory Tut...
Azure General Topics Playlist: • Azure Beginner Tutorials
Microsoft Fabric Playlist: • Microsoft Fabric Tutor...
Azure Databricks Playlist: • Azure Databricks Tutor...
Azure End to End Project Playlist: • End to End Azure Data ...
Databricks CICD Playlist: • CI/CD (Continuous Inte...
End to End Azure Data Engineering Project: • An End to End Azure Da...
- - - Let’s Connect: - - -
Email: [e-mail]
Instagram: mrk_talkstech
- - - Tools & Equipment (Gears I use): - - -
Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!
DJI Mic: amzn.to/3sNpDv8
Dell XPS 13 Plus 13.4" 3.5K : amzn.to/45KqH1c
Rode VideoMicro Vlogger Kit: amzn.to/3sVFW8Y
DJI Osmos Action 3: amzn.to/44KYV3x
DJI Mini 3 PRO: amzn.to/3PwRwAr
- - - About me: - - -
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst
3. Data Scientist
Can't wait to help people with my videos.
- - - Support me: - - -
Please Subscribe: / @mr.ktalkstech
#AzureDatabricks #ApacheSpark #Sparkcompute #clusters #notebooks #magiccommands #machinelearning #ETL

[@mr.ktalkstech]

I hope you all enjoy this video, I have spent multiple days in making this tutorial since my main intention is to cover almost all the topics in the most detailed way so that even an absolute beginner can understand the complete functionality of the CI/CD pipeline.
I would highly appreciate if you could give a Like to this video and subscribe to my YT channel which will motivate me to make more such videos in the future.
To get the YAML files used in this tutorial, please do the below as a token of support :)
1. Please follow my Instagram channel : mrk_talkstech (and)
2. Send me an message in Instagram with your email asking for the YAML code ( I will send you the files) ***Please Follow****
What else should I cover in my upcoming videos? Please let me know in the comments :)

[@deepanshujain4119]

Hello Mr K - Amazing videos. You have really nailed the art of teaching someone who is a beginer to data engineer. Would you be posting the video on how we can include integration tests in CI part as well?

[@TrinadhKumarMadupu]

Very Clean Explanation, Thanks. Eagerly waiting for the next part...

[@sharaniyaswaminathan8760]

Clearly explained. Thank you so much. Waiting for the next part.

[@mr.ktalkstech]

Thank you so much :)

[@sajadbinnazir2606]

Well explained , Waiting for part 3rd ..... Thanks

[@yahiachammemi8267]

I Highly recommend this playlist
Great contents !

[@mr.ktalkstech]

Thank you so much :)

[@CharanSaiAnnam]

thanks

[@jatinmadan8489]

Hi Mr K @mr.ktalkstech, I really enjoy your videos, but I am unable to generate the git credentials for azure repos git using azure devops.
Problem: While creating a new repo, I am not able to see the advanced setting to setup the git credential to Azure DevOps Services (Azure Active Directory). Can you please help me with the same.

[@RahulKumar-dy4gr]

Can you help us in estimating the full project cost in Azure...
I mean how to estimate the expected cost with respect to resources ? In data engineering projects 🚩🚩🚩🚩

[@destroyer3706]

very good explaination. I have one question. Where is the CD part? If i am not wrong, you have created repository in Azure devops and connect it to Databricks dev space. THen you pulled request and merge into main branch. When the Production environment will get updates? or files from dev to prod environment when this will happen.?

[@mr.ktalkstech]

Those will be uploaded in the future parts of the video :)

[@hritiksharma7154]

Hi Sir,
I have one doubt if someone what to create a new cluster in a workspace how he can create it by following devops process and what if unity catalog is also enabled in the workspace.

[@mr.ktalkstech]

Creation of the clusters and resources will be mostly done via templates like terraform or biceps, etc

[@anupshastry5969]

Do we need knowledge of devops to be data engineer

[@mdsultanahemad]

Hi if you could help me, I am trying to create repo . But I am not getting any option to add GIT credentials because of which I m not able to proceed. If you could help me ?
I am getting only 3 options git repository url ,git provider and repository name

[@jatinmadan8489]

Facing the similar issue, any update on it ?

[@mdsultanahemad]

@@jatinmadan8489 no bro

[@sm.prithiv7174]

Hi sir I am from banking domain planning to switch to data engineer. Your videos are very useful for my journey i am preparing for dp 203 exam. I got to know from mybfriend that in realworld we dont get access to azure portal. We dont built pipelines in azure console. Everything we do in IAC. So do i need to study teraform or template in azure. Please clarify me. Thanks in advance

[@deepakchadda]

What's your approach for certification preparation?

[@sm.prithiv7174]

@@deepakchadda i started doing your end to end project watched all your tutorials now i am reading microsoft documentation and book azure data engg cert by newton Alex and some youtube tutorials. I am aware of sql, python, spark and hadoop components like sqoop hive.

[@mr.ktalkstech]

Yup, mostly in real-time projects, all the resources will be created via templates like terraform, biceps, etc. But that is not the skillset needed for a Data Engineer- it's more like a devops engineer / network engineer work.