12.Continuous integration and delivery in azure data factory | CI/CD in ADF explained
ฝัง
- เผยแพร่เมื่อ 5 ต.ค. 2024
- In this video I have showed how to set up CI/CD release management in azure data factory with ARM template in detail. It also includes
Explanation of agent pools in Azure DevOps,
Configuring continuous deployment trigger ,
Adding scripts for stopping the trigger before deployment and starting the trigger after deployment(Microsoft Recommendation)
Prerequisites(Watch) :
Call API from ADF
• 10.ADF | Learn how to ...
Git setup in ADF
• 11.Configure GIT in Az...
Reference URL :
Microsoft documentation with trigger related scripts
docs.microsoft...
Free Azure DevOps agent pool request:
forms.office.c...
Follow me on Udemy
www.udemy.com/...
Follow me on LinkedIn
www.linkedin.com/in/ramkumar-rajendran
Learn Import ARM template(Manually)
• 9.Azure Data Factory |...
Playlist
Azure tutorial :
• What is Azure Resource...
AzureData factory Tutorial :
• Azure Data Factory
Git and Git Hub Tutorial :
• What is Git | version ...
Azure DevOps tutorial :
• Video
![[LIVE] : ONE ลุมพินี 82 วันนี้!! คู่เอก | ยอดไอคิว vs อับดัลลาห์](http://i.ytimg.com/vi/hSsyTl_9bqs/mqdefault.jpg)
The video was really helpful. Thank you for the detailed explanation.
Happy that you liked it
the best thing in youtube is those indian videos ! we can only LEARN !
Much required video. Thank you, and Please keep making such real-time use cases.
I'm happy that you liked the content, sure I will make similar real time use case videos
It was very nice video with good explanation... Keep it up.
Thank you ☺️
Nice explanation! Could you make one about how to use cd/ci in Databricks?
Nice video, very useful info
Thank you
nice one! thank you!
Hi ,Thanks for the wonderful video. Here in this, you have created ADF and Storage manually in Prod Env... and the deployments is through CI CD. cant we do the ADF and Storage creation in prod also through Pipeline ? this way, they will have a same setup between env Dev and prod.... if so how do we go about this ? Also if possible, can you a do a video on using YAML for deployment ?
You can have arm template of storage account and adf in the repository and use same arm template deployment in same pipeline
Though this is the most easy approach to begin with, i would opt to use yaml based approach and without using adf_publish branch. That way each commit will be deployed directly to df
Thanks for the information ✌
Good
Excellent sir,
Thank u for providing great tutorials...
Sir, I am stuck in release process where prod adf pipelines are having different triggers(schedule, event based).. And if I do release I need to disable them first.. which I really don't want .. please, give a suggestion to overcome this 💐🍁🧚
Then trigger your deployment when there is no ADF schedule
@@CloudTechRam thank u sir for replying .. Actually, getting that window is getting very difficult.. so, I was wondering if there is any way .. to tackle this.. Thank u for helping me 💐💐💐
hey bro, if we add new storage account, which has linked services for new pipelines, do we need to add those storage account's connection string as well in override template parameters of artifact?? plz do reply
I need the pre and post deployment script, The Micorsoft doc I have gone through but the script is not available. could you plz share with me the Script path scrips.
After adding stop and start trigger.. I am facing this error "Cannot index into a null array."? what does that mean? How to manage?
Could be issue with argument script that you are passing, please double verify them
@@CloudTechRam I Guess the argument script isn’t working
any idea what should be the issue ? can you please let me know if you find . i am also facing same issue ..
What is the roadmap for being an etl developer with azure or azure data engineer?
1.learn azure fundamentals AZ 900
2.learn SQL
3.ADF and synapse
4.data bricks
5.complete certification on dp 203
@cloudTechAcademy
If multiple use cases were present in SIT or QA and one of them got approved for prod deployment. How do you make sure only one use case related adf pipeline goes to prod?
Create different branches for each features
Will it debug if we try in prod datafactory...??
Yes
Can i know y are we stopping the trigger ? And when we are manually clicking publish manually it is not called Ci ryt ?
Say for example I have several new changes in dev and release pipeline take 5 to 10 minutes to deploy to prod,, in the mean time if there are any schedule it may fail or run may cause data inconsistency as we are in between deployment, so it is good to stop trigger before deployment.
And coming to your second question by CI/CD automated deployment means devops automatically picks up code from repos and push to respective environment, we need to say adf when it have to push code/logic to repos . once we are done with full implementation and testing in dev, we click publish in adf. Post that CI/CD will happen.
@@CloudTechRam automated publish in cicd will work if i dont need any maual publish . Have you tried the deletes.. i mean if i delete a pipeline in dev i dont see the same in test any idea on that ?
Seems strange, it should work, I will give a try
@@CloudTechRam yeah pls do and let me know if any solution
I got an error saying that "No hosted parallelism has been granted" while running the pipeline.
How can i resolve that?
It is a issue with agent pool, have you requested for free azure devops agent pool?
@@CloudTechRam Yeah, Just filed that form and submitted.
Okay wait for 3 business days for free agent pool activation, or incase if you want it urgent create a self hosted agent pool using your own machine.
@@CloudTechRam
Any video made on that self hosted thing?
No not yet
After adding stop trigger and tried to release i am facing this error "Could not find the modules: 'Az.Accounts' with Version: ''. Can you please help?
I hope you are using self hosted run time, if so the az.account module which you are using could be beta/preview version, try to find the current module version in local powershell and switch to stable older stable version
stackoverflow.com/questions/59681516/could-not-find-the-module-az-accounts-with-given-version-error-when-running-az
@@CloudTechRam please make a video considering all these situations while performing with self hosted agents
Sure
@@CloudTechRam thanks a ton
This is like the old way of doing CI/CD. Microsoft has a newer approach.
Please share relevant documentation link it would be helpful