39. How to Capture Changed Data using Data flows in Azure Data Factory

In this video, I discussed about how to capture changed data using data flows in Azure Data Factory
Link for Azure Synapse Analytics Playlist:
• 1. Introduction to Azu...
Link for Azure Databricks Play list:
• 1. Introduction to Az...
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
Link for Azure Data Factory Real time Scenarios
• 1. Handle Error Rows i...
Link for Azure LogicApps playlist
• 1. Introduction to Azu...
#Azure #AzureDatafactory #DataFactory

[@kishoreevani5813]

thank you so much for sharing this. Please keep doing more on ADF.

[@WafaStudies]

Sure. Thank you ☺️

[@antonyaswin1176]

I am implementing this , but will this be cost efficient for the system ? We know has to take only those delta data or the ones to be updated alone .. But does it have to select /scan the entire data in the beginning ?

[@CoolGuy]

This is similar to 35th video of the same playlist. Please correct me if I am wrong.

[@james30321]

Yes it's same

[@KrishnaGupta-dd1mo]

Informative... 🙂

[@WafaStudies]

Thank you 😊

[@sravankumar1767]

Nice explanation 👌

[@WafaStudies]

Thank you 😊

[@chriscusumano8210]

Very informative - well done!

[@WafaStudies]

Thank you 😊

[@tejaswinibellam3528]

Is it possible to copy from postgres to postgres

[@ViralDave_26]

can we do the same in Azure synapse ? if yes please prepare a demo

[@rohitkumar-it5qd]

How do we merge this changed data at the destination without having duplicates.. after the steps at this video??

[@shreeyashransubhe2537]

Thank you for great information. In an interview I faced one question: If there are 10 files in source and we use copy activity to copy files to destination and if 6th file fails, so on next pipeline run how can we start the copy activity from failed 6th file and skip all previous files till 5th. Thanks is advance.

[@shuaibsaqib5085]

re run from failed activity option is available under monitor tab in ADF.

[@sakshiagarwal9596]

Hi sir what if the file has 1000 of data in both source 1 and source 2 i am getting fisrt 100 data with different set of data due to which it is showing mismatch for all

[@ravisankar9453]

good scenario 🤗

[@WafaStudies]

Thank you ☺️

[@shreyashchoudhary6827]

One scenario trick one : increment copy data from onprem db by checking we are only copying the data which is not previously copied

[@pratikpatel3887]

This will still run at the trigger so It can't be real time. isn't it right?

[@balajikomma541]

Sir please make a video how much ADF knowledge is required/sufficient for power bi developer. Thank you sir

[@rayavaramprasad362]

Namaste sir

[@WafaStudies]

Namaste

[@rayavaramprasad362]

Your videos watching daily sir
I like your explanation sir

[@tejaswinibellam3528]

How to replace old data of table at si nk side with new data. Any replace function is there

[@CoolGuy]

We can set up incremental delta load pipeline based on " Incremental Behaviour" = LastModifiedTime

[@vigneshr8956]

Hi maheer ,
I want to copy 100 files from a folder. But the pipeline fails after copying 50 files. Now I want to rerun the pipeline but I should only copy the 50 files which are not already copied. Can you help me with this functionality, I couldn't find this in your real time scenario playlist.
Thank u.

[@ybharathchand]

Foeach iterate through files and perform copy activity - after each successful operation update the counter variable and write the value to a lookup file,
So, include a lookup activity before Foeach to see the latest on the file.. If it is first time the value is 0, if the operation broke on 50th file.. the value is 50.. So copy activity happens only if the counter is greater than the lookup value.. Hope that clarifies.

[@vigneshr8956]

@@ybharathchand thanks bud

[@phanikumar6107]

please do videos on sql server

[@phanikumar6107]

PL SQL