วีดีโอ

Hello i need help - my data sources are APIs from BC...so what I am supposed to do with those datasets?....copy them into blob in order to filter by last modified date back in ADF? That makes no sense what about copying IN to the blob? I just have to copy the whole fact table over everytime? That cant be right. All these methods of incremental copy seem to start with a csv from blob storage. But how do I even get to that point?

so what's the main take-away?

you should explain from very begining like when you created table in your datasource, nice

Could you please add a full load activity

Excellent approach, Please explain adequately to learners.

should explain how you are updating the table tbl_control

Thank you. This was helpful.

Hi sir.. where can i get this code? github link is not working

This is one of the best videos to understand the end to end data pipeline in azure. Perfect

What is difference between MANAGED delta table and EXTERNAL Delta table in azure databricks? Can we do insert , delete , update in both the types.

If a delta table is created on top of mounted datalake location , then can we do update , delete and insert to that delta table?

Hello thanks for amazing session.....just one query if data is having more than 5k lines will it work ?

7 mins video

Avo and orc both for Hadoop?

Nice explanation can you share these ppt

Nice can you share these presentation

Can you share these note book

Hi Cloud Guru! First of all, thanks for the clear explanation. It really helped me in creating a incremental data load in ADF. Never the less I was wondering if you could make a video of making the tables which should be synced incremental more dynamic. Now I've hard set in the lookup action that it should look for a specifc table. I've more tables which should be synced incremental, so I assume that should start working with an iterate activity like a for each(?). So that in a specific SQL table I've defined which table(s) should be synced incremental and that it does that automatically. Do you have any ideas of tips and trics for things like this? Thanks in advance!

Could you please share your email id, I have confusion about one of the question was asked me in interview

Hello Sir. I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table and store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

❤Thanks …it’s really informative. Could you plz share end to end pipeline with code of this project…

Can you let me know which tool will be used to create the diagram u are explaining

Great and complete tutorial that I have seen in TH-cam. Thank you

is it possible to add theses incrementals on parquet files on adls?

Please can someone please assist me on how I'm not able to do a Copy Activity to Stored Procedure on MySQL table in sink setting. Please help. Thanks

You didnt share the XLS window

why do we need control table for the last update time - since we can get it info in destination table

Please share the link of playlist

Hi sir, in my source path i have a files i have created a shedule trigger to run every 1 hour. My issue is from source to sink the files are getting triggered duplucates files Eg : last hour i have 10 files trigger nxt in my source path i recived 5more files. When it trigger the files of last 10+5 files are getting to the sink path

Great video! Have you ever set up an incremental load between an Oracle source table and an SQL sink table before? I am currently trying to do this but can only get my delta load to work when copying from SQL to SQL as you did in this video, any guidance would be greatly appreciated :)

I think it's for more easy with foreach loop..what's your views ?

Expression of type: 'String' does not match the field: 'additionalColumns' I am getting this error .. MY source is Servicenow and Destination is AZURE SQL DB .. Please help me on this

thank you for the video sir.. yesterday i got same question in the interview

This is good! Thanks for sharing. One suggestion is that it will be much more helpful to add how you deal with “update” and “delete” or “merge” under this architecture. That is the “soul” of leveraging delta lake!!

What if the copy activity fails for one of the iteration inside foreach? Will the foreach fails automatically?

Explained very nice

The explanation is very good, but while trying to implement the same, getting errors, especially with formulas. Kindly make videos little more detailed.

Hi Cloud Guru, In This video you explained about what are the resources to be added/configured in DEV,TEST,PROD. Can you send me video link for deployment in DEV and migrate to test . I dont see that deployment videos in that playlist

This video has content upto Full load , Please post the video for Incremental load by using that load_control table

Thank you so much sir for the valuable info

Very useful content. Thank you.