Thank you for watching my videos. You might need to have two branches one branch taking care of current data , while on another sourcs you are just crawling. And then you are comparing at a single Transform task like "custom transform". You can try many other approaches as well. I shall create video on your scenario if you can explain bit more.
Thanks for the quick response 🙂 My scenario is - 1. The source will be generating some ticketing information via kinesis stream 2. I am creating a report which is an aggregated table from almost 8 other tables 3. We are pushing data to an s3 data lake 4. So I need to to perform all the aggregated transformations related to the report on the flight within the stream
Again when you say aggregate from stream data + table stored data ( May be rds) , are you merging or joining data from stream with table an then storing s3 bucket data lake
Thank you for watching my videos. Apologies if I have not covered but let me tell you that it full admi access with required trust definition. You can watch other videos on ETL I have shown it.
Thank you Bro, your videos are very helpful. I was stuck in one issue, through your video I got the solution :)
Thank you for watching my videos.
Glad that it helped you.
Thank you so much. Perfectly reproducible. Awesome video.
Thank youfor watching my videos.
Glad that it helped you.
how to automate athena query from etl job completions?
Thank you for watching my videos.
Do you mean you want to run some query on your dataset in etl pipelines
thanq nicely explained.
Thank you for watching my videos.
Glad that it helped you.
What will be process if i need to write transformations on the data, by comparing the existing data (previously processed data).
Thank you for watching my videos.
You might need to have two branches one branch taking care of current data , while on another sourcs you are just crawling. And then you are comparing at a single Transform task like "custom transform". You can try many other approaches as well. I shall create video on your scenario if you can explain bit more.
Thanks for the quick response 🙂
My scenario is -
1. The source will be generating some ticketing information via kinesis stream
2. I am creating a report which is an aggregated table from almost 8 other tables
3. We are pushing data to an s3 data lake
4. So I need to to perform all the aggregated transformations related to the report on the flight within the stream
Again when you say aggregate from stream data + table stored data ( May be rds) , are you merging or joining data from stream with table an then storing s3 bucket data lake
Data in kinesis stream will be a CDC from RDS and the tables to join meanse we can say tables from data lake via data catalogues
Okay.. and where is target to store the merge of CDC RDS + Table from Datalake catalog?
you told like will say the iam role cofiguration setting in final. but you did'nt
Thank you for watching my videos.
Apologies if I have not covered but let me tell you that it full admi access with required trust definition. You can watch other videos on ETL I have shown it.