This is such a cruel world. People like technical guruji, just read out the specs of a phone that's already out there on the internet on 1000s of websites and have 20M plus subscriptions. And here is a gem, who is explaining such good info in detail, and has 2k. But I'm sure hard work pays, and you have our backing bro. Please continue with the good work!
Thank you 🙏 for watching my video and sharing your feedback and your word of appreciation really means a lot to me. The snowflake community is still very small, compare to other tech stacks... so less views... and that's why I am sharing my knowledge, so the snowflake community can learn and grows with this. Thanks again @aniket and feel free to share this channel detail with your community and your circle. Your msg is pinned.
Army jawan gets paid less but cricketer are highly paid. It's just about viewership sponsorship and market trends. Irrespective of this let's just appreciate this guy for his hard work.
I discovered you channel today I did masters in data science Italy they never covered any topic and we were left to figure it out oversell could not have learn without channels like this ,continue making videos
Really appreciate the level of content you have brought in this tutorial. And awe for the hard work you have taken to prepare this... Can't believe this you are providing for free. Thanks a lot for being generous.
This is the best series I have come across. Very well explained with examples and diagrams. Thank you! I have one questions - why didnt we use the merge approach for the aggregated tables? Join on the id, and then set target values to the source?
Thanks for your note if you would like to get all the video and unable to find what all are covered under different playlist, then you can download following cheatsheets. 🚀🚀 Cheat Sheets 🚀🚀 ➥ Complete Snowflake Tutorial rebrand.ly/maj4l6x ➥ Complete Snowflake JSON Guide rebrand.ly/d52cef
Hello, i am new at DE world, I am studying for the SnowflakePro Core. Let me tell you: this set of tutorials it is just amazing! (some small details maybe have changed/updated over the last 2 years, so it is good to complement your study with the Snowflake Documentation). This particular video of ETL Flow it is just awesome, thank you very much for show it, i specially love your practical approach. THANK YOU Data Enginnering Simplified! Please keep sharing and doing 'public' service :) P.S. If you want to expand your channel to dbt would be great!
Amazing explanation without wasting even a second. I could understand the end to end process of ETL workflow in Snowflake. Thanks for such a great video!
Glad it was helpful! and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEWYEAR50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEWYEAR35
glad you liked it.. and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEWYEAR50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEWYEAR35
You have taken so much efforts for making a single concept understand.. I don't think i came across any video on any topic which has explanation as your videos have. I feel lucky that I wanted to study snowflake and I found your videos. Keep doing good work 🥺
You're very welcome! ------ You can download this summary sheet in PDF version that has detail for all the chapters in one single file. Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB
Such a wonderful video with clear cut explanation. Thanks for all your great content. Could you also please create an End to End ETL workflow in snowflake for Azure cloud as well. This will help lot of folks.
You are doing good job, we are with you. Keep it up. Very nice tutorial. Is it possible to load the data files so that it will be perfect to follow the steps.
31:57 on the landing zone streams you are mentioning append_only = TRUE (cdc captures only inserted rows) 33:15 what's the purpose of update query ? cdc captures only inserted rows do we really need update query in 33:15 ?
I was so confused like i don’t even know how workflow of pipeline works this videos definitely helped me now i am planning to create my own project thanks to you sir!
great that you like it.. if you want to learn with lot of data.. you can also check my udemy courses. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEW-YEAR-2024 www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEW-YEAR-2024 www.udemy.com/course/snowflake-dynamic-table-masterclass-e2e-data-pipeline/?couponCode=AC7896594913CDFA6155
glad to know that you liked the contents here.. and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Its very good that you have covered end to end flow in this tutorial, if followed (which I did) step by step, concepts become very clear. Thanks. I would suggest to include the SQS notification part on Amazon S3 side in the Snowpipe related chapter(chapter 10), which will make that chapter complete. I got stuck over there when python script came in picture, as I was not knowing how to execute it and test.
Everyone dont have the heart to share learned things to others. But you have, hats off to you.....we are interested to learn and get more knowledge from you Sir.. keep sharing your knowledge. So many middle class people & financially less like me learning from your platform without joining the course outside which cost more. We are depend on your knowledge.. Everyone have mentor in IT field. For me this channel other than snowflake please also share job opportunity outside, current technology to learn, advice for the people who wants to grow to next level, working culture in IT field, some important topic as you like.. please sir. I want to move to next level in my career really depressed don't have any lead to help me.. How to be technically strong in IT field what are the technic to follow to achieve it. How was your experience how did you faced all struggle and reached this place. Can you share these kind of advice because struggling people like me and don't know where to start person helps a lot.
No doubt one of the best video for learning Snowflake from the scratch. I have gone through all 26 videos so far and enjoying each of the videos. Wanted to know if there is a video on Data Masking policy is also available? Also, it might be too much to ask but I am also curious to know if you are planning to publish any video on Snowflake integration with DBT or DataOps or with any modern ELT/ETL Tools.
Thanks a lot sharing your knowledge. Great tutorial. I had a doubt; Towards the end, there was a file which has failed. Could you tell us what are the next steps? how to fix the issue? How to trigger the file load again in snowflake( assuming we are not editing the source file in s3 or reuploading to s3) ?
Glad you enjoyed it.. and yes.. I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Have you done any examples with periodic snapshot or base transaction facts? Are there any examples for handling early arriving facts and/or late arriving dimensions? I suppose those cases can be applied or inferred from these examples?
Thank you very much for your video! I have a question about "deleting a record". Since this case sets "append_only=true" on e.g. landing_zone.Item_stm, then there will be no source-side deletes captured. If so, I think there is also no delete happening to Curated.Item table, then no deletes will be captured by curated_zone.item_stm, then the merge operation in task E will not work on any deletes since there should be no stream record with METADATA$ACTION='DELETE' AND METADATA$UPDATE='TRUE'. Am I correct? Thanks!
Glad you linked it.. you can find this summary sheet which has all the chapter of this playlist. Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB
Thanks for concise and crisp explanation. Could you please let us know how to get complete history data as only sample records are provided along with scripts.
Thanks for your note... glad you liked the content. pls check the description for the data set, it is hard to keep the large data, need to check if I still have the large data set with me.
Very useful to see how the continuous data load works under the hood with real time experience.Awesome work.!👏🏻👏🏻👏🏻I thoroughly enjoyed ur video and the way you put so much effort to larger audience is truly commendable.!
Thank you for all your effort. I am quiet sure it must be very time consuming to do all this. Much apprecaited. Following additional questions: 1. Is this realistic to maintain with streams where data volume and velocity is high? Would a similar approach with a traditional COPY INTO command be a more realistic implementation in real world example? Could you elaborate? 2. Would it be possible to have an alert mechanism that can notify when a data load fails for any reason? Using Snowpipe and with batch COPY INTO approach? 3. How would one track and monitor these objects in a more manageable way on a day to day basis for the entire data mart which might have 10's of dimensions and multiple fact tables?
at 50:44 when you added new record and existing record how does it work in snowflakes that it identifies the existing records . we have not defined any constraint and the constraint does not work in snowflakes AFAIK. Could you please clarify ?
that is a really amazing effort really appreciated, I have a question, and perhaps I've misconstrued something. When establishing the data flow from the landing zone to the curated zone, a parameter for "append only" is set. This implies that any updates made to the landing table won't be integrated into the curated table (only new inserts). Specifically, the logic task created to run every minute involves two checks: one for matched updates and the other for unmatched entries as new records. With the "append only" setting, only new inserts will be present in the stream, ensuring that no matched data appears in the created stream. So, I'm curious about the rationale behind adding the "append only" parameter in this scenario. -
in real life world, your data will not land to snowflake directly, it will come as new file into external stage location ... and that external stage location will be S3 or ADLS Gen2, that is also immutable storage.. so you need to place a new file.. even if it is an update.. so append only option is the right configuration... and if you would like to know more about it.. prefer to watch my end to end project.. that is more on Snowflake Snowpark.. but will help you to answer this question.. th-cam.com/video/1jC98XQwBZw/w-d-xo.html ------ and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Very Good explanations to understand and implement snowflake features, all chapters are detailed with hands-on examples. Do you have any use case to explain entire snowflake job orchestration to inject on-prem and cloud storage data/files at one given time also can we schedule CLI commands using task?
Excellent. I have subscribed and started going over the play list. Just wondering whether you have similar training videos of Snowflake in Azure as well please? These days...Data engineers without knowledge of Snowflake is not recognised that well in UK job market.
was not aware if UK Job market is really looking for snowflake aware developer, but love to know why? is it limited to data engineering domain or all other application development domains... I have not really concentrated on Azure, but it looks Azure+Snowflake partnership is growing and recently they published a video on it.. will focus on Azure also.. and will soon release videos on Snowflake+Azure.
Hi..if it's a one time load..means right from the source,should we go through all these landing zone and curated zone? Can't we directly load it to the consumption zone?
Yes, it can be done, if your data is curated and ready to fit into consumption zone... like 1st time migration project..load data to the respective table and then your cdc or incremental moves through left to right as shown in this table.. thanks for your note.. it helps me others to understand different scenarios...
This session really helps with ETL workflow. Thanks for providing such an amazing session. Actually, I tried to do the same practice which is shown in the session, but unfortunately, my snowflake has been hosted on Azure cloud. Due to this not getting the ARN number and auto ingest is not working. Could you please suggest how can I change the cloud from Azure to AWS? This will be very helpful for me. I tried to search from google, but could not get the correct solution.
Dont worry, you can watch my my ch-10 where you can also load continous data without any external stage th-cam.com/video/PNK49SJvXjE/w-d-xo.html toppertips.com/continuous-data-loading-ch10
Very useful video. I have couple of questions. How can we get data from other external source systems(Oracle, Excel, CSV, Teradata, MYSQL, DB2) like following ETL tools informatica, streamsets, or Datastage ..etc.. Snowflake may useful to read data from S3 backet to load into Cloud snowflake DB and from there data may be available for any other purpose. But initial step we need to use other ETL tool to get data into S3 bucket right? in this case we need to pay the money for other ETL tool as well right? Can you please suggest how snowflake is better than other cloud DB's and other ETL tools?
To collect data (extraction) and make it available to snowflake's external stage need similar tools like StreamSet or Fivetran or Matillion. The extraction or data collection part is missing. I would suggest to watch my data loading playlist th-cam.com/play/PLba2xJ7yxHB6NPEv8pp7j3zWibLZzwjvO.html
This is just as possible with Azure and Event Grid trigged storage events with ADF from an SFTP server as a source.......to ADLS for initial ingestion.....and then passed to snowpipe through event hub/storage queue/svc bus and snowpipe and integration object, correct? You should do an Azure version of this if you haven't already. I know Mohit Batra put one out on Pluralsight.....
Wonderful explanation!! thank you so much.. What is a good way to bring historic data from Oracle to Snowflake? Is AWS Snowball the option? any good recommendation please?
Thank you 🙏 for watching my video and your word of appreciation really means a lot to me. Loading data from Oracle to Snowflake, need data to be moved to S3 first and then it has to be moved to Snowflake, this can be done via different tool available based on data volume. There are other cloud tools like fivetran which can do history load as well as delta/cdc load and it is pipeline/data-ingesting as service platform.
@@DataEngineering thank you, if we are leveraging existing Azure setup, do you recommend ADF (Azure Data Factory) to stage the data in ADLS (Data lake store), if the data volume is ~200 TB from Oracle? or any other recommendations plz?
Great videos! I have been following along. However, when I go to your website to download the data, I only find the sample files (around 20 rows). When loading to the consumption_zone.order_fact I get 0 rows inserted as the sample files don't probably have the IDs in common. I've checked my code (which I just copied from the website) and it is definitely the data. Can you send a link to the actual data (or at least larger sample)?
Could you please share some more details on how to create an event on AWS to capture the Delta from a Source Table? I'm interested in loading only data from My source tables, but not the files from the S3 bucket, How to automate the load process?
Thank you 🙏 for watching my video and your word of appreciation really means a lot to me. ⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡ I have already published other knowledge series and snowflake certification videos and if you are interested, you can refer them. 🌐 Snowflake Complete Guide Playlist ➥ bit.ly/3iNTVGI 🌐 SnowPro Guide ➥ bit.ly/35S7Rcb 🌐 Snowflake SQL Series Playlist ➥ bit.ly/3AH6kCq 🌐 SnowPro Question Dump (300 questions) ➥ bit.ly/2ZLQm9E ⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡
Hi.. Thanks for your excellent tutorials on Snowflake .. I have a question on this video .. When you are creating landing_order_stream as append only stream , you'll only see new orders in the stream . Subsequently why are you trying to do a merge operation that looks for both inserts and updates .. You only need to worry about inserts .. Please correct me if my understanding is wrong
in the landing_order_stream it's only inserted, but when creating a task on this table, he created a merge statement in which we handle inserts and updates. So in the next phase, we need to handle updates...note that Snowflake treats metadata$update True only when data is loaded using update syntax.
Great video sir. Really appreciate your efforts. Just one thing need to understand if delta data is added into same csv file from which full load is done, how it will handle ,will it process only changed data?
Help me with a use case that you feel will be good to help larger audience. Drop your detailed requirement in my insta account (instagram.com/learn_dataengineering/)
Waouh dude, this is really impressive and very clear ! thank you so much ! I have one question : is it possible to replace AWS S3 stage by 'internal stage' for education purpose and test ? I would like to load manually 'delta' csv files' and the pipeline do the work.
yes you can do it.. watch my latest video that explain how to load large data set into internal storage th-cam.com/video/1jC98XQwBZw/w-d-xo.html (End to End ETL Using Snowpark)
Glad it was helpful! Thank you 🙏 for watching my video @Snehendu Dey and your word of appreciation really means a lot to me. ⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡ I have already published other knowledge series and snowflake certification videos and if you are interested, you can refer them. 🌐 Snowflake Complete Guide Playlist ➥ bit.ly/3iNTVGI 🌐 SnowPro Guide ➥ bit.ly/35S7Rcb 🌐 Snowflake SQL Series Playlist ➥ bit.ly/3AH6kCq 🌐 SnowPro Question Dump (300 questions) ➥ bit.ly/2ZLQm9E ⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡
Great videos and explanation .when inserting data from LANDING_CUSTOMER to CURATED_CUSTOMER, am getting an error "Numeric value 'BIRTH_DAY' is not recognized" ( conversion from varchar to number) . Anyway to avoid it?
you can try using cast function or double-semi-colon.. not sure why it is causing an issue when the demo is build on the data .. so it should not be an issue
Hello Sir . I really like the way you described the concepts and i want to practice the concepts by hands-on .I am unable to open your website to copy SQL script . Could you please help me in knowing how i can access ur website please .I am seeing an error saying Site under maintenance .your help will be much appreciated
Thank for doing this great session. it helps a lot. however I am getting error "S3 bucket 'toppertips' does not exist or not authorized" executing SQL.
Hello Abhishek, the S3 bucket created in my video is the S3 bucket I created for this tutorial, you have to replace it with your own S3 bucket where you have access to. If you need to understand how it works, refer ch-10 (th-cam.com/video/PNK49SJvXjE/w-d-xo.html)
Can a task from one schema call another task from a different schema? Which version or release this videos been created ? Is it still the case now with the latest versions of Snowflake?
as far as I know, this was one of the limitation and this has been added a new feature, I am not aware of... Will review the docs and will revisit my content and will adapt it if needed. You can download this master cheat sheet, that can help you to identify which all topics and sub-topics are covered under different playlist and different videos. 🚀🚀 Cheat Sheets 🚀🚀 ➥ Complete Snowflake Tutorial rebrand.ly/maj4l6x ➥ Complete Snowflake JSON Guide rebrand.ly/d52cef
All the sessions are really helpful. Thank you so much. Could you please let me know where can I get the sample data files for history load? I don't see any file to download on your site. if you could help here.
Glad you like them! you can refer the link and end of the blog, you have sample files content, not sure if helps you or not. toppertips.com/snowflake-etl-example-ch19-part01
@@DataEngineering Thank you! I am following the current practice video. while inserting data into consumption_zone.order_fact table, seems like other schema tables are not accessible. I tried several grant queries but nothing worked. could you please help?
Many standard data projects have curated layer and in this layer data is cleaned, curated & reconciled before the data is modelled (dimension modeling etc) for larger use. For example, you are getting employee data from two different sources and you want to make sure that they are reconciled before they can be used by business, in that case you need a middle layer and you can name it as curated layer. Hope this clarifies.
In part5, we are loading data from stream object of landing zone to curated zone . Q. We are wring Update command in task, but our stream was defined with APPEND_ONLY MODE , how does it tracks updates/deletes ?
as stream stores inserted new records with APPEND_ONLY mode then there is no necessity of using UPDATE right ? and we could insert all the new records into curated tables directly through task and Stream combination whenever there is change . please correct me here.
In my demo, no specific changes are done.. but in complex and large project, you will see lot of transformation logic, but it is not must to have so many layer..
1.Why we are creating sequence(eg like customer_pk,order_pk etc) in each table in curated zone and what is the purpose of this . 2. we are using transient tables in landing zone and curated zone. 3.How we are identifying SCD or delta or incremental data in this ETL approach
The purpose of this video is to show how data flow and the actual data model might differ from project to project. 1. the pk columns are just to desmostrate and it has no specific meaning in the given use case. 2. Transient tables does not incur fail safe cost and that's why we used it. If churn rate is high, permanent table cost a lot. Watch my this video for detail th-cam.com/video/S5NwU2o2Exg/w-d-xo.html 3. watch the stream chapter th-cam.com/video/DXI0GDSwE_E/w-d-xo.html before watching this video.
I had doubt, can someone clear me, how the stream table finding updated data from the landing table. The custom update into table can have metadata INSERT, true in stream table but how the updated record contains in file captured in stream?
Thanks a lot for the video. I tried loading some sample CSV file from my local drive into a table I created in Snowflake. I get the following. " There was an error while trying to stage "Customer.csv". Reason: Unable to upload the file to Azure Storage". The file I am trying to load is Customer.csv. Any clues as to why this is happening will be greatly appreciated. Cheers
check the description.. and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in udemy.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/ 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/
pls check the description of the video, it has link given for the web page and git location where you can download it. You can also download the quick guide, that has link for all videos.. (Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB)
This is such a cruel world. People like technical guruji, just read out the specs of a phone that's already out there on the internet on 1000s of websites and have 20M plus subscriptions. And here is a gem, who is explaining such good info in detail, and has 2k. But I'm sure hard work pays, and you have our backing bro. Please continue with the good work!
Thank you 🙏 for watching my video and sharing your feedback and your word of appreciation really means a lot to me.
The snowflake community is still very small, compare to other tech stacks... so less views... and that's why I am sharing my knowledge, so the snowflake community can learn and grows with this.
Thanks again @aniket and feel free to share this channel detail with your community and your circle. Your msg is pinned.
Army jawan gets paid less but cricketer are highly paid. It's just about viewership sponsorship and market trends.
Irrespective of this let's just appreciate this guy for his hard work.
I discovered you channel today I did masters in data science Italy they never covered any topic and we were left to figure it out oversell could not have learn without channels like this ,continue making videos
This is the best tutorial I've ever seen on Snowflake. Simply exceptional. Thanks sir.
Really appreciate the level of content you have brought in this tutorial. And awe for the hard work you have taken to prepare this... Can't believe this you are providing for free. Thanks a lot for being generous.
Thank for you putting all you time and efforts. Appreciate all you work here. This helps as lot.
This is the best series I have come across. Very well explained with examples and diagrams. Thank you!
I have one questions - why didnt we use the merge approach for the aggregated tables? Join on the id, and then set target values to the source?
feeling lucky I came across this course on YT, for FREE, and much better than paying for udacity
Thanks for your note
if you would like to get all the video and unable to find what all are covered under different playlist, then you can download following cheatsheets.
🚀🚀 Cheat Sheets 🚀🚀
➥ Complete Snowflake Tutorial rebrand.ly/maj4l6x
➥ Complete Snowflake JSON Guide rebrand.ly/d52cef
Hello, i am new at DE world, I am studying for the SnowflakePro Core. Let me tell you: this set of tutorials it is just amazing! (some small details maybe have changed/updated over the last 2 years, so it is good to complement your study with the Snowflake Documentation).
This particular video of ETL Flow it is just awesome, thank you very much for show it, i specially love your practical approach.
THANK YOU Data Enginnering Simplified! Please keep sharing and doing 'public' service :)
P.S. If you want to expand your channel to dbt would be great!
Amazing explanation without wasting even a second. I could understand the end to end process of ETL workflow in Snowflake. Thanks for such a great video!
Glad it was helpful!
and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEWYEAR50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEWYEAR35
This is the best tutorial playlist I have watched on TH-cam
glad you liked it..
and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEWYEAR50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEWYEAR35
Much better than Udemy n all !
Explained core concept, Design pattern, data model everything.... Thank you ...
You are most welcome
You have taken so much efforts for making a single concept understand.. I don't think i came across any video on any topic which has explanation as your videos have. I feel lucky that I wanted to study snowflake and I found your videos. Keep doing good work 🥺
I am just wondering, how come you have less subscribers. I AM SURE ONE DAY YOU WILL HAVE MILLION's. BEST CHANNEL SO FAR...
Thanks for support ... if you could share this channel to your internal community and circle... rest youtube will do it jobs..
Couldn’t ask for more! Perfect explanation. Has all industry concepts.
Thanks so so much!!!!
You got a subscriber:)
Welcome aboard!
you can find the playlist summary in this cheat sheet.
(Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB)
Appreciate your meticulousness and enthusiasm in your tutorials! Thanks a lot! 👍🏻
My pleasure!
I just went thru part of a video and i was impressed. Thank you for providing this. I need to renew my Snowpro certification. Excellent work.
Glad it was helpful!
Thank You so much. Such high quality content for free, you are truly generous.
You're very welcome!
------
You can download this summary sheet in PDF version that has detail for all the chapters in one single file.
Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB
undoubtedly best snowflake course. not just teach command
welcome..and glad that you like my videos.
Such a wonderful video with clear cut explanation. Thanks for all your great content.
Could you also please create an End to End ETL workflow in snowflake for Azure cloud as well. This will help lot of folks.
You are doing good job, we are with you. Keep it up. Very nice tutorial. Is it possible to load the data files so that it will be perfect to follow the steps.
Thank you so much for such knowledgeable content
31:57 on the landing zone streams you are mentioning append_only = TRUE (cdc captures only inserted rows)
33:15 what's the purpose of update query ?
cdc captures only inserted rows do we really need update query in 33:15 ?
Very useful.. easy explanation and understanding. Much Apprecaite your thrive to share the knowledge to others.. Keep up the Great work
Glad it was helpful!
I was so confused like i don’t even know how workflow of pipeline works this videos definitely helped me now i am planning to create my own project thanks to you sir!
great that you like it..
if you want to learn with lot of data.. you can also check my udemy courses.
www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEW-YEAR-2024
www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEW-YEAR-2024
www.udemy.com/course/snowflake-dynamic-table-masterclass-e2e-data-pipeline/?couponCode=AC7896594913CDFA6155
one of the best snowflake content.plz try to make a complete portfolio video on data analytics on snowflake tables.
glad to know that you liked the contents here..
and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Thanks a lot for this detailed video
Its very good that you have covered end to end flow in this tutorial, if followed (which I did) step by step, concepts become very clear. Thanks. I would suggest to include the SQS notification part on Amazon S3 side in the Snowpipe related chapter(chapter 10), which will make that chapter complete. I got stuck over there when python script came in picture, as I was not knowing how to execute it and test.
Everyone dont have the heart to share learned things to others. But you have, hats off to you.....we are interested to learn and get more knowledge from you Sir.. keep sharing your knowledge. So many middle class people & financially less like me learning from your platform without joining the course outside which cost more. We are depend on your knowledge.. Everyone have mentor in IT field. For me this channel other than snowflake please also share job opportunity outside, current technology to learn, advice for the people who wants to grow to next level, working culture in IT field, some important topic as you like.. please sir. I want to move to next level in my career really depressed don't have any lead to help me.. How to be technically strong in IT field what are the technic to follow to achieve it. How was your experience how did you faced all struggle and reached this place. Can you share these kind of advice because struggling people like me and don't know where to start person helps a lot.
Very useful for understanding the whole data lifecycle from S3 to data mart in Snowflake. Thank you so much!
Thank you 🙏 for watching my video and your word of appreciation really means a lot to me.
No doubt one of the best video for learning Snowflake from the scratch. I have gone through all 26 videos so far and enjoying each of the videos. Wanted to know if there is a video on Data Masking policy is also available? Also, it might be too much to ask but I am also curious to know if you are planning to publish any video on Snowflake integration with DBT or DataOps or with any modern ELT/ETL Tools.
added into my list.. will try in future.
Finally the wait is over! 😀👍 Thank you!
More to come! and again thanks a lot for following and providing your feedback on regular basis. Helps me a lot to continue to share my knowledge.
Good one. I have a clarification here. In this example, DELETE scenario will not work. Right?
hands down best tutorial I found on snowflake. wish I could get access to data files.
Thanks for your note..and glad you liked the content....
refer the description section of the video, it has link for blog page + data file link (git)
Very good tutorials! Happy to subscribe.
Awesome, thank you!
Thanks a lot sharing your knowledge. Great tutorial. I had a doubt; Towards the end, there was a file which has failed. Could you tell us what are the next steps? how to fix the issue? How to trigger the file load again in snowflake( assuming we are not editing the source file in s3 or reuploading to s3) ?
Great video
Glad you enjoyed it.. and yes.. I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Excellent Video! I had a doubt where can i get the dataset used in this tutorial?
Glad it was helpful!.. check the description of the video
Have you done any examples with periodic snapshot or base transaction facts? Are there any examples for handling early arriving facts and/or late arriving dimensions? I suppose those cases can be applied or inferred from these examples?
Thank you very much for your video! I have a question about "deleting a record". Since this case sets "append_only=true" on e.g. landing_zone.Item_stm, then there will be no source-side deletes captured. If so, I think there is also no delete happening to Curated.Item table, then no deletes will be captured by curated_zone.item_stm, then the merge operation in task E will not work on any deletes since there should be no stream record with METADATA$ACTION='DELETE' AND METADATA$UPDATE='TRUE'.
Am I correct? Thanks!
This is Just Brillant
Glad you linked it.. you can find this summary sheet which has all the chapter of this playlist.
Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB
Thanks for concise and crisp explanation. Could you please let us know how to get complete history data as only sample records are provided along with scripts.
Thanks for your note... glad you liked the content.
pls check the description for the data set, it is hard to keep the large data, need to check if I still have the large data set with me.
Very useful to see how the continuous data load works under the hood with real time experience.Awesome work.!👏🏻👏🏻👏🏻I thoroughly enjoyed ur video and the way you put so much effort to larger audience is truly commendable.!
Glad you liked it @Fahad Mahmood, your word of appreciation really means a lot to me.
@@DataEngineering 11aq
Thank you for all your effort. I am quiet sure it must be very time consuming to do all this. Much apprecaited. Following additional questions:
1. Is this realistic to maintain with streams where data volume and velocity is high? Would a similar approach with a traditional COPY INTO command be a more realistic implementation in real world example? Could you elaborate?
2. Would it be possible to have an alert mechanism that can notify when a data load fails for any reason? Using Snowpipe and with batch COPY INTO approach?
3. How would one track and monitor these objects in a more manageable way on a day to day basis for the entire data mart which might have 10's of dimensions and multiple fact tables?
Awesomely explained
Glad you liked it
at 50:44 when you added new record and existing record how does it work in snowflakes that it identifies the existing records . we have not defined any constraint and the constraint does not work in snowflakes AFAIK. Could you please clarify ?
Appreciate for posting good videos like this for free and it helps all types of learners. Please keep continuing the great share of knowledge.
Thanks, will do!
Great work, Appreciated for your hard work,
How can we get this complete course html Page which you are following for explain each and every topic?
that is a really amazing effort really appreciated, I have a question, and perhaps I've misconstrued something. When establishing the data flow from the landing zone to the curated zone, a parameter for "append only" is set. This implies that any updates made to the landing table won't be integrated into the curated table (only new inserts). Specifically, the logic task created to run every minute involves two checks: one for matched updates and the other for unmatched entries as new records. With the "append only" setting, only new inserts will be present in the stream, ensuring that no matched data appears in the created stream. So, I'm curious about the rationale behind adding the "append only" parameter in this scenario. -
in real life world, your data will not land to snowflake directly, it will come as new file into external stage location ... and that external stage location will be S3 or ADLS Gen2, that is also immutable storage.. so you need to place a new file.. even if it is an update.. so append only option is the right configuration...
and if you would like to know more about it.. prefer to watch my end to end project.. that is more on Snowflake Snowpark.. but will help you to answer this question.. th-cam.com/video/1jC98XQwBZw/w-d-xo.html
------
and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Very Good explanations to understand and implement snowflake features, all chapters are detailed with hands-on examples. Do you have any use case to explain entire snowflake job orchestration to inject on-prem and cloud storage data/files at one given time also can we schedule CLI commands using task?
Excellent. I have subscribed and started going over the play list. Just wondering whether you have similar training videos of Snowflake in Azure as well please? These days...Data engineers without knowledge of Snowflake is not recognised that well in UK job market.
was not aware if UK Job market is really looking for snowflake aware developer, but love to know why? is it limited to data engineering domain or all other application development domains...
I have not really concentrated on Azure, but it looks Azure+Snowflake partnership is growing and recently they published a video on it.. will focus on Azure also.. and will soon release videos on Snowflake+Azure.
@@DataEngineering almost every data engineering and data architect job in UK is asking for Snowflake experience. Do you provide virtual training too ?
Very nice session and the way ETL process has been explained
Glad you liked it
Very Good explanations
glad you liked it..
Thanks for your lesson and where are the test csv files?
In the description
Hi..if it's a one time load..means right from the source,should we go through all these landing zone and curated zone? Can't we directly load it to the consumption zone?
Yes, it can be done, if your data is curated and ready to fit into consumption zone... like 1st time migration project..load data to the respective table and then your cdc or incremental moves through left to right as shown in this table..
thanks for your note.. it helps me others to understand different scenarios...
Acharya devobhava🙏...
You are a great teacher
Thanks a lot.. and glad you liked the videos and my contents..
@@DataEngineering yes.. They are truly good..
Can you please share your LinkedIn profile name/link if you don't mind 🙂...
This session really helps with ETL workflow. Thanks for providing such an amazing session.
Actually, I tried to do the same practice which is shown in the session, but unfortunately, my snowflake has been hosted on Azure cloud. Due to this not getting the ARN number and auto ingest is not working.
Could you please suggest how can I change the cloud from Azure to AWS? This will be very helpful for me. I tried to search from google, but could not get the correct solution.
Dont worry, you can watch my my ch-10 where you can also load continous data without any external stage
th-cam.com/video/PNK49SJvXjE/w-d-xo.html
toppertips.com/continuous-data-loading-ch10
Very useful video. I have couple of questions. How can we get data from other external source systems(Oracle, Excel, CSV, Teradata, MYSQL, DB2) like following ETL tools informatica, streamsets, or Datastage ..etc.. Snowflake may useful to read data from S3 backet to load into Cloud snowflake DB and from there data may be available for any other purpose. But initial step we need to use other ETL tool to get data into S3 bucket right? in this case we need to pay the money for other ETL tool as well right? Can you please suggest how snowflake is better than other cloud DB's and other ETL tools?
To collect data (extraction) and make it available to snowflake's external stage need similar tools like StreamSet or Fivetran or Matillion. The extraction or data collection part is missing. I would suggest to watch my data loading playlist th-cam.com/play/PLba2xJ7yxHB6NPEv8pp7j3zWibLZzwjvO.html
and the syntax is => AFTER task_name right ? , and not AFTER = task_name (timestamp : 35:18 part6)
This is just as possible with Azure and Event Grid trigged storage events with ADF from an SFTP server as a source.......to ADLS for initial ingestion.....and then passed to snowpipe through event hub/storage queue/svc bus and snowpipe and integration object, correct? You should do an Azure version of this if you haven't already. I know Mohit Batra put one out on Pluralsight.....
Wonderful explanation!! thank you so much..
What is a good way to bring historic data from Oracle to Snowflake? Is AWS Snowball the option? any good recommendation please?
Thank you 🙏 for watching my video and your word of appreciation really means a lot to me.
Loading data from Oracle to Snowflake, need data to be moved to S3 first and then it has to be moved to Snowflake, this can be done via different tool available based on data volume.
There are other cloud tools like fivetran which can do history load as well as delta/cdc load and it is pipeline/data-ingesting as service platform.
@@DataEngineering thank you, if we are leveraging existing Azure setup, do you recommend ADF (Azure Data Factory) to stage the data in ADLS (Data lake store), if the data volume is ~200 TB from Oracle? or any other recommendations plz?
Great videos! I have been following along. However, when I go to your website to download the data, I only find the sample files (around 20 rows). When loading to the consumption_zone.order_fact I get 0 rows inserted as the sample files don't probably have the IDs in common. I've checked my code (which I just copied from the website) and it is definitely the data. Can you send a link to the actual data (or at least larger sample)?
Could you please share some more details on how to create an event on AWS to capture the Delta from a Source Table? I'm interested in loading only data from My source tables, but not the files from the S3 bucket, How to automate the load process?
Waiting for video finally yehhh
Thank you @subh Reddy for following me and sharing your valueable feedback. Helps me to continue with my effort and share my knowledge.
All the sessions are really helpful. Thank you so much. need sample data files for history load? I don't see any link or dataset or file .
Working on them and will soon will make it part of video description.
ohh best please make more videos like this
Thank you @Bright Spark for following me and sharing your valueable feedback. Helps me to continue with my effort.
@@DataEngineering could you provide access to the script
Excellent !!!
Thank you 🙏 for watching my video and your word of appreciation really means a lot to me.
⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡
I have already published other knowledge series and snowflake certification videos and if you are interested, you can refer them.
🌐 Snowflake Complete Guide Playlist ➥ bit.ly/3iNTVGI
🌐 SnowPro Guide ➥ bit.ly/35S7Rcb
🌐 Snowflake SQL Series Playlist ➥ bit.ly/3AH6kCq
🌐 SnowPro Question Dump (300 questions) ➥ bit.ly/2ZLQm9E
⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡
Hi.. Thanks for your excellent tutorials on Snowflake .. I have a question on this video .. When you are creating landing_order_stream as append only stream , you'll only see new orders in the stream . Subsequently why are you trying to do a merge operation that looks for both inserts and updates .. You only need to worry about inserts .. Please correct me if my understanding is wrong
They are just examples for demo purpose, the real life scenario and implementation depends on use case.
in the landing_order_stream it's only inserted, but when creating a task on this table, he created a merge statement in which we handle inserts and updates. So in the next phase, we need to handle updates...note that Snowflake treats metadata$update True only when data is loaded using update syntax.
Thank you for your efforts. Amazing material.The link with the code does not work.
Thanks that was very useful .
Glad it was helpful!
Great video sir. Really appreciate your efforts. Just one thing need to understand if delta data is added into same csv file from which full load is done, how it will handle ,will it process only changed data?
Snowflake will not handle that.. it need to be taken care with manual logic.
pl upload a video doing some ETL transformation with any use case (End 2 end).
Thanks for the video, really helpful.
Help me with a use case that you feel will be good to help larger audience.
Drop your detailed requirement in my insta account (instagram.com/learn_dataengineering/)
you are doing god's work.
thanks...
Waouh dude, this is really impressive and very clear ! thank you so much ! I have one question : is it possible to replace AWS S3 stage by 'internal stage' for education purpose and test ? I would like to load manually 'delta' csv files' and the pipeline do the work.
yes you can do it.. watch my latest video that explain how to load large data set into internal storage
th-cam.com/video/1jC98XQwBZw/w-d-xo.html (End to End ETL Using Snowpark)
@@DataEngineering i will for sure. Thank you
@@DataEngineering On last question, on your last schema, why you didn't build foreign key constraint on fact table for your dimensions tables ?
If possible, please share knowledge about DBT tool.
Great lab
regards...
glad you linked it..
You can find more content in this Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB
Very useful, Thank you
Glad it was helpful!
Thank you 🙏 for watching my video @Snehendu Dey and your word of appreciation really means a lot to me.
⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡
I have already published other knowledge series and snowflake certification videos and if you are interested, you can refer them.
🌐 Snowflake Complete Guide Playlist ➥ bit.ly/3iNTVGI
🌐 SnowPro Guide ➥ bit.ly/35S7Rcb
🌐 Snowflake SQL Series Playlist ➥ bit.ly/3AH6kCq
🌐 SnowPro Question Dump (300 questions) ➥ bit.ly/2ZLQm9E
⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡⚡
Great videos and explanation .when inserting data from LANDING_CUSTOMER to CURATED_CUSTOMER, am getting an error "Numeric value 'BIRTH_DAY' is not recognized" ( conversion from varchar to number) . Anyway to avoid it?
you can try using cast function or double-semi-colon.. not sure why it is causing an issue when the demo is build on the data .. so it should not be an issue
Hello Sir . I really like the way you described the concepts and i want to practice the concepts by hands-on .I am unable to open your website to copy SQL script . Could you please help me in knowing how i can access ur website please .I am seeing an error saying Site under maintenance .your help will be much appreciated
Thank for doing this great session. it helps a lot. however I am getting error "S3 bucket 'toppertips' does not exist or not authorized" executing SQL.
Hello Abhishek, the S3 bucket created in my video is the S3 bucket I created for this tutorial, you have to replace it with your own S3 bucket where you have access to.
If you need to understand how it works, refer ch-10 (th-cam.com/video/PNK49SJvXjE/w-d-xo.html)
@@DataEngineering -Thank you I will do. is it possible for you to provide you one time data load files?
@@singh_abhishek1429 can you check the link.. add 1st few lines for history and delta load
toppertips.com/snowflake-etl-example-ch19-part01
Nice sir
Thanks and welcome
Can a task from one schema call another task from a different schema? Which version or release this videos been created ? Is it still the case now with the latest versions of Snowflake?
as far as I know, this was one of the limitation and this has been added a new feature, I am not aware of... Will review the docs and will revisit my content and will adapt it if needed.
You can download this master cheat sheet, that can help you to identify which all topics and sub-topics are covered under different playlist and different videos.
🚀🚀 Cheat Sheets 🚀🚀
➥ Complete Snowflake Tutorial rebrand.ly/maj4l6x
➥ Complete Snowflake JSON Guide rebrand.ly/d52cef
Thank you, i cleared the exam.
Well done!
All the sessions are really helpful. Thank you so much. Could you please let me know where can I get the sample data files for history load? I don't see any file to download on your site. if you could help here.
Glad you like them!
you can refer the link and end of the blog, you have sample files content, not sure if helps you or not.
toppertips.com/snowflake-etl-example-ch19-part01
@@DataEngineering Thank you! I am following the current practice video. while inserting data into consumption_zone.order_fact table, seems like other schema tables are not accessible. I tried several grant queries but nothing worked. could you please help?
@@DataEngineering But How to download them?
Can you please explain why we need extra curated layer?
Many standard data projects have curated layer and in this layer data is cleaned, curated & reconciled before the data is modelled (dimension modeling etc) for larger use. For example, you are getting employee data from two different sources and you want to make sure that they are reconciled before they can be used by business, in that case you need a middle layer and you can name it as curated layer.
Hope this clarifies.
can you explain the data migration from aws to snowflake and what tools are required for that
When you say AWS .. what all tool are being used in AWS side?
@@DataEngineering AWS service like migration MDS, lambda and Glue
Thanks for the amazing content! btw still waiting to download the csv files!
Will load all the file content.. pls give me some time.
Hi Sir , how do i download the source data to follow along in this tutorial ? you provided only sample data in the SQL scripts link
Hi.. what is the alternative of stream if delta records are more than 10k.. as stream has limitation of 10k
where
can i find the quires that used in videos
In part5, we are loading data from stream object of landing zone to curated zone .
Q. We are wring Update command in task, but our stream was defined with APPEND_ONLY MODE , how does it tracks updates/deletes ?
as stream stores inserted new records with APPEND_ONLY mode then there is no necessity of using UPDATE right ?
and we could insert all the new records into curated tables directly through task and Stream combination whenever there is change .
please correct me here.
Why we are moving data from landing zone to curated zone and to consumption zone and what are the changes
In my demo, no specific changes are done.. but in complex and large project, you will see lot of transformation logic, but it is not must to have so many layer..
1.Why we are creating sequence(eg like customer_pk,order_pk etc) in each table in curated zone and what is the purpose of this . 2. we are using transient tables in landing zone and curated zone. 3.How we are identifying SCD or delta or incremental data in this ETL approach
The purpose of this video is to show how data flow and the actual data model might differ from project to project.
1. the pk columns are just to desmostrate and it has no specific meaning in the given use case.
2. Transient tables does not incur fail safe cost and that's why we used it. If churn rate is high, permanent table cost a lot. Watch my this video for detail th-cam.com/video/S5NwU2o2Exg/w-d-xo.html
3. watch the stream chapter th-cam.com/video/DXI0GDSwE_E/w-d-xo.html before watching this video.
I had doubt, can someone clear me, how the stream table finding updated data from the landing table. The custom update into table can have metadata INSERT, true in stream table but how the updated record contains in file captured in stream?
Thanks a lot for the video. I tried loading some sample CSV file from my local drive into a table I created in Snowflake. I get the following. " There was an error while trying to stage "Customer.csv". Reason: Unable to upload the file to Azure Storage".
The file I am trying to load is Customer.csv. Any clues as to why this is happening will be greatly appreciated. Cheers
Strange error.. how are you uploading the csv file? Via WebUI?
sir where can i access the dataset for item,order and customer tables used in ETL flow..???
check the description..
and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in udemy.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/
@dataengineering I am new to snowflake, can you please share link to all csv files I am unable to find it in description link
it is available in the description.. pls check once again
@@DataEngineering It is not available in description, on website only 20 to 22 records are given. Please share the csv
Thank you very much
Thank you @Perumala Shureshh for following me, this helps me to continue with my effort and share my knowledge.
Can you please share how the solution will change if we have to use internal named stage instead of external?
Yes, it can be done using Snowpipe APIs with python program which was covered in ch-9 to ch-11
Sir where can i download file .csv at 14:11 ?
pls check the description of the video, it has link given for the web page and git location where you can download it.
You can also download the quick guide, that has link for all videos..
(Snowflake End To End Guide Cheat Sheet - bit.ly/43t5IPB)
Could you please provide the data set for this project?