Thanks for your note... If you want to manage snowflake more programatically.. you can watch my paid contents .. many folks don't know the power of snowpark... this 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... and make all SQL statement available for CI/CD... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=SPECIAL50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=SPECIAL35
Great video. It would have been wonderful if it also contained 1. how to do end to end CICD 2. How to setup pipeline dependency between data ingestion tool and snow flake task ( assuming we can bundle up all the loading steps you have covered in this video into a snowflake task) Apologies if you have already got these covered else where , if so please direct me. Many Thanks
Great video. It would have been wonderful if it also contained 1. how to do end to end CICD 2. How to setup pipeline dependency between data ingestion tool and snow flake task ( assuming we can bundle up all the loading steps you have covered in this video into a snowflake task) Apologies if you have already got these covered else where , if so please direct me. Many Thanks 1:21:30
yes, you can do it... and snowpark is nothing but SQL generator with current version .. may be you can watch this video.. what it is and what it is not 1. th-cam.com/video/-awSPRW9AOY/w-d-xo.html (What is snowpark) 2. th-cam.com/video/7tToBddZ_is/w-d-xo.html (What is NOT snowpark)
complete data set is too big.. the desc has the link that has limited data. ----- and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in udemy.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/ 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/
HI , i was tryin to run the getting the csv etc. files on the internal stage , I even get the output of the directory but Im not able to see the data as a result ! pls do help have been trying to rectify for hours but got no clue! thanks
Not sure which step you are talking about... if you can give me a timestamp, it will be helpful or you share a screenshot to my instagram account (instagram.com/learn_dataengineering/)
@@DataEngineering-Its Step4,I am using windows 11 os and struggling to run the python script also tried using put command from snow SQL, but nothing is working please advise and help
Thanks for everything. U helped a lot ❤! May i ask if u can make videos on the exception handling and error logging? E.g. one of the csv has an additional column. Another example is when loading data into the internal stage, wifi connection failed and how to resume the job? Thanks bro! :)
Quick question: I am on Part 4 but for some reason I downloaded the data to my computer but it does not show when I try to run it through Snowflake. Is there a reason for that?
Can you please tell me how to update row in snowflake_sample_data.tpch_sf100.orders??? getting error - "Object 'ORDERS' does not exist or not authorized."
Thank you for sharing such a good content. I should say you are a rockstar in Snowflake world. I have a question. In this case, since there are lot of Data frames created in snowpark-python scripts and running the code from local machine ,does it consume local system storage/compute or push everything to the Snowflake storage/Compute? Thank you in advance!
thanks for your note.. when you perform an operation using dataframe in snowflake, it uses snowflake's compute power. When you pull data to your location machine..in that case.. it uses your local compute...
Thank you for your quick response. If i would like to push everything to the snowflake storage and compute, how should we do it? How should we register the snowpark-python programs in snowflake database and run/debug it(Instead of Stored proc route) ? is is really possible? May be having a separate video might help@@DataEngineering
Watch ch-08 from this snowpark playlist.. and you would understand how to deploy it (playlist link th-cam.com/play/PLba2xJ7yxHB4yPg3pUrobdzeMxk4mP24S.html)
@@DataEngineeringThank you. I already watched it. Does that mean we should test it locally first and then deploy on SF sandbox. I am looking for options if we can develop,test, debug and deploy directly in the SF sandbox itself? Is it possible? Any insight?
Hi Bro, While going through the course, I found out that not all the data is provided in the gitlab link as well as the exchange_rates.csv at 50:00 . The exchange rate column is null for all rows after moving the file into curated stage. Could you update the link with all the files as mentioned in the course? Thanks
The dataset in the gitlab is only for 30 days and the forex date is also different for what you have. Can you please crosscheck the files and share the correct ones?
When you connect to any BI tool like PowerBI..they need these relationship.. and can build the model for slice and dice... and if you have to draw the ER diagram.. to understand the relationship.. in such case.. you have to have those relationship are important..
then you have to write program for it.... snowpark can do .. or you can also write python unless snowsight come up with some kind of UI for that.. and yes, If you want to manage snowflake more programatically.. you can watch my paid contents .. many folks don't know the power of snowpark... this 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... and make all SQL statement available for CI/CD... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=SPECIAL50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=SPECIAL35
it is in the description... and yes..I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Hi I keep getting this error File "C:\Users\anbest\OneDrive - Capgemini\Documents\Git\Snowpark_project\LoadData.py", line 57, in main put_result(file_element," => ",put_result[0].status) TypeError: 'list' object is not callable I tested the traverse func on its own, and it is picking up my file names location etc. seems to be the put_result causing issues
Not sure clear what kind of error you are getting... your result is not what the program expect.. so you need to check the typeof(object) and if it is list or not.
These contents are available in udemy.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/ 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/
wow superb you covered almost all the things required to survive as an data engineer in industry 🙂🤐
Thanks for your note...
If you want to manage snowflake more programatically.. you can watch my paid contents .. many folks don't know the power of snowpark... this 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... and make all SQL statement available for CI/CD...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=SPECIAL50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=SPECIAL35
Great video. It would have been wonderful if it also contained
1. how to do end to end CICD
2. How to setup pipeline dependency between data ingestion tool and snow flake task ( assuming we can bundle up all the loading steps you have covered in this video into a snowflake task)
Apologies if you have already got these covered else where , if so please direct me. Many Thanks
Appreciate the time and effort you've put in to create this tutorial.
Great video. It would have been wonderful if it also contained
1. how to do end to end CICD
2. How to setup pipeline dependency between data ingestion tool and snow flake task ( assuming we can bundle up all the loading steps you have covered in this video into a snowflake task)
Apologies if you have already got these covered else where , if so please direct me. Many Thanks 1:21:30
Glad you linked the content and your request for CI/CD is noted. The CI/CD is not yet covered in my video.
Sir , We appreciate your efforts and knowledge sharing.
Glad you liked the content.. thank you so much for you note Anil...
Awasome just awesome 👍
Very comprehensive, thank you :)
I don't see 2 years data in the gitlab link but just one month, Jan 2020. Am I missing looking at the wrong location?
thanks a lot. great work...👏
Thank you for informative session. Can’t we achieve same solution through snowflake sql baes queries/stored procs
yes, you can do it... and snowpark is nothing but SQL generator with current version .. may be you can watch this video.. what it is and what it is not
1. th-cam.com/video/-awSPRW9AOY/w-d-xo.html (What is snowpark)
2. th-cam.com/video/7tToBddZ_is/w-d-xo.html (What is NOT snowpark)
Hi, thanks for this end to end project, Where can I find complete dataset which you used in this video ?
complete data set is too big.. the desc has the link that has limited data.
-----
and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in udemy.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/
great tutorial. thanks!
Thanks for Everything...
Always welcome
Thanks for you effort. A question: how would you productionise the entire flow. Should your python code be converted to a stored procedure?
Yes, exactly.. or it needs a runtime environment outside of snowflake and it has to be scheduled with some kind of scheduler.
@@DataEngineeringCan we have a video on that as well? Like how can .py first become a stored procedure 😅😅😅
HI , i was tryin to run the getting the csv etc. files on the internal stage , I even get the output of the directory but Im not able to see the data as a result ! pls do help have been trying to rectify for hours but got no clue! thanks
Not sure which step you are talking about... if you can give me a timestamp, it will be helpful or you share a screenshot to my instagram account (instagram.com/learn_dataengineering/)
@@DataEngineering-Its Step4,I am using windows 11 os and struggling to run the python script also tried using put command from snow SQL, but nothing is working please advise and help
great video, thanks, would it be possible to share the data used in this project
Thanks for everything. U helped a lot ❤!
May i ask if u can make videos on the exception handling and error logging? E.g. one of the csv has an additional column.
Another example is when loading data into the internal stage, wifi connection failed and how to resume the job?
Thanks bro! :)
if file are not loaded, and if you try to load, the load will ignore them...
Interesting video and great explanation.
Glad you liked the content...
Hi, thank you for sharing the project. Where to find the python code ?
Gitlab has data files. Please advise
In the description
Appreciate your work
Thank you so much 😀
Really great project 👌
glad you liked it..
Quick question: I am on Part 4 but for some reason I downloaded the data to my computer but it does not show when I try to run it through Snowflake. Is there a reason for that?
could you please provide additional detail, not able to understand the issue. pls attach a time stamp or share a screenshot via my instagram account.
Thank you very much but Step 4.2 Loading Data To Internal Stage Using Snowpark File API is not working for me. I run the code but no data is there.
Can you please tell me how to update row in snowflake_sample_data.tpch_sf100.orders??? getting error - "Object 'ORDERS' does not exist or not authorized."
that is a shared object, you can not update it.
Thank you for sharing such a good content. I should say you are a rockstar in Snowflake world. I have a question. In this case, since there are lot of Data frames created in snowpark-python scripts and running the code from local machine ,does it consume local system storage/compute or push everything to the Snowflake storage/Compute? Thank you in advance!
thanks for your note..
when you perform an operation using dataframe in snowflake, it uses snowflake's compute power. When you pull data to your location machine..in that case.. it uses your local compute...
Thank you for your quick response. If i would like to push everything to the snowflake storage and compute, how should we do it? How should we register the snowpark-python programs in snowflake database and run/debug it(Instead of Stored proc route) ? is is really possible? May be having a separate video might help@@DataEngineering
Watch ch-08 from this snowpark playlist.. and you would understand how to deploy it (playlist link th-cam.com/play/PLba2xJ7yxHB4yPg3pUrobdzeMxk4mP24S.html)
@@DataEngineeringThank you. I already watched it. Does that mean we should test it locally first and then deploy on SF sandbox. I am looking for options if we can develop,test, debug and deploy directly in the SF sandbox itself? Is it possible? Any insight?
Hi Bro,
While going through the course, I found out that not all the data is provided in the gitlab link as well as the exchange_rates.csv at 50:00 . The exchange rate column is null for all rows after moving the file into curated stage. Could you update the link with all the files as mentioned in the course? Thanks
The dataset in the gitlab is only for 30 days and the forex date is also different for what you have. Can you please crosscheck the files and share the correct ones?
the data size is too big and that's wh only 30 days data set is provided.
Sir can you please do one end to end project in snowsql as well. that will be very beneficial for us.
snowsql is just a cli tool.. you mean Snowflake SQL? if so.. watch ch-19 from my snowflake tutorial .. the end to end flow is covered using SQL.
curated to consumption code is not working properly.
There is no primary key in snowflake. Then why do we use Primary and Foreign key here ? Can you please explain.
When you connect to any BI tool like PowerBI..they need these relationship.. and can build the model for slice and dice... and if you have to draw the ER diagram.. to understand the relationship.. in such case.. you have to have those relationship are important..
sir...where i can get all these code ?
Check description
How we can do reverse engineering without third party tool
then you have to write program for it.... snowpark can do .. or you can also write python unless snowsight come up with some kind of UI for that..
and yes, If you want to manage snowflake more programatically.. you can watch my paid contents .. many folks don't know the power of snowpark... this 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... and make all SQL statement available for CI/CD...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=SPECIAL50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=SPECIAL35
Hello, How can I get the source code of this project?
it is in the description... and yes..I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge..
These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35
Hi
I keep getting this error
File "C:\Users\anbest\OneDrive - Capgemini\Documents\Git\Snowpark_project\LoadData.py", line 57, in main
put_result(file_element," => ",put_result[0].status)
TypeError: 'list' object is not callable
I tested the traverse func on its own, and it is picking up my file names location etc. seems to be the put_result causing issues
Not sure clear what kind of error you are getting... your result is not what the program expect.. so you need to check the typeof(object) and if it is list or not.
Hi bro
pls share your query..
You have any udemy cource.Can you please send me gitlab repo or udemy course details
These contents are available in udemy.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command...
1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/
2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/
How to contact you brother
instagram.com/learn_dataengineering/
@@DataEngineering already sent a msg to you but no reply
gitlab link just have one month, data Jan 2020. Can you send us the proper link containing whole data file. @DataEngineering
Yes, if it hard to put so much of data in any platform.... that's why given only 1 month of data.