@Sanjay Kattimani Bro Awesome explanation... 👌 Using this explanation I have created project on my resume using my current ETL project business with additional transformation mapping rules... I can now start marketing my resume for snowflake positions... Thank you so much...
Hi Sanjay, I am following your video and tried to create event subscription as mentioned in video but i am getting error "The subscription is not registered to use namespace Microsoft.EventGrid". Could you please help me why i am getting this error?
This is due to unregistered Evengrid subscription. Follow the instructions in the following link to resolve the issue: docs.microsoft.com/en-us/azure/azure-resource-manager/templates/error-register-resource-provider
Thanks for the Video sanjay, this was very helpful. Can you please share the example, with multiple files if got loaded in BLOB Storage, how the snowpipe will behave then?
Thank you for the video. If I have an on prem oracle database as the source and the frequency with which the data changes is quite high (insert, update and delete) and the current requirement from the client is near real time data insights. so do you recommend using snowpipe to ingest data from azure(adls in this case) into snowflake tables. but what should be the load mechanism or tool or technology used for data ingestion into azure from the source database? Thanks in advance :)
You need to script out most of these tasks instead of using user interface. You can parameterize the scripts and automate most of these using python or other sctiping.
@@SanjayKattimani-tech TYSM .. could you also explain how to create audits like rec count check , tine of load , error capture n notification and other kind of audits . If you have any video already pls send the link. Thank you
Hello Sanjay - Could you Please explain the standard steps to build an ETL in snowflake from file extraction to load into DWH , where to use views , streams va merge. It would be really great !!
Hi Sanjay, Nice demonstration. I am facing issue, files are getting load when I am manually refreshing pipe. Seems auto ingest not working. Please suggest
Great video! No data is loaded into the same 'BANK_TRANSACTIONS' table if the same file is uploaded again after the first time, even if it is deleted and uploaded again. Any idea?
Great video, I have an additional question though. I want to store multiple in a container. File_1 should be copied into table_1 in snowflake File_2 should be copied into table_2 in snowflake etc. Please could you advise how I should approach this?
In your pipe definition, use pattern or the file name along with stage name. You can also run a copy command on stage if this is a one off table/data in your external stage
@@vivekpandey728 something like this - CREATE OR REPLACE pipe azure_data_pipe auto_ingest=TRUE AS COPY INTO table1 from @azure_stage) pattern='.File_1*.csv'; and another pipe for another type of file. You can use sub directory as well.
Hi, thanks for the video. A question for you...In the case of Azure Service Bus Queue, when messages get received what is the best way to get them into Snowflake? Triggering too many messages will create locking. What is the best path?
Great video Sanjay .. how can i load data from sap ecc tables into snowflake by staging it in azure adls? All cdc on source tables can be captured as files in adls and use snowpipe to write those in sequence into target snowflake table ?
You could use ADF to connect to SAP (docs.microsoft.com/en-us/azure/data-factory/connector-sap-table) and load it in to ADLSG2, you can then use snowflake streams & merge the data into your final tables.
I am getting this error message, if I gave the permission also. If possible can you please help me on this issue. "Pipe Notifications bind failure "Could not access queue:snowflakesnowpipes.queue.core.windows.net/snow-data-queue; Message: This request is not authorized to perform this operation using this permission. (Status Code:403; Error Code:AuthorizationPermissionMismatch)" erroe
Hi Sanjay, Thank you so much for this vidio. I followed the instruction and Setuped the process. Can you please also guide how to maintain Log file so when ever any new file ingested it make entry in the log table (Which is a table in Snowflake) like what is file name, When it get ingested and whether process get success or failed. Also i wants to implement email notifcation whether it sucess or fail.
Thats interesting, i hardly see use cases of crossing the cloud providers. but it could be possible, check with snowflake reps if they need to change some settings to allow cross cloud connection. You could also use a kafka connector to push the data.
Getting an error while creating notification integration in snowflake SQL execution internal error: Processing aborted due to error 370001:1831050371; incident 3767626.
It’s temporarily disabled by TH-cam because they suspected it was being operated by underage kid. Expected to be back in 2 days. I am glad he earned some ardent followers 😇
Sanjay Thanks for wonderful video on snow pipe set up from Azure and automation. Is there any issue to use storage integration in stage instead of generating token? I am unable to automate using storage integration method, although with refresh data is loaded but auto ingest is not happening.
Thanks for the video, very nice explanation. If we have to extend this ingestion pipeline for data quality checks say using Data Profiler (as there were 2 records in the example provided) and loop back the records in to the pipeline after correction, how would you suggest I proceed?
Sanjay, Thanks for posting this excellent overview. You mentioned that it took about a minute to load the file from Azure storage. These are very small files so I was wondering if the latency is in the queue/event time, transfer, or load? Do you have any knowledge of using SnowPipe at scale? Thanks!
That 1 minute is usually on Azure side to detect the new file, notify the stage and everything in between. Once the stage receives notification snowpipe usually take very little time (usually under a second for small files). That one minute is from the moment you uploaded your file to data showing up in snowflake table. But this time can vary a little based on vpn config and various other reasons. It has no issues with scale. Even if you upload thousands of files every minute, it will handle it without issues.
Great video. Typically in projects we use a Azure SF Storage Integration instead of using a SAS key while creating a Stage.
yup, if Integration is there , then better use it rather than SAS keys which is temp
@Sanjay Kattimani Bro Awesome explanation... 👌
Using this explanation I have created project on my resume using my current ETL project business with additional transformation mapping rules... I can now start marketing my resume for snowflake positions... Thank you so much...
Hey Sanjay, really great video, well explained. Thank you.
Thank You Very Much Sanjay for the excellent session.
Thank you so much Sanjay for this awesome video
Hi Sanjay,
I am following your video and tried to create event subscription as mentioned in video but i am getting error "The subscription is not registered to use namespace Microsoft.EventGrid". Could you please help me why i am getting this error?
Please check with your azure administrator if it is not your personal account , you may need elevated access on event grid usage.
This is due to unregistered Evengrid subscription. Follow the instructions in the following link to resolve the issue: docs.microsoft.com/en-us/azure/azure-resource-manager/templates/error-register-resource-provider
Thanks for the Video sanjay, this was very helpful. Can you please share the example, with multiple files if got loaded in BLOB Storage, how the snowpipe will behave then?
Thank you for the video. If I have an on prem oracle database as the source and the frequency with which the data changes is quite high (insert, update and delete) and the current requirement from the client is near real time data insights. so do you recommend using snowpipe to ingest data from azure(adls in this case) into snowflake tables. but what should be the load mechanism or tool or technology used for data ingestion into azure from the source database? Thanks in advance :)
hi, how can we know the name of the snowpipe while loading
Thanks Sanjay for your explanation. Could you please explain how to automate the entire process.
You need to script out most of these tasks instead of using user interface. You can parameterize the scripts and automate most of these using python or other sctiping.
@@SanjayKattimani-tech TYSM .. could you also explain how to create audits like rec count check , tine of load , error capture n notification and other kind of audits . If you have any video already pls send the link. Thank you
Hello Sanjay - Could you Please explain the standard steps to build an ETL in snowflake from file extraction to load into DWH , where to use views , streams va merge. It would be really great !!
Hi Sanjay,
Nice demonstration.
I am facing issue, files are getting load when I am manually refreshing pipe. Seems auto ingest not working.
Please suggest
Great video! No data is loaded into the same 'BANK_TRANSACTIONS' table if the same file is uploaded again after the first time, even if it is deleted and uploaded again. Any idea?
Hi Sanjay - why storage integration is not done ? is it already done before ?
Great Video
What happenes to files present in stage after data loading? Also if we refresh pipe after data loading will it reload the data?
snowflake will not reload the same files, you may also set on the copy statement the option to purge the file after the succesful data load
Clear explanation.
Great video, I have an additional question though. I want to store multiple in a container.
File_1 should be copied into table_1 in snowflake
File_2 should be copied into table_2 in snowflake
etc.
Please could you advise how I should approach this?
In your pipe definition, use pattern or the file name along with stage name. You can also run a copy command on stage if this is a one off table/data in your external stage
@@SanjayKattimani-tech can you please elaborate or give syntax that would really help, thanks in advance
@@vivekpandey728 something like this - CREATE OR REPLACE pipe azure_data_pipe
auto_ingest=TRUE
AS
COPY INTO table1
from @azure_stage)
pattern='.File_1*.csv';
and another pipe for another type of file. You can use sub directory as well.
What if there's a requirement to archive files after they've been processed.
You got your answer for this?
Hi, thanks for the video. A question for you...In the case of Azure Service Bus Queue, when messages get received what is the best way to get them into Snowflake? Triggering too many messages will create locking. What is the best path?
Hi Sanjayji, could you please help me on how to call snowflake procedure in ADF and how pass input parameter dynamically?? Thanks in Advance .
Great video Sanjay .. how can i load data from sap ecc tables into snowflake by staging it in azure adls? All cdc on source tables can be captured as files in adls and use snowpipe to write those in sequence into target snowflake table ?
You could use ADF to connect to SAP (docs.microsoft.com/en-us/azure/data-factory/connector-sap-table) and load it in to ADLSG2, you can then use snowflake streams & merge the data into your final tables.
Nice 👍 video !!
I am getting this error message, if I gave the permission also. If possible can you please help me on this issue. "Pipe Notifications bind failure "Could not access queue:snowflakesnowpipes.queue.core.windows.net/snow-data-queue; Message: This request is not authorized to perform this operation using this permission. (Status Code:403; Error Code:AuthorizationPermissionMismatch)" erroe
When I executed the last copy statement. I got this error message.
Hi Sanjay, Thank you so much for this vidio. I followed the instruction and Setuped the process. Can you please also guide how to maintain Log file so when ever any new file ingested it make entry in the log table (Which is a table in Snowflake) like what is file name, When it get ingested and whether process get success or failed. Also i wants to implement email notifcation whether it sucess or fail.
Sanjay, Can Snowpipe load from S3 to Snowflake in Azure without any intermediate storage in Azure blob?
Thats interesting, i hardly see use cases of crossing the cloud providers. but it could be possible, check with snowflake reps if they need to change some settings to allow cross cloud connection. You could also use a kafka connector to push the data.
Very good Video
Getting an error while creating notification integration in snowflake SQL execution internal error: Processing aborted due to error 370001:1831050371; incident 3767626.
Excellent!! thanks
*Why did you close Rishab Teaches Tech ?*
It’s temporarily disabled by TH-cam because they suspected it was being operated by underage kid. Expected to be back in 2 days. I am glad he earned some ardent followers 😇
@@SanjayKattimani-tech
I was promoting the channel... but i got several feedbacks that it is closed ... then recognized that is really closed..
@@theautomationchannel6920 thanks for promoting, it will be back in 2-3 days. 🤞
Its back to life :-) Thanks.
@@theautomationchannel6920 Here is the whole story - www.linkedin.com/pulse/learning-journey-12-year-old-international-python-event-kattimani/
Sanjay
Thanks for wonderful video on snow pipe set up from Azure and automation.
Is there any issue to use storage integration in stage instead of generating token?
I am unable to automate using storage integration method, although with refresh data is loaded but auto ingest is not happening.
sir did you find the solution,because i am also facing same issue!?
Thanks for the video, very nice explanation. If we have to extend this ingestion pipeline for data quality checks say using Data Profiler (as there were 2 records in the example provided) and loop back the records in to the pipeline after correction, how would you suggest I proceed?
instablaster
WhatsApp group is full can you give us some other link where we can join
Sanjay, can you please do a video on loading data from Azure synapse to Snowflake
Hmm, i wonder about this use case.. Are you or your client moving out of Azure synapse and moving to Snowflake?
@@SanjayKattimani-tech yes we are moving to snowflake
Sanjay, Thanks for posting this excellent overview. You mentioned that it took about a minute to load the file from Azure storage. These are very small files so I was wondering if the latency is in the queue/event time, transfer, or load? Do you have any knowledge of using SnowPipe at scale? Thanks!
That 1 minute is usually on Azure side to detect the new file, notify the stage and everything in between. Once the stage receives notification snowpipe usually take very little time (usually under a second for small files). That one minute is from the moment you uploaded your file to data showing up in snowflake table. But this time can vary a little based on vpn config and various other reasons. It has no issues with scale. Even if you upload thousands of files every minute, it will handle it without issues.
Can u make new videos pls
In my company your vedios integrated for self course