Roshan , it's for generating public & private key using openssl , as I am using widnows machine , I can't directly execute the commands related to openssl.. for that I needed one linux machine , so launched the ec2 machine to perform that step only ... If you have wsl installed in windows machine or using ubuntu or linux machine , ec2 not needed..
Hi @chrisbrad3650, it's there in description box, anyway, sharing here too FYR -- github.com/SatadruMukherjee/Data-Preprocessing-Models/blob/main/kafka%20snowflake%20integration.txt
Sure Gopi Nath, you can do that , in-fact I have created the private and public key files using wsl only in this video -- th-cam.com/video/j-XEcZfW-WQ/w-d-xo.htmlsi=2laqDdSsXpnXltEr
Hi if i'm getting avro messages in my kafka topic and loading this avro messages in a snowflake table (into two columns RECORD_CONTENT and RECORD_METADATA), how can i translate this avro files into a columnar table? using flatten function? thanks
Here you have used Snowpipe internally to ingest Kafka messages. How to do the same with Snowpipe streaming? Can we get rid of the Snowflake connector jar file by using this method? Many sensitive data systems will not allow to place jars in their lib folder, so is there a way where in we just configure the properties file and achieve Snowpipe streaming?
Just one questions i have Looks like in this case kafka connect is running on broker node (in this case your local windows machine) and is this possible to run kafka connect away from broker ?
Hello Poorab Joshi, I know it's bit late , but if you are still looking for the solution , then this video is dedicated to you -- th-cam.com/video/4wULNXlXVwA/w-d-xo.html Happy Learning
Hey Ramu, great playlist. I was able to get started on Kafka locally. I am still trying to figure out how I can read files from a folder into a Kafka cluster . Are you able to shed some light on that please ?
Thank you for the video. It looks like Snowflake is using Snowpipe to move the underlying data. Is it possible to use Snowflake Streaming Ingest SDK for Python to achieve the same?
i have a question ? can I insert in multiples snowflake. schema with one connection or how can i create multiple connections to the same database but differents snowflake.schemas? thanks
Thanks for great tutorial. Question: How do I save my messages to a postgresql database? Is it possible to create the messages via a POST method using POSTMAN?
Hi I am getting errors while running the connector. ERROR [SF_KAFKA_CONNECTOR] Validate Error msg:[SF_KAFKA_CONNECTOR] Exception: Failed to prepare SQL statement Error Code: 2001 Detail: SQL Exception, reported by Snowflake JDBC Message: SQL compilation error: Object does not exist, or operation cannot be performed. Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s): Snowflake.database.name database does not exist But the database is there.
Thank you Bacche Man ke sacche ! For mysql , if you want to implement streaming pipeline for CDC , you can check these 2 links (used kinesis , you can use kafka instead of that)-- CDC Pipeline from MySQL RDS to Snowflake (from scratch) th-cam.com/video/RDxGVZH36fo/w-d-xo.html&feature=shares Adding RESUME Capability while working with MySQL Binlog Streamer th-cam.com/video/7CeS1OgJ624/w-d-xo.html&feature=shares Comping to kafka connect for MYSQL or cassandra , noted in backlog , will be uploaded in future , stay tuned! Happy Learning
I'm still not getting the reason to include ec2 instance here? Would be great if you can help on this. Thank you
Roshan , it's for generating public & private key using openssl , as I am using widnows machine , I can't directly execute the commands related to openssl.. for that I needed one linux machine , so launched the ec2 machine to perform that step only ...
If you have wsl installed in windows machine or using ubuntu or linux machine , ec2 not needed..
I got no any error but data not load into DB ? why is that ?
Please provide notepad file used in this . I am not able to find it.
Hi @chrisbrad3650, it's there in description box, anyway, sharing here too FYR -- github.com/SatadruMukherjee/Data-Preprocessing-Models/blob/main/kafka%20snowflake%20integration.txt
Can we do the SSL connection through WSL(Windows Sub system for Linux)? instead of launching EC2 instance
Sure Gopi Nath, you can do that , in-fact I have created the private and public key files using wsl only in this video -- th-cam.com/video/j-XEcZfW-WQ/w-d-xo.htmlsi=2laqDdSsXpnXltEr
Hi if i'm getting avro messages in my kafka topic and loading this avro messages in a snowflake table (into two columns RECORD_CONTENT and RECORD_METADATA), how can i translate this avro files into a columnar table? using flatten function? thanks
Here you have used Snowpipe internally to ingest Kafka messages.
How to do the same with Snowpipe streaming? Can we get rid of the Snowflake connector jar file by using this method?
Many sensitive data systems will not allow to place jars in their lib folder, so is there a way where in we just configure the properties file and achieve Snowpipe streaming?
Can u share for mssql to kakfa connectoe demo pls
when connecting with putty i get fatal error .. can you pls help me
Just one questions i have
Looks like in this case kafka connect is running on broker node (in this case your local windows machine) and is this possible to run kafka connect away from broker ?
Hello Poorab Joshi, I know it's bit late , but if you are still looking for the solution , then this video is dedicated to you --
th-cam.com/video/4wULNXlXVwA/w-d-xo.html
Happy Learning
can you make a video for the reverse pattern ? I mean to say , from snowflake to kafka. CDC from snowflake to kafka queue.
Thanks
Hello Pankaj kumar Bhoi, not yet , you can refer this -- medium.com/snowflake/streaming-data-from-snowflake-to-kafka-ed76ce0400c2 Happy Learning
Hey Ramu, great playlist. I was able to get started on Kafka locally.
I am still trying to figure out how I can read files from a folder into a Kafka cluster . Are you able to shed some light on that please ?
Thank you for the video. It looks like Snowflake is using Snowpipe to move the underlying data. Is it possible to use Snowflake Streaming Ingest SDK for Python to achieve the same?
Thank you for the video. it was very informative. 🙏
Glad to know that you found this video informative __Gangst3r__! Happy Learning :-)
i have a question ? can I insert in multiples snowflake. schema with one connection or how can i create multiple connections to the same database but differents snowflake.schemas? thanks
Thanks for great tutorial. Question: How do I save my messages to a postgresql database? Is it possible to create the messages via a POST method using POSTMAN?
yes is it possible
Very well explained 👍
Thank you Vivek Patel! Happy Learning :-)
instead of AWS instance, we can try with windows DSL
Yes , windows ssl can also be used to generate the private & public key files ..
Hi
I am getting errors while running the connector.
ERROR [SF_KAFKA_CONNECTOR] Validate Error msg:[SF_KAFKA_CONNECTOR] Exception: Failed to prepare SQL statement
Error Code: 2001
Detail: SQL Exception, reported by Snowflake JDBC
Message: SQL compilation error:
Object does not exist, or operation cannot be performed.
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
Snowflake.database.name database does not exist
But the database is there.
Your playlist of KAFKA is really awesome, can you make the same for streaming data pipeline for other db such as mysql, cassandra , Thanks again😃
Thank you Bacche Man ke sacche ! For mysql , if you want to implement streaming pipeline for CDC , you can check these 2 links (used kinesis , you can use kafka instead of that)--
CDC Pipeline from MySQL RDS to Snowflake (from scratch)
th-cam.com/video/RDxGVZH36fo/w-d-xo.html&feature=shares
Adding RESUME Capability while working with MySQL Binlog Streamer
th-cam.com/video/7CeS1OgJ624/w-d-xo.html&feature=shares
Comping to kafka connect for MYSQL or cassandra , noted in backlog , will be uploaded in future , stay tuned! Happy Learning
@@KnowledgeAmplifier1 thanks and God bless you 🙏