Learn Kafka in 10 Minutes | Most Important Skill for Data Engineering
ฝัง
- เผยแพร่เมื่อ 29 มิ.ย. 2024
- Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?pl...
USE CODE: COMBO50 for 50% off
Learn more about courses here - datavidhya.com/courses
Code used in video - github.com/darshilparmar/kafk...
Apache Kafka is one of the most important skills to learn as a Data Engineer,
Top companies use Kafka to process real-time data streams and give you a better experience, let's quickly explore how it works!
DISCLAIMER: I understand a 10min video does not teach you everything but getting a quick overview can help you navigate courses/books/docs easily when you get started
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
🎯Twitter - / parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Tags ✨
✨ Hashtags ✨
#dataengineer #apachekafka #aws
Don’t forget to LIKEEEE
Never... Never Forget to 👍
Great overview. Well done!
Great content to understand what Kafka is and it's components and how they work together
Top video! in such short time, you explained it clearly, Darshil!
Glad you liked it!
Very good content, looking for more videos...
More to come!
Wow I was just looking for this !
Glad you found it
Thanks for the video. It was useful.
You're welcome
Best one, Please explain indepth architecture in Kafka!!! Love to hear more!
Very soon!
love this content, keep making it sir ❤
More to come!
Thank you ❤❤ It was really helpful 🎉.
You're welcome 😊
Great content vro.. build a video series of the format. Killed it 🫡
Thank you
For data engineers scope is mostly limited to consumption side and rarely you write to topics as producers (for specific use cases) . Most of the complexity of setting up kafka and micro service should be handled by source systems. Consumer should only worry about consuming those events and setting up pipelines which is pretty easy.
100%
Great stuff, could you also explain how to handle DLQ messages and consumer lag in sink connector
Will create some tutorial on this
This Explanation was best , Can we get same for apache nifi?
Nice information thanks ❤
Welcome
great content and i have been looking at your content for years now, i still learning but i feel like something is missing, like a real life project , real time processing where one can use kafka,pyspark , orchastrated by airflow and the data goes in a medallion achitecture then transformed by dbt and then deployed in production using ci/cd ,and then monitor the dags using promethus and grafana ,something of that sort and then mayb deploy it on gcp or aws
Hello Darshil is great to watch this content and learn some new concept each day.
Can we get the same for Apache Hadoop , Apache Nifi the more reliabe tools which are ongoing used in Data engineering Projects
Thank you!
Hi Darshil
First of all , Great Content !
I am trying to replicate the steps as the video , and it works fine from the terminal. Messages sent from the producer is displayed at consumer end.
But when I'm trying to run it with the producer/consumer scripts , the consumer does not receives the messages.
Any idea why ? Both containers(kafka & zookeeper) are on the same bridge n/w on same subnet.
Can you please come up with Kafka end to end series for your people...
will be launching course on this soon
Go deeper into architecture and implementation of Kafka. Great video!
Thanks, will do!
Please consider this request from my side as well.
@@ManishSharma-fi2vr Sure
Yes, please go deeper
What about Google Cloud Pub/sub. Both Kafka and pub/sub are same
Yes it’s similar
Hi bhaiya
Needed a laptop in college and programming i started learning data science and ml but dont do gaming so i didnt take any GPU laptop
I have 2 options
First lenovo ideapad pro 5 with 14" ultra 9 H processor 32 gb ram 1 tb ssd and 84 wh battery
Second is lenovo yoga slim 7i with ultra 7 processor 32 gb ram 1 tb ssd and 65 wh battery
Both price same
Which one should i go for?
if i started learning for now than there are very less data engineering internships(if one has data internship it is easy to get a job ,and that person also get hand on experience ) ,,,,so should i do data analyst first only for internship and after that i will study for data engineering for job because it is also a good way , and i need to do an internship because it is in our college criteria , they do not support much if you do not have done internship??
nice video on kafka
Start with any data field that you get, foundation skills are same for all
Hey let's connect and talk more about it
@@devparmar7420 ok
@@devparmar7420 ok
@@devparmar7420 share me your email are you also an student
make a full courese on kafaka and snowflake integration
I already have on snowflake
@@DarshilParmar but kafka integration kind of
That will come soon
I have done bcom want to enter in it what should i do mtech vs mca
MTech is good option you’ll get in-depth understanding
What is the work of the zookeeper here?
Manage resources and work on coordination/synchronization
We are running multiple machines
We need a full tutorial on Kafka
on it!
Guys remember if you wanna learn kafka, go to documentation section and read it
All time best resource
I have some queries what is the way to contact you?