Learn Kafka in 10 Minutes | Most Important Skill for Data Engineering
ฝัง
- เผยแพร่เมื่อ 14 ต.ค. 2024
- Check Out My Data Engineering Bootcamp: bit.ly/3yXsrcy
USE CODE: COMBO50 for a 50% discount
Learn more about courses here - datavidhya.com...
Code used in video - github.com/dar...
Apache Kafka is one of the most important skills to learn as a Data Engineer,
Top companies use Kafka to process real-time data streams and give you a better experience, let's quickly explore how it works!
DISCLAIMER: I understand a 10min video does not teach you everything but getting a quick overview can help you navigate courses/books/docs easily when you get started
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
🎯Twitter - / parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Tags ✨
✨ Hashtags ✨
#dataengineer #apachekafka #aws
Don’t forget to LIKEEEE
Never... Never Forget to 👍
What's the prior knowledge should have to learn Apache Kafka?
For tech enthusiasts, one common issue is falling asleep while watching learning videos. However, after watching one such video, I found this gem. Thanks Man !
I really love the way you explain. Especially you showed it in both Jupyter notebook and terminal.
No waste of time straight to the point. Loved it.
Very good content, looking for more videos...
More to come!
Great content to understand what Kafka is and it's components and how they work together
Thanks for the video. It was useful.
You're welcome
falling in love with your content quality 🔥
Best one, Please explain indepth architecture in Kafka!!! Love to hear more!
Very soon!
@@DarshilParmar Hey Darshil, did you release this later?
Top video! in such short time, you explained it clearly, Darshil!
Glad you liked it!
Great stuff, could you also explain how to handle DLQ messages and consumer lag in sink connector
Will create some tutorial on this
This Explanation was best , Can we get same for apache nifi?
Great content vro.. build a video series of the format. Killed it 🫡
Thank you
Hello Darshil is great to watch this content and learn some new concept each day.
Can we get the same for Apache Hadoop , Apache Nifi the more reliabe tools which are ongoing used in Data engineering Projects
Thank you!
great content and i have been looking at your content for years now, i still learning but i feel like something is missing, like a real life project , real time processing where one can use kafka,pyspark , orchastrated by airflow and the data goes in a medallion achitecture then transformed by dbt and then deployed in production using ci/cd ,and then monitor the dags using promethus and grafana ,something of that sort and then mayb deploy it on gcp or aws
love this content, keep making it sir ❤
More to come!
Great overview. Well done!
Hi Darshil
First of all , Great Content !
I am trying to replicate the steps as the video , and it works fine from the terminal. Messages sent from the producer is displayed at consumer end.
But when I'm trying to run it with the producer/consumer scripts , the consumer does not receives the messages.
Any idea why ? Both containers(kafka & zookeeper) are on the same bridge n/w on same subnet.
Go deeper into architecture and implementation of Kafka. Great video!
Thanks, will do!
Please consider this request from my side as well.
@@ManishSharma-fi2vr Sure
Yes, please go deeper
For data engineers scope is mostly limited to consumption side and rarely you write to topics as producers (for specific use cases) . Most of the complexity of setting up kafka and micro service should be handled by source systems. Consumer should only worry about consuming those events and setting up pipelines which is pretty easy.
100%
Nice information thanks ❤
Welcome
Wow I was just looking for this !
Glad you found it
Great content man.
when this course or combo will come on it ?, i just want to ask something regarding course buy, can we connect ?
Hi bhaiya
Needed a laptop in college and programming i started learning data science and ml but dont do gaming so i didnt take any GPU laptop
I have 2 options
First lenovo ideapad pro 5 with 14" ultra 9 H processor 32 gb ram 1 tb ssd and 84 wh battery
Second is lenovo yoga slim 7i with ultra 7 processor 32 gb ram 1 tb ssd and 65 wh battery
Both price same
Which one should i go for?
Thank you ❤❤ It was really helpful 🎉.
You're welcome 😊
Can you please come up with Kafka end to end series for your people...
will be launching course on this soon
@@DarshilParmar i am thinking to buy, when it will come please let me know its very urgent !!!!!!
Guys remember if you wanna learn kafka, go to documentation section and read it
All time best resource
make a full courese on kafaka and snowflake integration
I already have on snowflake
@@DarshilParmar but kafka integration kind of
That will come soon
if i started learning for now than there are very less data engineering internships(if one has data internship it is easy to get a job ,and that person also get hand on experience ) ,,,,so should i do data analyst first only for internship and after that i will study for data engineering for job because it is also a good way , and i need to do an internship because it is in our college criteria , they do not support much if you do not have done internship??
nice video on kafka
Start with any data field that you get, foundation skills are same for all
Hey let's connect and talk more about it
@@devparmar7420 ok
@@devparmar7420 ok
@@devparmar7420 share me your email are you also an student
Epic 🥰
I am facing issue while installing pydoop which is needed for me to write data from kafka topics to hadoop. Please can you help me put in installing pydoop.
What about Google Cloud Pub/sub. Both Kafka and pub/sub are same
Yes it’s similar
We need a full tutorial on Kafka
on it!
3:32 how to create this map?
I have done bcom want to enter in it what should i do mtech vs mca
MTech is good option you’ll get in-depth understanding
What is the work of the zookeeper here?
Manage resources and work on coordination/synchronization
We are running multiple machines
After typing command docker-compose up i am getting eeror like incorrect username and password
you will have to login to docker using cli, just google about it
I have some queries what is the way to contact you?