Kafka in 100 Seconds
ฝัง
- เผยแพร่เมื่อ 5 ก.พ. 2025
- Apache Kafka is a distributed event streaming platform used to handle large amounts of realtime data. Learn the basics of Kafka in this quickstart tutorial.
#programming #datascience #100SecondsOfCode
💬 Chat with Me on Discord
/ discord
🔗 Resources
Kafka Homepage kafka.apache.org/
Kafka Github github.com/apa...
RabbitMQ in 100 Seconds • RabbitMQ in 100 Seconds
🔥 Get More Content - Upgrade to PRO
Upgrade at fireship.io/pro
Use code YT25 for 25% off PRO access
🎨 My Editor Settings
Atom One Dark
vscode-icons
Fira Code Font
🔖 Topics Covered
What is Apache Kafka?
Who created Apache Kafka?
What is Kafka used for?
How do large apps handle streaming data?
Apache Kafka basic examples
The thumbnail is unexplainably aesthetic, please dont stop this series , always learning something new from these
has the aethetic of the japanese prefecture flags
@@SpektralJoOsaka++
@@2ku4 yeah it looks very similar to osakas symbol
That’s the Kafka logo?
Aesthetic is not an adjective
hey this is not the kafka I am searching for, but thank you fireship
bro’s looking for stellaron hunters☠️
Hey man, love your videos, but a small correction for what you mention at 1:45
"Kafka guarantees that any consumer of a given topic will always read the event in the exact same order"
That's true but at partition level. A topic is made of partitions, and a topic with 1 partition will guarantee that. But as soon as you need parallel consumption you need more partitions (one per concurrent consumer). When having multiple partitions, a single, non-concurrent consumer will get the events of any partition in the proper order but events from different partitions may be consumed with different interleaving
This is why it's so important to put a proper key for your messages, as it's the default value used for sending your message to different partitions (different events with the same key will always end up in the same partition)
Again, love your content, you explain super great. But I thought that that small point had to be addressed
😄
Yes that's an important detail to understand: order is guaranteed across one partition and all messages sharing a key are stored on the same partition, hence order is guaranteed for all messages sharing a key.
But order isn't guaranteed on a whole topic.
Thanks for the clarification, that's an important detail.
@@Fireship pin the comment my man
"small correction"
@@lonkhoi6764 humble man
This series never fails to teach me about things I didn't know I needed to know about.
Everyone is gangsta until you have to handle reset flow
Reset flow? What is that
@@HelloWorld-fg2nm when there is a failed process in Kafka, ideally you have to rollback every process that has been succeded before. Just like rolling back query in ORM. That is why you need separate flow for reset
Im sorry if you are confused by what im saying. Im still learning english
@@farhansangaji5029you can have a windowing topic for these cases and do reconciliation
so true!
As someone who works for Confluent and has worked on Kafka for some time now, I absolutely love your video. 🔥
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
I've watched this before doing the quickstart on apache kafka website, to have a general idea of what it was, and ended up with a really abstract idea of it. Then after doing the quickstart, and playing with it a little bit, I re watched this video and now it felt like the perfect summary, all became crystal clear!
I'll encourage anyone who feels like it is too abstract or is bit confused to follow similar steps :)
I'd love a 100 seconds on a CI tool like Jenkins, your videos are always so informative :)
Leeroy Jenkings
You read my mind, I’ve been hoping you’d cover Kafka for months now!
I didn't think I'd be interested in Kafka but I'm glad I watched. You're like a tech magazine, broadening my horizon.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
Good time to be a coder and a Honkai star rail fan.
Lmao fr
let's goooo
I always learn something new from these vids they're truly great
Then let's hope he does a video on "they're vs their"
@@Djulio vs vs. vs.
@@Djulio why you so smart
Wow. I started working on Kafka and had no clue what it was. After months got a basic idea but this video makes it so helpful
excellent timing! Had kafka come up in a meeting at work last week and didn’t know anything about it.
The short video idea is straight genius.
As always, I leave your videos with more questions than answers. Thanks !
I'd love to see a quick into to Event Sourcing on this channel!
That would be useful.
I'd love that.
Kafka is perhaps the best options for it. You just store the message forever and ur done. I recommend book Designing Event-Driven Systems by confluent CTO. It's awesome except it's a bit too bias for Kafka...tho nothing like Kafka so u can't blame the man.
Do your own research mate if you really want it.
I would like to see it with EventStoreDB
I just started at a new position and needed to understand Kafka. Wow, Fireship keeps reading my mind
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
There's a different Kafka that's even better 👀
*sigh* HSR fans are everywhere
HERE'S TO EVERYONE WINNING THEIR 50/50 RAHHHHHHHHH 🗣‼🔥💯
One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.
What’s more is you can use consumer groups to split the workload if you have massive data to compute. And what is more is you can use different strategies to reassign partitions to new workers to increase ingestion throughput
you're really wonderful. ive been struggling to understand since 2 yrs and you made it clear in just 10 min
0:10
yesss, i was hoping that you will make this reference, and you did! yaay
also great video as always
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
Thanks for giving these overviews! I often share these as introductions for coworkers!
Great timing, we literally talked about Kafka in my Intro to Big Data Analytics class today!
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
These videos are fantastic. My background is embedded software and hardware design, but I enjoy learning new technologies. Lately I've wanted to understand the basics of Kafka, RabbitMQ, and others, and these videos are perfect for giving me a quick understanding of how they are used.
I have a theory about why Kafka the messaging system is named Kafka after Franz Kafka the writer.
'Das Schloss'(The Castle) is a Novel written by Franz Kafka in which he describes a castle having occupants structured as the ultimate bureaucracy.
Members of the Castle does not know each other and communicate via a paper based messaging system.
You don't want to deal with such a bureaucracy, but you certainly would want to design large complex systems like so: Independent services not knowing about each other communicationg via an asynchronous messaging system. I think that is why Kafka is named Kafka, although just a personal theory...
That particular writer was known for writing stories about enigmatic, baffling events where the characters have no idea what’s going on. Like one about a trial where the person on trial is not even told what the charges are.
And that's what I like about Kafka , you can literally read your own meaning to it, here a team of developers have used his stories to create a great service.
Pretentious nerds.
That's a way better explanation that "it's a system optimized for writing" especially considering Kafkas writing process beeing so chaotic and unorganized that he himself did not see the any meaning in his stories
2:20 in java, since you’re implementing single-method interfaces you can just make it a lambda and can omit the argument types
This is why you use IntelliJ for Java!
Love this style and kind of bite-sized informative video!
Thank you! I never understood the differences between the message broker systems and Kafka before. But the real time streaming example and the amount of throughput made it clear to me.
1:09 Rest in Peace party guy from that Radio Disney song from my childhood.
man I was searching for honking star rails kafka but stumbled on this gem, thanks dude
Nice video! Kafka is very in demand rn, big movement towards NRT. Keep it up !
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
My dear Jeff, we wait prolog in 100 seconds, we want it NOW.
Thought this was meme compilation about Kafka from Honkai Star Rail for a sec
this video was very Kafka-esque
These are the first vids to watch to lay down the overarching concepts of a technology. Nice!
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
The footage at 1:46 is probably the single best piece of stock footage out there
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
Much awaited, thanks!
Imagine a Kafka topic is like a multi-lane highway, where each lane (partition) has cars (messages) driving in a single direction.
*Order Guarantee:*
Within each lane (partition), cars (messages) drive one after the other in a strict sequence.
If you're observing just one lane, you'll always see cars in the exact order they entered that lane.
*Multiple Lanes (Partitions):*
If you're watching the whole highway (the entire topic with multiple partitions), cars from different lanes might cross you at slightly different times.
So, while cars within each lane are in order, across lanes, they might appear mixed up.
*Why Lanes (Partitions) Matter:*
More lanes mean more cars can drive simultaneously, allowing for faster traffic flow.
In Kafka terms, this means more consumers can read messages concurrently, leading to faster data processing.
*Choosing the Right Lane with a Key:*
When a car (message) enters the highway (topic), it needs to pick a lane (partition). This choice is based on the car's license plate number (message key).
Cars with the same license plate number will always choose the same lane. In Kafka, messages with the same key always go to the same partition, ensuring they're read in order.
In summary, while Kafka keeps the order of messages within each partition, when you have multiple partitions, the order of messages across them can be mixed. Choosing the right key for your messages ensures they land in the expected partition.
this channel is just amazing.. truly keep up the extremely good work.
The only channel keeping my attention span entertained longer than chat gpt
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
You should make a "Qt in 100 seconds" :3
its a very very popular native Desktop Development framework that can compile for nearly any platform and is written in C++
Many of your production software like earlier iterations of Photo Shop or After Effects, currenr Davinci Resolve, OBS, Krita and many more are written with Qt
and still holds strong as one of the best native GUI solutions (native, aka. no web related stuff)
Qt is also supported in python. There is also QML (QT markup lanugage I guess) which also supports running javascript.
@@mkhuzaima But if u are going to use JS for any desktop application, html is wayy better as it is open standard. Also, you can use webview and write ur gui in HTML CSS JS and use C++, Java, Python or any other favorite language for the main logic of the app.
As someone who worked with pyQt I agree
I doubt that any Adobe product (PS..) is made with QT - no Linux version.
I used to be a QT fanboy, but not anymore. Licensing, enough said. And seemingly lagging behind on technology (C++20.. ) and tooling (QTcreator needed).
Adobe uses their own internal tool
Great video! I would love to see a video of "Scala in 100 seconds" :)
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
Yeeesss
wants a long series of it, detailed description.
Prashant Sharma , if you are interested in learning Kafka in-depth , you can refer the kafka playlist in my channel , all topics are covered in detail with practical ...
I like how confident he says " I will see you in the next one"
He can see us, even though we can’t see him ...
Dear Fireship,
ily.
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
It also used with frameworks as Apache Flink or Apache Spark combined together for distributed event system such as streaming , application to alert ⚠ on some event and now with many machine learning Apps. Great explanation consise love u're 100s @fireship
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Nice! Please do Apache Spark next!
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
Finally, my weekly dose of Fireship is here.
As Fireship awoke one morning from uneasy dreams he found himself transformed in his bed into a gigantic insect
nice
Been waiting for this video for a while maybe do couple deep dives on kafka?
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
i was the crazy developer about 3 years ago for wanting to introduce Kafka in a project which was having load problems, great now Kafka is famous.
Please, do a video in beyond fireship about kafka :)
Just to you know, nice job
Thank you fireship for making this videos they are super useful😎😎
Indian
@@PawitSahare True
@@apijayxero I am also
1:09 aaron carter came out of the blue
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Excelente, me encanto la forma y toda la informacion, increible, muy bueno 🔥🔥🔥🔥🔥🔥
Excelente, me encanto la forma y toda la informacion, increible, muy bueno 🔥🔥🔥🔥🔥🔥
I cant wait to see what's next in this series
The Brazilian Central Bank uses Kafka in its PIX system, that makes money transactions between individuals or entities in real time between bank accounts in any Brazilian bank. If anything, that show how powerful Kafka can be (2 billion transactions per month, with almost R$1 billion - which is roughly US$200 million - moved between accounts in that period).
FYI most databases can handle 2 billion transactions a month so not sure that is a win. Good use case though for immutability.
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
@@JTWebMan for a financial application nothing is ever that simple, specially since it has become such a key service in day-to-day use by the general population. But I do understand your point
twitter handles like 2 billion transactions per day bro, kafka is more powerful than you think, lol
Kafka rerun in 2.0❤❤🔥🔥🔥 i wanna get her lc
Trueeeee, definitely pulling for her now that I have the chance to do so (lost pity to yanKING)
Good video to introduce one about Scala :D
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
omg this is exactly what I needed lol my new job requires Kafka. Thanks!
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
I've been waiting for this.
I couldn’t find a good kafka introduction video on TH-cam yesterday and today you posted it. It’s like you can read my mind 😂
Thanks, very useful!
Now I wait for the Celery in 100 Seconds video to add it to my CV
Thanks for info! Can you please do a full video to teach js ?
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
Ive been wanting an introduction to Kafka forever now
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
Nice comprehensive video! I wonder when you will show us some impactful showcase. Like how should i build kafka better?
will be happy to see prolog in 100 seconds😄
need more about microservices! 🚀🚀🚀🚀🚀🚀
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
Love this explanation!
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
Just when i needed to learn kafka.
Happy New Year 🎉🎊 👆👆 Thanks for Watching And Commenting, Text Me On 👆👆 WhatsApp We Have a Prize For you🎁
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
Sometimes I feel like these videos are created just to take me off the peak of my Dunning-Kruger curve.
Hey congratulation, you are in the top list - the results are in for 'State of Javascript' Survey
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Yesss we also need one about nifi!
Totally not here while searching for kafka Honkai star rail..😂😂😂
here i come
IM HERE
@@wellwhatdo1do 🙂
me searching for Arknights's Kafka.
Same
You read my mind sir!
great overview man
Request for GTK(specifically GTK4 and libadwaita). Its really hard to understand for absolute beginners. And also how to read its documentation and how the documentation are relevant for PyGTK, or Relm or GTK-rs. For example, its really confusing for devs who started by learning web technologies first and also its quite different from XAML based winapps. So it will be really be helpful if you guide us.
Well, it is hard to make a tutorial for GTK because you can code it in C, Python, Rust, etc, there are many language bindings. Also there are two ways to create GTK user interfaces, either programmatically by calling functions/classes, or decoratively in a XML file.
Agree, a Python GTK tutorial would be good
Iced is a far easier gui library to use with Rust. It may not be as full fledged, but the way it's designed is easier to understand, even without a tutorial vs gtk4rs with a tutorial.
these aren't tutorials though, they're just an overview. You are not really understanding anything from the video unless you already know the technology beforehand, just getting why it exists
@@javierflores09 I think he has made tutorials before, 100s is just one of the series on the channel
Sounds like something I'd use on a beach vacation project.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
In Hyperledger Fabric, which I use for our Thesis, it uses Kafka for consensus. Like, leader election and storing of data.
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
I needed this during my enterprise software semester 😢
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
Not sure about the developer but it's a good channel for a Product manager who can look at multiple techs from the top without going deep i to weeds
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
In an Iron Man movie, Kafka could have been useful in a scene where Tony Stark (Iron Man) needs to process and manage a large volume of real-time data or communications. Here's a hypothetical scenario where Kafka could play a role:
Scene: Tony Stark is in his high-tech lab, and he's remotely controlling his Iron Man suit, which is deployed in a distant location to handle a crisis. He needs to receive and process real-time data from various sensors on the suit, such as vital signs, telemetry data, and external environmental data, while also receiving live video feeds.
How Kafka could be useful:
1. **Real-time Data Ingestion**: Kafka could be used to ingest data from these sensors and video feeds in real-time. Each type of data (vital signs, telemetry, video) could be treated as a separate Kafka topic.
2. **Data Processing**: Tony needs to process this data for real-time decision-making. Kafka Streams, a component of Kafka, could be used to perform real-time data processing, such as analyzing vital signs for signs of distress, stabilizing the suit's functions, and identifying threats in the video feed.
3. **Reliability**: In a high-stakes situation like this, Kafka's reliability ensures that no data is lost. If there are network interruptions or delays, Kafka can buffer and replay messages, ensuring that Tony has access to all the critical data.
4. **Scalability**: If the crisis intensifies and more data needs to be processed, Kafka can scale horizontally by adding more Kafka brokers, allowing Tony to handle the increased data flow without performance issues.
5. **Monitoring**: Kafka provides extensive monitoring capabilities, which could be depicted in the movie as Tony monitoring the health of the data pipeline in real-time, ensuring that he has a clear view of the suit's status.
In this scenario, Kafka would enable Tony Stark to efficiently manage and respond to real-time data, enhancing his ability to control the Iron Man suit and handle the crisis effectively. It would add a layer of realism to the technological aspects of the movie.
This by far the best robot-voice I have ever heard, it took me hours of watching before it dawned on me that this is the result of text->voice tech.
Not a robot voice but that would be cool.
Here's his making of video th-cam.com/video/N6-Q2dgodLs/w-d-xo.html as proof (maybe)
It isn't a robot voice lol, he probably just edits out the audio so his voice doesn't fluctuate that much, giving it that effect
@@fiorenzorutschmann3656 Ooh, yeah I took a look, seems like he cuts his audio a lot (time-shaver).
I just watched this one:
th-cam.com/video/1xipg02Wu8s/w-d-xo.html
And tbh it sounds like he has a bot doing it, because every single sentence ends in the same tone, combined with that micro-second silence between words made it seem like that to me lol.
I guess he is just very coherent in the way he speaks
@@tally3018 put some decent quality headphones on and blast the volume to the max, you can hear the liquid in his mouth moving
@@mihailmojsoski4202 LMAO!
Very useful thanks
NATS is another great tool that does sort of the same thing as Kafka. it's JetStream module is amazing!
currently doing some r&d on nats for my company and so far it looks way less complicated than kafka
@@RedditFam check out NSQ as well, it's slightly different but also written in Go and a great option.
Just was researching on RabbitMQ and Kafka. Your contents are amazing.
I love honkai star rail 😨
thanks for the video
Pretty sure more people are making new web libraries than actually building websites that people actually use at this point.
Apache are the absolute gigachads of the open source community. They essentially have all software you need to run a billion dollar company, open sourced.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
Bro you should make a video about delphi
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Thank you for this.
Please do hot wire (/howired) as an alternative to regular web architecture
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
but what's the point of events? what need isbeing metbybeingable to reliably stream sequential data to multiple parties? can't this be done much more simply? or is scalability really that important?
Scale is important for big huge companies. Most system and people do not need Kafka!
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Apache Kafka must be the coolest tech name ever
Was expecting a philosophical analysis