Great fundamentals series. I am a network guy moving up to the application layers working with banking eservices and Kafka will be one of the carriage horses in my new team, so I needed this overview. I am glad I found this clear and to the point explanation.
Beautiful! If nothing else, I'll build a complex piece of architecture, only so "The slide looks easy". :) Jokes aside, this is really helpful. Thanks a ton for putting this together.
Hi there! 11:13 Why does topic A and Topic B are inside the producer? I guess, they are on the broker side. The video is awesome! I can easily understand the material!
8:42, when a broker dies and a new partition elected as the leader, how can we guarantee that no data is lost? Those messages sent to the dead broker but not replicated to its followers yet, are lost, aren't they?
So if we alwas write to and read from LEADER partition why you sai in a previous video that partitioning is a way how scaling worka in Kafka. Its not about scaling, but about durability. In orther words partitioning does not give us increased performance of message number which might be processed per second. Right?
Since the segments are continually removed from the head of the partition queue, how does the Kafka maintain the correct index offset after a segment expires ? Here is what I mean: as the segment with 0 offset expires, do the other segments get to keep their indexes, or the segment 1 will become the new head segment with offset 0 ?
Consumer Application is POLLING. What is this Polling actually? Connection to Kafka Server, its Client Server Connection? Polling connection thread ever gets expired? Its kind of PULL events/messages? Please suggest. Thank You.
Talking about GDPR... if a customer asks me to wipe it's data out, how would I deleted events originated by this user once Kafka's logs are immutable? 🤔
Just spitballing, encrypt all user messages with a cryptographic hash function. Retain the key for that user's hash for the lifetime of their account. Use that key to decrypt that user's messages when consuming them. Then throw the key away when and if they ask to be forgotten or delete their account. Messages can then stay in the log but are no longer readable by anyone because the key is gone. Don't listen to me, literally never solved this problem before, but I imagine others have worked around log immutability without the cost of, well, mutating a log. I bet there are open source libraries or blog posts out there that have some better solutions baked in already.
Thanks for the presentation, i'm starting to learn Kafka, but i have a question about compacted log: How, based on timestamp, the brokers know exactly what is the last value for the key when you talked about compacted log ? If there are multiple brokers on multiple datacenters how the clock synchronization between these brokers is done to avoid a more recent event to be override by a old one ?
This was honestly, the best explanation of any technology I've seen on the Internet! Thanks Confluent.
You are an AMAZING teacher and a presenter.
Don't do this often, but I got to do it here - my complements to the author's great narration/delivery skills.
One of the best lectures for Kafka. I always come here to refresh stuff
This the best kafka architecture explanation that i ever seen
A super amazing teacher, I highly resect your teaching methodology...
This video answered so many questions I had about Kafka. Awesome!
wow, this was a great presentation.
Great fundamentals series. I am a network guy moving up to the application layers working with banking eservices and Kafka will be one of the carriage horses in my new team, so I needed this overview. I am glad I found this clear and to the point explanation.
U 0:07 😊😂❤😂😂😂😂❤😂😂😂😂😮😂😢😂😂😂
U 0:07 😊😂❤😂😂😂😂❤😂😂😂😂😮😂😢😂😂😂
Great presentation and clear information.
Thank you
great explanation!, so detailed, a very good course about Kafa fundamentals!
Fantastic talk, thanks for sharing this with us 👍🏻
Beautiful! If nothing else, I'll build a complex piece of architecture, only so "The slide looks easy". :)
Jokes aside, this is really helpful. Thanks a ton for putting this together.
This is unbelievably good !! 😍
Hi there!
11:13 Why does topic A and Topic B are inside the producer? I guess, they are on the broker side.
The video is awesome! I can easily understand the material!
Amazing. Thanks so much ❤.
Thanks for a great explanation. This video has definitely answered some of the questions that I had.
Could you please list the documentation sites from confluent web pages in the description as well?
8:42, when a broker dies and a new partition elected as the leader, how can we guarantee that no data is lost? Those messages sent to the dead broker but not replicated to its followers yet, are lost, aren't they?
Depends on your ack settings, if NONE or LEADER yes, possible data loss, but if ALL, followers already have a copy.
Thanks for the very interesting presentation! Just enough to start with Kafka! Are the slides shared publicly? Any link please?
So if we alwas write to and read from LEADER partition why you sai in a previous video that partitioning is a way how scaling worka in Kafka. Its not about scaling, but about durability. In orther words partitioning does not give us increased performance of message number which might be processed per second. Right?
Congratulations folks: if you’re watching this video than you’re ready to transit to the Senior Engineer level
But actually new grads who can’t find a job ;(
wow Simply Brilliant !
Thanks, Tim, great vid!
Since the segments are continually removed from the head of the partition queue, how does the Kafka maintain the correct index offset after a segment expires ?
Here is what I mean: as the segment with 0 offset expires, do the other segments get to keep their indexes, or the segment 1 will become the new head segment with offset 0 ?
Using zookeeper
Does it make sense to consume a compacted log? It’s not event driven anymore, more like snapshots
Great tutorials, thank you sir
Can we get the links provided at 26:25? cannot click, or copy those links
How and where can I get that shirt???
That was awesome! Thank you so much!!
Consumer Application is POLLING. What is this Polling actually? Connection to Kafka Server, its Client Server Connection? Polling connection thread ever gets expired? Its kind of PULL events/messages?
Please suggest. Thank You.
great talk
Amazing.
11:00 bro I if everything was fine, I wouldn't be here. Great explanation tho!
I see a lot of dad-jokes on this one.
Learnt a few for my child.
Talking about GDPR... if a customer asks me to wipe it's data out, how would I deleted events originated by this user once Kafka's logs are immutable? 🤔
Just spitballing, encrypt all user messages with a cryptographic hash function. Retain the key for that user's hash for the lifetime of their account. Use that key to decrypt that user's messages when consuming them. Then throw the key away when and if they ask to be forgotten or delete their account. Messages can then stay in the log but are no longer readable by anyone because the key is gone.
Don't listen to me, literally never solved this problem before, but I imagine others have worked around log immutability without the cost of, well, mutating a log. I bet there are open source libraries or blog posts out there that have some better solutions baked in already.
could we have a link for sharing slide???
Hello. Can we have the slides please?
Thanks for the presentation, i'm starting to learn Kafka, but i have a question about compacted log:
How, based on timestamp, the brokers know exactly what is the last value for the key when you talked about compacted log ?
If there are multiple brokers on multiple datacenters how the clock synchronization between these brokers is done to avoid a more recent event to be override by a old one ?
Yeah, but I want exactly-once delivery from a Kafka consumer to an external API :v
Set the speed to 0.75
“Is the disc on fire?!” :)
Super
Compacted Topics are more subjective
This course would be so much better if the speaker would not change tone, volume, and speed of his voice so frequently.
how come it has only 258 likes?