@12:28 she talks about a use case in which @ 12:35 she says need of having consumer groups of size 100s of consumers. The justification goes : "If there is a fail over kafka-cluster, and it's really big and you have to copy everything from one kafka-cluster to another in real time you need a large consumer group to do it because you're consuming every single partition in your kafka-cluster in a single group." - Because you don't want to waste time. And the Kafka-System must be strong enough to support it quickly, before the fail over the current kafka-cluster eats it all up and you've all your data gone.
@12:28 she talks about a use case in which @ 12:35 she says need of having consumer groups of size 100s of consumers.
The justification goes : "If there is a fail over kafka-cluster, and it's really big and you have to copy everything from one kafka-cluster to another in real time you need a large consumer group to do it because you're consuming every single partition in your kafka-cluster in a single group."
- Because you don't want to waste time. And the Kafka-System must be strong enough to support it quickly, before the fail over the current kafka-cluster eats it all up and you've all your data gone.
Great talk Gwen!
Very insight on kafka internals