Topics, partitions, and offsets in Kafka

[@KnowledgeAmplifier1]

Codes used in the demo :
--------------------------------------------
Start Zookeeper:
---------------------------
F:/kafka_2.12-3.2.0/bin/windows/zookeeper-server-start.bat F:/kafka_2.12-3.2.0/config/zookeeper.properties
Start Kafka-server:
-----------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server.properties
Create a topic:
-------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-topics.bat --create --topic demo_testing --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3
Start Producer:
--------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-producer.bat --topic demo_testing --bootstrap-server localhost:9092
Start Consumer:
----------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --from-beginning --bootstrap-server localhost:9092
Reading message from specific partition:
---------------------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --from-beginning --bootstrap-server localhost:9092 --partition 2
Reading message from specific offset inside a specific partition:
---------------------------------------------------------------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --bootstrap-server localhost:9092 --partition 2 --offset 1
Reading message from specific offset :
-------------------------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --bootstrap-server localhost:9092 --offset 0

[@avinashbasetty-r4d]

Explained well. concepts are so clear.
Below are the topics learned so far -
1. Verifying multiple partitions in the server logs directory
2. Checking the offset based on the flag parameter within the partition
3. Content check for each message within the partition

[@shobhitarya1637]

Wow, you explained the topics very well with practical as well as with internals. Its a very good tutorial for learning kafka. Thank you .

[@KnowledgeAmplifier1]

Thank you for your kind words Shobhit Arya! Happy Learning

[@shreyas-s5g]

absolute genius. amazing xplaination sir, thankyou so much.

[@KnowledgeAmplifier1]

Thanks and welcome

[@bigdataprofessional1880]

Crisp explanation, 100% perfect

[@KnowledgeAmplifier1]

Thank you BigData Professional! Happy Learning

[@ohuuthai1678]

Why does my data only save to partition 0 when I was like you, I tried adding many other input data examples, but it only jumps to partition 0

[@kamalnayan9157]

Great work man! Thank you

[@KnowledgeAmplifier1]

Thank you Kamal Nayan! Happy Learning

[@ayansrivastava731]

if someone is using kafka >2.4 , this approach of round robin(random partitioning) will NOT work, instead there is something called sticky partitioner where you'll see all msgs going to same partition. WHY? because we didn't specify keys in the start. The partitioning strategy by kafka has changed.

[@KnowledgeAmplifier1]

You are correct Ayan , here are the videos where I have explained the same -- th-cam.com/video/-eo4IPUshRo/w-d-xo.htmlsi=o_tQF0TYGE5qFO_D

[@BOSS-AI-20]

Thanks, I was also thinking about the same problem

[@digitaltechconnect1318]

Please provide us the session for Confluent Kafka

[@vigneshr4197]

One kafka producer,two consumer subscribes to kafka topic then how partition works?

[@KnowledgeAmplifier1]

Hello Vignesh R , you can have a look in this video , I explained your scenario in detail -- th-cam.com/video/yKzdum_vX7k/w-d-xo.html
Hope this will be helpful! Happy Learning

[@KeshavDeoSharma]

Thank you Sir

[@KnowledgeAmplifier1]

You are @user-vp4by6sb1m ! Happy Learning

[@alapysalapylalapy2352]

Can we connect?

[@KnowledgeAmplifier1]

Hello Namrata Pathak, please post your doubt here , If I know that topic or have any idea , I will surely reply asap , also for any kafka related doubts , please check this playlist once , might help you if the topic is covered in the playlist already -- th-cam.com/play/PLjfRmoYoxpNrs0VmIq6mOTqXP52RfZdRf.html

[@alapysalapylalapy2352]

@@KnowledgeAmplifier1 hi i am facing one error with my assignment where to post the picture??

[@alapysalapylalapy2352]

@@KnowledgeAmplifier1 what are the rules of data ingestion i.e Kafka to hdfs in which video it is present??