Thanks. it was really helpful but for me as a non-native English speaker, the speed of talking was so fast, and I had to use subtitles for better understanding
If each topic is a file, how is about a partition? Does it means the aforesaid file will be divided into a subset of file again where each sub-file is a partition?
Hi Shyam, I'm Gilles Philippart from Confluent. Sophie says things like "This log is a file and it's a special kind of file" then later on "the topic itself really is just a name that is given to the log. So on the broker, they store each log and directory names by that topic". I agree that this part is a bit confusing as we normally associate a " Log" to a "Partition" not a topic. Sophie glossed over the details of how the data is stored physically on disk because that's not really the purpose of the course, but let me get this right: 1. A "topic" is a logical stream of records, and it is the primary unit of data organization in Kafka. 2. A "topic partition" is a logical unit that represents a subset of the data within a Kafka topic. Each partition is an ordered and immutable sequence of records. 3. Now, Kafka stores the data for each topic partition in a set of "log segments". Each log segment is a file on disk and represents a sequential chunk of data. The size of each log segment is configurable and typically ranges from a few kilobytes to several gigabytes. As new records are appended to a partition, they are written to the active log segment. For example, the directory structure of the first partition of the "account-deposits" topic would look like this: /var/lib/kafka/data/account-deposits-1 00000000000047926734.log 00000000000047926734.index ... 00000000000052497535.log 00000000000052497535.index If you want to learn about Kafka internals, check out developer.confluent.io/learn-kafka/architecture/broker/ Hope this helps!
Hi, I'm Gilles Philippart from Confluent. I don't think there's anything that would prevent you from doing so, Kafka Streams is just another library that you pull into your application dependencies after all. I'm not sure there would be a good reason to do that though, as it might prove harder to tune the performance, scale, secure, troubleshoot, etc. Have you considered creating another microservice that would consume or produce to a topic to interact with the Spark Streaming app?
Kafka Streams is a streaming processing library to join streams (topics) and simplify the building of stateful applications (windowing, aggregations). If might consume more network traffic due to the fact that most of the local state is also stored in Kafka for backup, but that's usually a good thing as it make your application more reliable!
Very good explanation Sophie. This video should be in the main Kafka website itself.
Thanks, this is very helpful info in a short amount of time!
Thanks. it was really helpful but for me as a non-native English speaker, the speed of talking was so fast, and I had to use subtitles for better understanding
The subscribe button in TH-cam lit up when you said “subscribe to the topic” 😂
4:50 Is when she starts explaining about Kafka Stream
@@sonnyps413 thanks
okay my long pending doubt is cleared now 🙏🏽
Fantastic tutorial. Thanks so much Sophie!
Good talk, thanks
Excellent. Wondering can I use Kafka Streams with the KafkaJS library for Node JS?
you may.. i am using with python
Thank You!
Thank you for giving helpful information
Thanks, very helpful
Kafka.
It’s no difference than philosophy names and definitions and describing how one thing or human can go from point A to point B.
That was a great introduction
Good info 😊
Does kafkajs in nodejs provide support for kafka streaming API ?
If each topic is a file, how is about a partition? Does it means the aforesaid file will be divided into a subset of file again where each sub-file is a partition?
Hi Shyam, I'm Gilles Philippart from Confluent.
Sophie says things like "This log is a file and it's a special kind of file" then later on "the topic itself really is just a name that is given to the log. So on the broker, they store each log and directory names by that topic".
I agree that this part is a bit confusing as we normally associate a " Log" to a "Partition" not a topic. Sophie glossed over the details of how the data is stored physically on disk because that's not really the purpose of the course, but let me get this right:
1. A "topic" is a logical stream of records, and it is the primary unit of data organization in Kafka.
2. A "topic partition" is a logical unit that represents a subset of the data within a Kafka topic. Each partition is an ordered and immutable sequence of records.
3. Now, Kafka stores the data for each topic partition in a set of "log segments". Each log segment is a file on disk and represents a sequential chunk of data. The size of each log segment is configurable and typically ranges from a few kilobytes to several gigabytes. As new records are appended to a partition, they are written to the active log segment.
For example, the directory structure of the first partition of the "account-deposits" topic would look like this:
/var/lib/kafka/data/account-deposits-1
00000000000047926734.log
00000000000047926734.index
...
00000000000052497535.log
00000000000052497535.index
If you want to learn about Kafka internals, check out developer.confluent.io/learn-kafka/architecture/broker/
Hope this helps!
Can we use Kafka Streams and Spark Streaming in the same project ?
Hi, I'm Gilles Philippart from Confluent. I don't think there's anything that would prevent you from doing so, Kafka Streams is just another library that you pull into your application dependencies after all. I'm not sure there would be a good reason to do that though, as it might prove harder to tune the performance, scale, secure, troubleshoot, etc. Have you considered creating another microservice that would consume or produce to a topic to interact with the Spark Streaming app?
So kafka streams is just a convinience client api? will the network traffic be equivalent?
Kafka Streams is a streaming processing library to join streams (topics) and simplify the building of stateful applications (windowing, aggregations). If might consume more network traffic due to the fact that most of the local state is also stored in Kafka for backup, but that's usually a good thing as it make your application more reliable!
Excellent
Kafka Streams, aka reducing boilerplate by transitioning to functional programming
TY
🔥🔥🔥
Very confident
Very strange choice of name, definitely not going to lead to concerns of big tech overreach :)
Her tongue is F1 car not able to understand good
Give some real world example not the virtual one please 🙄
estoy enamorado
Don't be that guy
Oye concentrate que esto es técnico 😁😁😁