Finding your channel feels like finding gold! There are ton of SD videos on youtube with shallow content basically exactly like you mention what a junior or mid level candidate would do. Going indepth for senior and staff is one of the highlight of your content. Please continue doing that. Also please don't worry about length of the video. Keep the gold coming :)
Watch this and then imagine if Evan puts together a System Design Learning Course. Just image that !!!! I mean we (the learners) will jump on it sooo quickly. This is just absolutely amazing. This is combining years of experience with hands on actual approach that works along with book contents presented in a very professional manner. Evan, think about it 🙂
Awesome walkthrough! As a junior engineer I learned a lot. Near the end you said you wanted to keep it short but I appreciate the nuances you point out for your decisions including design choices between mid and higher level candidates. Time didn't faze me at all, I watched every second!
You are the best at what you do, after listening to your videos I cannot now like and tolerate other system design videos, please make more content for system design
I rarely leave comments but this is BY FAR the best system design video I have seen on youtube. The thing I like most is that it differentiates from common textbook solutions that we see everywhere and you explain why certain choices were made. Thanks!
Really helpful videos - especially the breakdown for different expectations at mid/senior/staff levels, common things you see from candidates, and context into the actual systems like the shard limits for events streams. I used to work on Kinesis - happy you chose it!
These interview preps make you feel like if you know enough of system design knowledge, have good cross team examples for bq, and can solve leetcode medium-hard fast, you can get to higher level quicker than going through internal promotions.
I discovered your channel a few days and love all the videos so far. I love the solution enforce idempotent click tracking. There are a lot of SD videos but only your SD videos provide guidelines on the expectations based on candidates' levels. So far, I watch one video per day so I will run out of videos to watch very soon xD
This video helped me ace the system design interview. The detailed explanations provided in-depth knowledge of various components, which was extremely helpful for answering follow-up questions during my interview.
I have purchased so far Alex Xu grokking system design by Mikhail Smarshchok i would say i learned a LOT from the Mikhail Smarshchok as far as internals go and then i bumped into your Hello Interview ABSOLUTE BEST stuff that can replace Alex,grokking for me. I was ALWAYS looking for how to connect the functional requirements into the later high level design/ deep dives but both Alex, Grok fail many times in connecting the dots. Subscribed and a big fan of you. I will consider purchasing coaching or mocks once I am through your videos and feel confident.
I’ve purchased both of those contents (I’m a big fan of Mikhail’s TH-cam channel, but once he became popular he stopped posting and decided to monetize). Alex Yu is very dry and hand wavy for me. I’ve asked questions on the chat (Discourse?) but those are answered by other students, and I don’t have any confidence that they’re speaking from experience.
Ah, nice, you re-uploaded! Thanks a lot for taking the feedback and acting quickly on this. And, sorry if it caused inconvenience for you 😄Thanks a lot for all of your hard work. 🙏
Great job with creating this content to help us prep for interviews! Just one thing to note, you can't send a 302 redirect for POST requests, it has to be a GET request.
Absolutely understood everything in this, being a starting data engineer myself and looking for my next venture. this is class content.Thank you so much.🎉❤
Thank you so much for these videos! They’re absolute goldmines-I’m genuinely amazed at the depth and clarity they offer. I’ve learned more from these than from many of the books I’ve read. Truly invaluable!
Excellent, as usual! Thanks so much 🙏 While concluding, you mentioned you want to keep the videos short. Please don't reduce the length. 1 hour is a sweet spot and it's necessary to capture all the important "juicy" tidbits and details you highlight. Please keep it coming 🙌
I just found your channel and these videos are really great and super helpful! I love how you verbalize the challenges and tradeoffs as they come up, it provides a great example for how to communicate in an interview.
This was such a pleasure to watch. Thank You. I would love to see a video on a metrics monitoring system. There will be some common components with ad-click aggregators.
Thank you for a great video. For a senior candidate it will be helpful, in my opinion, to narrate the data structures that underpin these solutions in addition to the supporting vendor products/technologies. In that, for fast aggregation of counters, one could demonstrate the use of fast but slightly imprecise counters using a count-min-sketch-like data structure, and for a slower but more accurate counter the use of a Map Reduce jobs. Aggregates and statistics over varying windows are almost a necessity for contemporary Monitoring, Analytics and ML Systems. And in such cases retaining data in-memory backed by persistent storage in the form of tree data structures keyed by aggregation windows are useful for range queries at varying granularities. For e.g.: root window [0-100). Immediate child node windows [0-50), [50-100) etc. It could be helpful to talk about idempotency within queue consumers. And also out-of-sequence arrival of events in the queue (handled through watermarking)
The content is great as always. However I found this system a bit frustrating, there is so much specific knowledge one should know to really end up with a good design for this kind of problem. I wish I don't get a system like this in an interview.
If we replace kinesis stream with Kafka we can perhaps get rid of Flink and S3. Kafka has the concept of aggregating stream by group function and time windows (tumbling, hopping etc). Further it has a ksqldb that stores all events for a 7 day period which can be queried and compared with OLAP to verify data integrity. Also, I love your videos. Looking forward to more of those.
you are honestly the best content on system design , can you do some playlist on the system design topics themselves ? i mean a video where you discuss replications in depth , concurrency , etc.
these are so good , such deep dive and so clear ! Thanks because they cover so many different aspects . i am looking forward to many more videos . I am currently preparing for interviews and these are so helpful ..
I saw some videos and your content is so great. Thank you so much for clarifying the SQL vs NoSQL debate. I always thought that bringing that into an interview was irrelevant but was afraid to do it. 😅 Keep up the amazing work.
For hot shard problem in Kafka, you can salt things but in Flink, we will no longer have all the events aggregated in one Flink task. In the case where we are sinking to Redshift, we could have aggregate it there. but if we want to access it in stream, maybe we need a secondary stream that aggregates everything? So that added salt needs to be handled one way or the other
Stellar video. Only suggestion for the next ones: your drawing tool seem to have keyboard shortcuts for those diagrams and other options on the toolbar. It'll greatly improve your quality of life! Keep it up!!
I believe the checkpoint is still necessary so it avoids data loss. If the server crashed, the checkpoint stores what the offset in kafka/kinesis stream is at and restart from the offset in the checkpoint. Without the checkpoint, flink has no idea where it should be restart from and the data will be lost between the last time it send data to the OLAP and the the it crashed.
Hey, I love these videos. I only used your videos and designing data intensive applications and that was enough for an E4 offer at Meta, I love the advice you give and common pitfalls you provide.
Awesome content!! I was hoping to get more clarity on how same flink node determines the partitions it need to consume from in case of hot shard issue which we are effectively handling by adding a number b/w 0-n to the partition key which leads to same adId clicks published to multiple partitions in the stream.
Great video! I learnt a lot! One question I have is regarding the hot shard handling. When the click processor detects there is a hot shard and decides to change the key from AdId to AdId:n, how would it let Flink know that it now needs to change the aggregation logic (and sharding logic) for that particular AdId? (I believe this would also have a race condition when it changes within a minute window, but any data integrity issue that arise from it should get resolved by the batch job)
Thanks for the content; It's great; One query on 1. As we compute 10sec and write it to OLAP DBs ; The deep dive on how the user queries for 1min window, 1hr window, may be 1 day and all time window would be great info; 2. And what could be that OLAP database choice would be and what factors to consider. 3. Another aspect is; Freshness of the data; What does each component in the system contributes to the latency to make the event available for aggregates; Other aspects are great; thanks
Since you are filtering for a specific AdId (AdId = 13), grouping by AdId is redundant. The result will only include data for AdId = 13, so grouping won't change the output
Thanks a lot ! This video is super helpful. It not only helped me understand the key components of ad click aggregator, but more importantly it taught me what does the interviewer expect for different level SDEs. One question --- do we need to dive into the aggregation logic details in Flink?
Hi, First of all thank you for the great video, question : at @58:00 how does ad-click service know if the signed impression id is legit,because ad impression id is generated and signed by the ad-placement service?
Thanks for the video, it is really in-depth and informative. You covered the flink + streaming part really well and why not to use checkpointing was really a good point. Saw some videos in the past that mentioned let's do checkpointing and never explained why it will work or not work. For the streaming components both streams (kinesis or kafka) and stream processor (Flink/kafka-streams) I got following questions ( (I was asked similar questions in an interview.) 1. we have said that we will partition the shards or streams using adId but we have 10M ads at any given point of time. In this case, if I consider other things like replicas in kafka topic or even kinesis shards, don't we have to create lots of shards? 2. The max traffic is 10k cps so most of the ads are not clicked and won't have active traffic so most of the shards will be empty, in this case what should be the approach? 3. If we group some ads based on let's say advertiserId than we introduce the noisy neighbour problem where one ad generates lots of traffic and blocks the other ads.
Those questions are great and definitely come up often with streaming setups. I'm no expert but ... Partitioning with 10M Ads ... Yeah, if we just went with adId for partitioning, it would mean loads of shards or partitions, which isn’t practical. Instead, you can use consistent hashing or modulo partitioning to map multiple adIds to a smaller, manageable number of partitions.
I’ve become an annual premium member of HelloInterview after watching couple of your videos. It’d be really helpful if you started a chat/messaging platform for the paid subscribers, so that we could get our questions answered without relying on TH-cam.
Question: Why do we need the reconciliation mechanism? And why use a batch processing for the purpose? Do we know/assume stream processing would accumulate errors compared to batch processing (the latter being accurate?)?
I like your videos, I have learned a lot. A couple comments on this video: a. I think the system would benefit from a Redis in the click processor service, not the idempotency lock but a redirectUrl cache {adId: redirectUrl} to reduce reads in the Ad DB. It might be a MRU cache to avoid overloading the Redis. b. I'm not sure why you are pushing Kinesis so hard in this solution, I mean yes I learned something about Kinesis, but it would be more practical just to place a Kafka that can handle the load peaks and has event history as well so it is possible to write the reconciliation procedures from it. c. I learned about Flink, thanks. I used a redis aggregator in my own solution. Thank you so much for your work!
For Kinesis hot shards, we don't know if an ad is hot beforehand. So are these ad_id 0-N always active? Is it ok to use x10 the amount of streams we need under normal circumstances? For Flask, we have the same amount of flask servers as the Kinesis shards right? If the server dies, how will the new server keep track of the pointer from the old server? Are they statefull backups instead of stateless
This is a great question. In reality you can make predictions here. We know based on budget and historical performance which ads we’d need to be worried about before hand
I rewatched and had some new thoughts. Wonder what are the costs of using streaming solution? I seems like the database for clicks that was used in batching solution is completely replaced by the streaming components, so benefits from having the previous database queries are lost? 34:52 streaming solution real time is by dumping to OLAP?
Hey Evan, your videos have been the best, seriously! The only suggestion I have is if you could use a dedicated mic that would be wonderful. The volume is too low even on max volume on my earphones.
Thank you so much for the explanation. However I more like you write the functional and non functional requirements instead of paste from the clipboard. The main reason is that it's make me thinking about what should be the requirements. It's feels more like code pairing
We initially said that we want to perform complex queries. In addition to time range, we want to check who were the users, and maybe where the clicks were from, etc. So that means we need more types of aggregations, for example by user id, by location, etc. How do we handle that? What if we get a query that we didn't prepare an aggregation for in advance?
This is great content. However, I would like to point out that the Lambda architecture includes both batch and speed layers, which process historical and real-time data in parallel. It’s not solely reliant on batch processing.
Question: For the sharding while processing the events through Kinesis, the adId was suggested as the sharding key. This doesn't look like the best approach. At scale, millions of ads are being run on the platform and a good share of them have high enough volume. Going by the presented logic, the number of shards would explode. What do you think about this?
Why did we not use a Redis instead of using Flink? Redis is also and in memory structure, that will meet our write throughput demand and we should scale it by sharding on Ad id?
Few questions: - Even in spark solution, how would you know the keys of which one to aggregate? Either you have to emit keys of which has changed or scan the db by time and get keys which is again costly. - Aren't this falls under lambda architecture as we are using both realtime stream processing and batch processing to ensure data integrity?
The out-of-scope non-functional requirements seem to be more like out-of-scope functional requirements. I feel that (spam detection, demographic profiling, conversion tracking) are essentially features rather than characteristics of the system. How should I be thinking about this?
also at 41.28, after appending 0..n to the hot ad id and then distributing to different shards in kinesis, we 're letting flink handle finding the same ad_id from all different shards, so for a single flink job, which is responsible for a given adId, it will be responsible for checking every shard and see if the given ad ID exist in that shard, this will happen for each ad id, won't it be too much to check and aggregate for a flink ? how can it scale better being in memory as compared to other db services ? the more load will still ask for more instances, isn't it ?
As i watch this again, why do you say checkpointing is not required at 45:30 ish time? How does a job keep track of which time interval to process the click rate? Does flink internally maintain a state of the processed timestamp? (im not familiar with flink! )
I think he mentioned the retention period in kinesis should take care of the failed flink jobs and I looked up that flink has timestamps and also called watermarks to deal with out of order jobs
Amazing content! Very much appreciate you posting these 🙌 System design padawan here. I have a question about the hybrid approach .. what makes us trust the "Kinesis -> Connector -> S3 -> Spark -> Worker -> OLAP" results more than the "Kinesis -> Flink -> OLAP" results? Is it a guarantee where the connector always successfully writes the data to S3? or does Flink make some kind of tradeoff for speed? kind of confused about that piece and figured i'd ask. thanks again!
+1 on this question. IMO, Spark is useful when you have really out-of-order events, like events arriving half an hour late or something. Then, by using Spark, you can reorder the events and get a more accurate count. On the other hand, for events that are only a few minutes late, you can configure Flink with the allowed lateness of a few minutes. That being said, the cost of maintaining 2 code bases and ensuring that they are updated at the same time (to avoid triggering discrepancy alerts) doesn't seem worth it for such edge cases. I'd be interested to hear @hello_interview's thoughts on this though.
> Is it a guarantee where the connector always successfully writes the data to S3? Yes. That is something provided by AWS as a managed service, and it should meet their SLAs. AWS would've created fallbacks and fault tolerance to ensure that all events that are in Kinesis reach S3. I don't see there being any compromise due to speed. Kinesis has retention policy, so the data isn't going anywhere, and S3 is highly available, so the data shall be written there as well.
Thanks alot, great explanation \o/ Regarding handling the idempotency, why can we not get away with only the ad impressionID? Is signed one actually required?
Hi Great explanation for a complex topic in such a easy way.Is Once only processing also critical when we are passing messages from kafka to flink.If we want one to enable once only processing on flink then checkpointing will be required juts a though
You can set it per message! From the docs, “Every Amazon SQS queue has the default visibility timeout setting of 30 seconds. You can change this setting for the entire queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue. When receiving messages, you can also set a special visibility timeout for the returned messages without changing the overall queue timeout.”
Do you think knowing OLAP is important for a senior/staff role? Having no experience with analytics, I'd just go for an RDS - guess it'd probably be fine?
Can you please expand on below questions? or Link a small video/article if possible 1. How will "Click Processor SVC" know which AdID is popular/hard? 40:24 2. How will "Flink" handle further aggregation of AdID:0, AdID:1,..., AdID:N to AdID 40:43
1. Could be based on past performance or budget. Realistically, companies have ML models to predict this. 2. A single job will read from different partitions and aggregate.
This kind of content can make someone fall in love with software engineering.
Finding your channel feels like finding gold!
There are ton of SD videos on youtube with shallow content basically exactly like you mention what a junior or mid level candidate would do.
Going indepth for senior and staff is one of the highlight of your content. Please continue doing that.
Also please don't worry about length of the video. Keep the gold coming :)
@hello_interview waiting eagerly for next video
This is by far the best system design interview ever seen on the internet. Keep doing the great work sir...
You somehow managed to make preparing for system design interviews really fun. Massively underrated channel
You’re the best!
This is not just interview prep, this is some serious stuff here. Thanks a lot!
🫡
Watch this and then imagine if Evan puts together a System Design Learning Course. Just image that !!!! I mean we (the learners) will jump on it sooo quickly. This is just absolutely amazing. This is combining years of experience with hands on actual approach that works along with book contents presented in a very professional manner. Evan, think about it 🙂
Maybe one day! For now just happy with all the people learning about hello interview and getting tons of free value
Honestly, it is the best SD showcase I’ve ever seen. You are the best. I watched all your videos and whiteboard them myself then. Thank you!
So glad you like them and very smart to try them yourself and not just blindly consume!
Awesome walkthrough! As a junior engineer I learned a lot. Near the end you said you wanted to keep it short but I appreciate the nuances you point out for your decisions including design choices between mid and higher level candidates. Time didn't faze me at all, I watched every second!
Hell yah!
You are the best at what you do, after listening to your videos I cannot now like and tolerate other system design videos, please make more content for system design
Coming soon! 🫡
Totally agreed! I also cannot tolerate anyone else at this point
I rarely leave comments but this is BY FAR the best system design video I have seen on youtube. The thing I like most is that it differentiates from common textbook solutions that we see everywhere and you explain why certain choices were made. Thanks!
Thanks for commenting!! Glad you enjoyed it
Literally recommended Hello Interview to everyone I've mocked interviewed with
You rock
Really helpful videos - especially the breakdown for different expectations at mid/senior/staff levels, common things you see from candidates, and context into the actual systems like the shard limits for events streams. I used to work on Kinesis - happy you chose it!
How cool! That must’ve been fun to work on :)
This is the best ad click aggregation system design video and article I have ever seen!
One of the best channels on system design! Please keep going!
These interview preps make you feel like if you know enough of system design knowledge, have good cross team examples for bq, and can solve leetcode medium-hard fast, you can get to higher level quicker than going through internal promotions.
I discovered your channel a few days and love all the videos so far. I love the solution enforce idempotent click tracking. There are a lot of SD videos but only your SD videos provide guidelines on the expectations based on candidates' levels. So far, I watch one video per day so I will run out of videos to watch very soon xD
This video helped me ace the system design interview. The detailed explanations provided in-depth knowledge of various components, which was extremely helpful for answering follow-up questions during my interview.
I have purchased so far
Alex Xu
grokking
system design by Mikhail Smarshchok
i would say i learned a LOT from the Mikhail Smarshchok as far as internals go
and then i bumped into your Hello Interview
ABSOLUTE BEST stuff that can replace Alex,grokking for me. I was ALWAYS looking for how to connect the functional requirements into the later high level design/ deep dives but both Alex, Grok fail many times in connecting the dots.
Subscribed and a big fan of you. I will consider purchasing coaching or mocks once I am through your videos and feel confident.
I’ve purchased both of those contents (I’m a big fan of Mikhail’s TH-cam channel, but once he became popular he stopped posting and decided to monetize). Alex Yu is very dry and hand wavy for me. I’ve asked questions on the chat (Discourse?) but those are answered by other students, and I don’t have any confidence that they’re speaking from experience.
Ah, nice, you re-uploaded! Thanks a lot for taking the feedback and acting quickly on this. And, sorry if it caused inconvenience for you 😄Thanks a lot for all of your hard work. 🙏
Thanks so much for calling that out! Glad to get it fixed within the first day :)
Is e-commerce (design amazon / ebay) not as common as it once was?
Great job with creating this content to help us prep for interviews!
Just one thing to note, you can't send a 302 redirect for POST requests, it has to be a GET request.
Absolutely understood everything in this, being a starting data engineer myself and looking for my next venture. this is class content.Thank you so much.🎉❤
Thank you so much for these videos! They’re absolute goldmines-I’m genuinely amazed at the depth and clarity they offer. I’ve learned more from these than from many of the books I’ve read. Truly invaluable!
So glad you like them!
I felt this was much better than the Alex Xu System Design Vol 2 on the same topic. Great Job1
High praise!
Excellent, as usual! Thanks so much 🙏 While concluding, you mentioned you want to keep the videos short. Please don't reduce the length. 1 hour is a sweet spot and it's necessary to capture all the important "juicy" tidbits and details you highlight. Please keep it coming 🙌
If you HAVE to reduce something, please reduce the time between videos to 1 week 😛
No, seriously, thanks so much.
Haha trying 😝
I just found your channel and these videos are really great and super helpful! I love how you verbalize the challenges and tradeoffs as they come up, it provides a great example for how to communicate in an interview.
This is the best system design video I have seen on youtube till now. Really loved the in depth discussion. Would love to see more videos. 👍
This is great. And in case someone wants to deep dive specifically into the Lambda and Kappa, you can refer to the Alex Xu book 2.
Really good! Listened 3 times to pick up every single detail. Thanks.
This was such a pleasure to watch. Thank You. I would love to see a video on a metrics monitoring system. There will be some common components with ad-click aggregators.
Thank you for a great video.
For a senior candidate it will be helpful, in my opinion, to narrate the data structures that underpin these solutions in addition to the supporting vendor products/technologies. In that, for fast aggregation of counters, one could demonstrate the use of fast but slightly imprecise counters using a count-min-sketch-like data structure, and for a slower but more accurate counter the use of a Map Reduce jobs. Aggregates and statistics over varying windows are almost a necessity for contemporary Monitoring, Analytics and ML Systems. And in such cases retaining data in-memory backed by persistent storage in the form of tree data structures keyed by aggregation windows are useful for range queries at varying granularities. For e.g.: root window [0-100). Immediate child node windows [0-50), [50-100) etc.
It could be helpful to talk about idempotency within queue consumers. And also out-of-sequence arrival of events in the queue (handled through watermarking)
Can have some future videos which go deeper on probabilistic data structures or other more foundational topics.
The content is great as always. However I found this system a bit frustrating, there is so much specific knowledge one should know to really end up with a good design for this kind of problem. I wish I don't get a system like this in an interview.
If we replace kinesis stream with Kafka we can perhaps get rid of Flink and S3.
Kafka has the concept of aggregating stream by group function and time windows (tumbling, hopping etc). Further it has a ksqldb that stores all events for a 7 day period which can be queried and compared with OLAP to verify data integrity.
Also, I love your videos. Looking forward to more of those.
Amazing, really better than other stuff I found on internet!
Kudos to you!
you are honestly the best content on system design , can you do some playlist on the system design topics themselves ?
i mean a video where you discuss replications in depth , concurrency , etc.
Will definitely consider this!
Thanks a lot for uploading these videos. They are very informative. Keep doing the good work.
I like the signed impressionID for deduping. Great video. Thanks!
these are so good , such deep dive and so clear ! Thanks because they cover so many different aspects . i am looking forward to many more videos . I am currently preparing for interviews and these are so helpful ..
Love these! And can't recommend the Hello Interview mock interviews enough!
Wahoo thanks Ben!
Thanks so much for doing this! Greatly appreciated! By far the best system design videos I've seen.
need more like this man!
Amazing explanation skill you have, OMG.
I saw some videos and your content is so great. Thank you so much for clarifying the SQL vs NoSQL debate. I always thought that bringing that into an interview was irrelevant but was afraid to do it. 😅
Keep up the amazing work.
Yah funny how that was evangelized in a couple books and then just stuck
For hot shard problem in Kafka, you can salt things but in Flink, we will no longer have all the events aggregated in one Flink task. In the case where we are sinking to Redshift, we could have aggregate it there. but if we want to access it in stream, maybe we need a secondary stream that aggregates everything?
So that added salt needs to be handled one way or the other
A single Flink task can read from various shards.
absolutely brilliant content mate. keep em coming. only channel for which I have a notification on.
Very useful video and its best among others. I got this same question twice in my loop interview.very happy how I answered
Nice. Good job!
You are a legend man. Make some more videos which are mentioned on your websites. Search, E-commerce , Hotel Booking system etc.
Beautiful Design and Amazing explanation - just impressed with the elegance of the design and the beauty of software engineering.
😍
This is gold right here. Thank you!
Thanks for the detailed explanation! Definitely learned some new things in this video.
Stellar video. Only suggestion for the next ones: your drawing tool seem to have keyboard shortcuts for those diagrams and other options on the toolbar. It'll greatly improve your quality of life! Keep it up!!
I believe the checkpoint is still necessary so it avoids data loss. If the server crashed, the checkpoint stores what the offset in kafka/kinesis stream is at and restart from the offset in the checkpoint.
Without the checkpoint, flink has no idea where it should be restart from and the data will be lost between the last time it send data to the OLAP and the the it crashed.
However, Kinesis is aware of all subscription offsets, allowing Flink to read from the correct position in the stream.
I love this channel. Very good job sir, your strategy is really good a comprehensive. Straight to the main points. Bravo
Hey, I love these videos. I only used your videos and designing data intensive applications and that was enough for an E4 offer at Meta, I love the advice you give and common pitfalls you provide.
Crushed it. Congrats on your offer!
Looking for your next videos. Pls upload more design problems. It almost 1month you have not uploaded. Love your content.
Sorry, was traveling. Recording a video today! Up by EOW
Awesome content!! I was hoping to get more clarity on how same flink node determines the partitions it need to consume from in case of hot shard issue which we are effectively handling by adding a number b/w 0-n to the partition key which leads to same adId clicks published to multiple partitions in the stream.
Great video! I learnt a lot! One question I have is regarding the hot shard handling. When the click processor detects there is a hot shard and decides to change the key from AdId to AdId:n, how would it let Flink know that it now needs to change the aggregation logic (and sharding logic) for that particular AdId? (I believe this would also have a race condition when it changes within a minute window, but any data integrity issue that arise from it should get resolved by the batch job)
Thanks for the content; It's great; One query on
1. As we compute 10sec and write it to OLAP DBs ; The deep dive on how the user queries for 1min window, 1hr window, may be 1 day and all time window would be great info;
2. And what could be that OLAP database choice would be and what factors to consider.
3. Another aspect is; Freshness of the data; What does each component in the system contributes to the latency to make the event available for aggregates;
Other aspects are great; thanks
Since you are filtering for a specific AdId (AdId = 13), grouping by AdId is redundant. The result will only include data for AdId = 13, so grouping won't change the output
25:20 I thought for DB, time series databases can write fast and also handle ranged based queries quickly? Or some wide column databases
yah can be a good consideration. don't know enough about the ins and outs of popular TS DBs to offer a strong justification either way
Thanks! I see it getting name dropped in a lot of books , but outside the books I haven't see it a lot
Thanks a lot ! This video is super helpful. It not only helped me understand the key components of ad click aggregator, but more importantly it taught me what does the interviewer expect for different level SDEs. One question --- do we need to dive into the aggregation logic details in Flink?
How can you identify if interviewer is asking Product vs Infrastructure system design question?
Hi, First of all thank you for the great video, question : at @58:00 how does ad-click service know if the signed impression id is legit,because ad impression id is generated and signed by the ad-placement service?
Thanks for the video, it is really in-depth and informative. You covered the flink + streaming part really well and why not to use checkpointing was really a good point. Saw some videos in the past that mentioned let's do checkpointing and never explained why it will work or not work.
For the streaming components both streams (kinesis or kafka) and stream processor (Flink/kafka-streams) I got following questions ( (I was asked similar questions in an interview.)
1. we have said that we will partition the shards or streams using adId but we have 10M ads at any given point of time. In this case, if I consider other things like replicas in kafka topic or even kinesis shards, don't we have to create lots of shards?
2. The max traffic is 10k cps so most of the ads are not clicked and won't have active traffic so most of the shards will be empty, in this case what should be the approach?
3. If we group some ads based on let's say advertiserId than we introduce the noisy neighbour problem where one ad generates lots of traffic and blocks the other ads.
Those questions are great and definitely come up often with streaming setups. I'm no expert but ...
Partitioning with 10M Ads ... Yeah, if we just went with adId for partitioning, it would mean loads of shards or partitions, which isn’t practical. Instead, you can use consistent hashing or modulo partitioning to map multiple adIds to a smaller, manageable number of partitions.
I’ve become an annual premium member of HelloInterview after watching couple of your videos. It’d be really helpful if you started a chat/messaging platform for the paid subscribers, so that we could get our questions answered without relying on TH-cam.
Question: Why do we need the reconciliation mechanism? And why use a batch processing for the purpose? Do we know/assume stream processing would accumulate errors compared to batch processing (the latter being accurate?)?
Why this is not lambda architecture? It has both realtime and batch processing, so what is the difference?
Thank you, btw
I like your videos, I have learned a lot.
A couple comments on this video:
a. I think the system would benefit from a Redis in the click processor service, not the idempotency lock but a redirectUrl cache {adId: redirectUrl} to reduce reads in the Ad DB. It might be a MRU cache to avoid overloading the Redis.
b. I'm not sure why you are pushing Kinesis so hard in this solution, I mean yes I learned something about Kinesis, but it would be more practical just to place a Kafka that can handle the load peaks and has event history as well so it is possible to write the reconciliation procedures from it.
c. I learned about Flink, thanks. I used a redis aggregator in my own solution.
Thank you so much for your work!
Incredible video with excellent drawing and explanation.
For Kinesis hot shards, we don't know if an ad is hot beforehand. So are these ad_id 0-N always active? Is it ok to use x10 the amount of streams we need under normal circumstances?
For Flask, we have the same amount of flask servers as the Kinesis shards right? If the server dies, how will the new server keep track of the pointer from the old server? Are they statefull backups instead of stateless
This is a great question. In reality you can make predictions here. We know based on budget and historical performance which ads we’d need to be worried about before hand
I didn't understand the idempotency part, what if the same user sees the ad two times in a day. Wouldn't we like to count it
I rewatched and had some new thoughts. Wonder what are the costs of using streaming solution? I seems like the database for clicks that was used in batching solution is completely replaced by the streaming components, so benefits from having the previous database queries are lost?
34:52 streaming solution real time is by dumping to OLAP?
Hey Evan, your videos have been the best, seriously! The only suggestion I have is if you could use a dedicated mic that would be wonderful. The volume is too low even on max volume on my earphones.
Updates in latest video! Have a nice mix now :)
@hello_interview yay 😀
Why do we have to rely on Kinesis retention policy, when Flink ensures fault tolerance using check-pointing?
The final design has both real time data processing and batched processing. Why is it not lambda architecture?
Thank you so much for the explanation. However I more like you write the functional and non functional requirements instead of paste from the clipboard. The main reason is that it's make me thinking about what should be the requirements. It's feels more like code pairing
I really like the leveling information.
We initially said that we want to perform complex queries. In addition to time range, we want to check who were the users, and maybe where the clicks were from, etc. So that means we need more types of aggregations, for example by user id, by location, etc. How do we handle that? What if we get a query that we didn't prepare an aggregation for in advance?
This is great content. However, I would like to point out that the Lambda architecture includes both batch and speed layers, which process historical and real-time data in parallel. It’s not solely reliant on batch processing.
Question: For the sharding while processing the events through Kinesis, the adId was suggested as the sharding key. This doesn't look like the best approach. At scale, millions of ads are being run on the platform and a good share of them have high enough volume. Going by the presented logic, the number of shards would explode. What do you think about this?
Why did we not use a Redis instead of using Flink? Redis is also and in memory structure, that will meet our write throughput demand and we should scale it by sharding on Ad id?
best way to learn system design.
Few questions:
- Even in spark solution, how would you know the keys of which one to aggregate? Either you have to emit keys of which has changed or scan the db by time and get keys which is again costly.
- Aren't this falls under lambda architecture as we are using both realtime stream processing and batch processing to ensure data integrity?
21:48 I like how DB can be used for simplest case consistently in these approaches
The out-of-scope non-functional requirements seem to be more like out-of-scope functional requirements. I feel that (spam detection, demographic profiling, conversion tracking) are essentially features rather than characteristics of the system. How should I be thinking about this?
Honestly, fair point
I think bloom filter would be a good choice to check on duplicate impression id. I think, it is also supported by redis.
also at 41.28, after appending 0..n to the hot ad id and then distributing to different shards in kinesis, we 're letting flink handle finding the same ad_id from all different shards, so for a single flink job, which is responsible for a given adId, it will be responsible for checking every shard and see if the given ad ID exist in that shard, this will happen for each ad id, won't it be too much to check and aggregate for a flink ? how can it scale better being in memory as compared to other db services ? the more load will still ask for more instances, isn't it ?
As i watch this again, why do you say checkpointing is not required at 45:30 ish time? How does a job keep track of which time interval to process the click rate? Does flink internally maintain a state of the processed timestamp? (im not familiar with flink! )
I think he mentioned the retention period in kinesis should take care of the failed flink jobs and I looked up that flink has timestamps and also called watermarks to deal with out of order jobs
Amazing content! Very much appreciate you posting these 🙌
System design padawan here. I have a question about the hybrid approach .. what makes us trust the "Kinesis -> Connector -> S3 -> Spark -> Worker -> OLAP" results more than the "Kinesis -> Flink -> OLAP" results? Is it a guarantee where the connector always successfully writes the data to S3? or does Flink make some kind of tradeoff for speed? kind of confused about that piece and figured i'd ask. thanks again!
I am also curious about this
+1 on this question.
IMO, Spark is useful when you have really out-of-order events, like events arriving half an hour late or something. Then, by using Spark, you can reorder the events and get a more accurate count. On the other hand, for events that are only a few minutes late, you can configure Flink with the allowed lateness of a few minutes.
That being said, the cost of maintaining 2 code bases and ensuring that they are updated at the same time (to avoid triggering discrepancy alerts) doesn't seem worth it for such edge cases.
I'd be interested to hear @hello_interview's thoughts on this though.
> Is it a guarantee where the connector always successfully writes the data to S3?
Yes. That is something provided by AWS as a managed service, and it should meet their SLAs. AWS would've created fallbacks and fault tolerance to ensure that all events that are in Kinesis reach S3.
I don't see there being any compromise due to speed. Kinesis has retention policy, so the data isn't going anywhere, and S3 is highly available, so the data shall be written there as well.
These are Excellent! Please keep going.
Love the content! Thank you for making these!
Why do we use nosql for the write heavy db? Since the data is very structured and you might want complex queries…
Thanks alot, great explanation \o/
Regarding handling the idempotency, why can we not get away with only the ad impressionID? Is signed one actually required?
Hi Great explanation for a complex topic in such a easy way.Is Once only processing also critical when we are passing messages from kafka to flink.If we want one to enable once only processing on flink then checkpointing will be required juts a though
You can set it per message! From the docs, “Every Amazon SQS queue has the default visibility timeout setting of 30 seconds. You can change this setting for the entire queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue. When receiving messages, you can also set a special visibility timeout for the returned messages without changing the overall queue timeout.”
Do you think knowing OLAP is important for a senior/staff role? Having no experience with analytics, I'd just go for an RDS - guess it'd probably be fine?
Yah you’d be fine likely :)
@@hello_interview Thanks, also thank you for all the resources you've created. Amazing.
This is a very high quality video.
Looking forward to more great videos from you! :)
Can you please expand on below questions? or Link a small video/article if possible
1. How will "Click Processor SVC" know which AdID is popular/hard? 40:24
2. How will "Flink" handle further aggregation of AdID:0, AdID:1,..., AdID:N to AdID 40:43
1. Could be based on past performance or budget. Realistically, companies have ML models to predict this.
2. A single job will read from different partitions and aggregate.
If we use Cassandra with adId as primary key and timeStamp as the sort key, will the read be fast enough?
Not to aggregate over large periods
The final solution _is_ literally lambda architecture.
when will you post the next interview video? waiting for it about 1 month!!! really appreciate the effort.
Tomorrow!!
Do you think using Timeseries DB will be better choice here to store click related information?