Brief Outline 00:00:59 Notification Service 00:02:12 Problem Requirements/Capacity Estimates 00:03:56 Fan Out Design Pattern 00:06:38 Idempotence 00:08:17 Idempotence Continued 00:11:17 Idempotence Continued 00:12:16 Bloom Filters 00:14:15 Client Connections to Notification Server 00:16:56 Final Diagram - Notification Service Thanks, Jordan~
Love your videos Jordan! What are your thoughts on creating videos where you build simple projects/programs that implement certain concepts from your systems design playlist? I personally can't find other channels that teach these concepts as well as you, and I feel like doing so would be pretty complementary to your already quality content.
Really appreciate recording and sharing these videos. They are super helpful. For idempotency part, I think we can give each message (per user dimension) one increasing number and keep maximum number on server side. By comparing service and client side's number, we can know if any message is missing or not.
Jordan, another great video, thank you! Would you be interested in doing the system design of a Google Sheets like application(not the collaboration functionality that you have already covered in Google Docs) but with features more closely resembling Airtable/Causal, focusing on real-time formula application?
Hey Jordan, watched all your videos...blah blah blah... great content love it blah blah blah... funny descriptions blah blah blah... now that all the formalities have been taken care of (lol I promise I'm a fan), could you make the OneNotes for all the Sys Design Interview problems accessible as well please? I love going back to the playlist 1.0 powerpoints for concepts, but it would also be really nice to have these oneNote accessible too - ideally in a way I can clone and scribble lotion and tissue boxes on and oh annotate stuff. Willing to send feet pics if you get the job done before my interviews :P
Haha - appreciate the formalities. Going to do the actual questions themselves when the series is done, have been punting on the concepts. Have some time today, so I'll have to see if I can clutch up.
Thanks for the video. How feasible would that be to build reverse mapping at scale? Sounds like expensive operation + at what cadence, we will either spent a lot of compute rebuilding revery sharding or there will be a signiy lag beyey users subscribing and starting to recrcey notifications for that topic.
I really don't think it would be super significant, we're really just going through a load balancer and into Kafka- and from there we can receive the message in flink and we're good to go
Hi Jordan I have a very basic question. Are we building the notification service for all notifications (from all applications) for all users ? In the final diagram, you have mentioned that flink node receives data from 2 sources. 1. Topic subscriptions of user. Basically topic_id --> userId's. 2. Messages from Notification Queue. topic_id --> message. Using above 2 sources, how does the flink sends the real time messages ? Some messages are user specific right for example one to one pings. Sorry if this sounds silly.
Yeah. I'm not very concerned with one to one pings in this video. If every user is connected to one notification server, we can just figure out where they are connected to and send the message there. In the general case, flink figures out the user IDs the message needs to go to, and via a load balancer/proxy sends it to all of the proper servers which then forward them to users via we sockets
Why is the subscription change events topic sharded on user id? I expected it to be on topic id as this stream needs to be joined with the events in notification queue which are also partitioned on topic id
Video Editing is too low 👎👎👎👎 Edit the video in a good style. Add nice b-roll animation when you are speaking or explaining something. This will increase visualization. Put proper subtitles, effects, transitions, and colors in the video. Insert the B-Roll (footage and animation) very well because it is absolutely low in your video. Do all this and then see how the video doesn't turn out good. 🍏
![CAMPปลิ้น | EP.76[2/2] เดินมาสิบกว่าป่า วันนี้รู้แล้วว่าโจโฉกลัวอะไร!](http://i.ytimg.com/vi/R7Yor-t2Au0/mqdefault.jpg)
Brief Outline
00:00:59 Notification Service
00:02:12 Problem Requirements/Capacity Estimates
00:03:56 Fan Out Design Pattern
00:06:38 Idempotence
00:08:17 Idempotence Continued
00:11:17 Idempotence Continued
00:12:16 Bloom Filters
00:14:15 Client Connections to Notification Server
00:16:56 Final Diagram - Notification Service
Thanks, Jordan~
Thanks for this work. Really helpful for me
Leon is the real mvp
Ive been waiting for this video. This is one of the basic services
Love your videos Jordan! What are your thoughts on creating videos where you build simple projects/programs that implement certain concepts from your systems design playlist? I personally can't find other channels that teach these concepts as well as you, and I feel like doing so would be pretty complementary to your already quality content.
Hey! It's definitely something that I've considered - when I run out of shit to do/post in a bit I may turn in that direction :)
My third favourite system design TH-camr just uploaded, sweet!
Hell yeah brother
What are the the other ones?
@@nepo2616 Jordan (with a life), and Jordan (with 50% of a life)
Lol 😂
Really appreciate recording and sharing these videos. They are super helpful. For idempotency part, I think we can give each message (per user dimension) one increasing number and keep maximum number on server side. By comparing service and client side's number, we can know if any message is missing or not.
Yep!
You forgot about 2nd requirement - Offline clients must be able to receive notifications later
Throw the notifications to a DB cache per user (so partition based on userId). Same deal as Twitter basically.
Jordan, another great video, thank you!
Would you be interested in doing the system design of a Google Sheets like application(not the collaboration functionality that you have already covered in Google Docs) but with features more closely resembling Airtable/Causal, focusing on real-time formula application?
Hey Harika! I'd have to think about this one a bit, versus whether it's just we have a formula per cell and then it gets applied on all client devices
Hey Jordan, watched all your videos...blah blah blah... great content love it blah blah blah... funny descriptions blah blah blah... now that all the formalities have been taken care of (lol I promise I'm a fan), could you make the OneNotes for all the Sys Design Interview problems accessible as well please? I love going back to the playlist 1.0 powerpoints for concepts, but it would also be really nice to have these oneNote accessible too - ideally in a way I can clone and scribble lotion and tissue boxes on and oh annotate stuff.
Willing to send feet pics if you get the job done before my interviews :P
Haha - appreciate the formalities. Going to do the actual questions themselves when the series is done, have been punting on the concepts. Have some time today, so I'll have to see if I can clutch up.
@@jordanhasnolife5163 +1 to OP. Really love your content and access to slides would be super helpful. Thanks a lot in advance!
Jordan - an even better approach to the fanout would be to store the last-sync time with the client and the created_at timestamp for the notification.
You'd need that timestamp per flink node as Kafka is only ordered per partition
Thanks for the video.
How feasible would that be to build reverse mapping at scale? Sounds like expensive operation + at what cadence, we will either spent a lot of compute rebuilding revery sharding or there will be a signiy lag beyey users subscribing and starting to recrcey notifications for that topic.
I really don't think it would be super significant, we're really just going through a load balancer and into Kafka- and from there we can receive the message in flink and we're good to go
Thank you.
Hi Jordan
I have a very basic question.
Are we building the notification service for all notifications (from all applications) for all users ?
In the final diagram, you have mentioned that flink node receives data from 2 sources.
1. Topic subscriptions of user. Basically topic_id --> userId's.
2. Messages from Notification Queue. topic_id --> message.
Using above 2 sources, how does the flink sends the real time messages ? Some messages are user specific right for example one to one pings.
Sorry if this sounds silly.
Yeah. I'm not very concerned with one to one pings in this video. If every user is connected to one notification server, we can just figure out where they are connected to and send the message there.
In the general case, flink figures out the user IDs the message needs to go to, and via a load balancer/proxy sends it to all of the proper servers which then forward them to users via we sockets
Jordan, Can you use the bigger mouse pointer, that way we can track, which box you are talking about? Thanks
I'm using an apple pencil, to my knowledge there is no mouse pointer.
Why is the subscription change events topic sharded on user id? I expected it to be on topic id as this stream needs to be joined with the events in notification queue which are also partitioned on topic id
See 19:26, typo on my part
only fans i m 🤣🤣
Video Editing is too low 👎👎👎👎
Edit the video in a good style.
Add nice b-roll animation when you are speaking or explaining something. This will increase visualization.
Put proper subtitles, effects, transitions, and colors in the video.
Insert the B-Roll (footage and animation) very well because it is absolutely low in your video.
Do all this and then see how the video doesn't turn out good. 🍏
See duplicate comment
First