Design a Distributed Message Queue - System Design Mock Interview

Make sure you're interview-ready with Exponent's system design interview prep course: bit.ly/3MWBqxs
Read our complete guide to system design interviews here: www.tryexponent.com/blog/syst...
A senior engineering manager, designs a distributed message queue.
When designing a distributed message queue, consider both functional and non-functional requirements. Use a topic-based system with a pull model for scalability and control. Use a key factor for sharding. Store data with SQL, NoSQL, or a write-ahead log system, and use metadata storage for important info.
To ensure fault tolerance, use a leader-follower approach with a coordination service and acknowledgment for write success assessment. Follow a structured approach and incorporate multiple producers and consumers for different topics to design an effective message queue.
Chapters (Powered by ChapterMe) -
00:00 - Intro
00:13 - Functional and distributed queue requirements
01:42 - Queue types topic base, fan out, order creation
02:36 - Direct message queues in ecommerce
06:39 - High-level design for messages with producers
11:10 - Scaling consumer for faster consumption
12:20 - Different options for queue design
13:29 - Key and sharding for message storage
13:56 - Different sharders for different buyers
14:21 - Storage options SQL, no SQL, write ahead
15:00 - SQL-based log management solution achieves high performance
18:01 - Partitioning 300TB files using buyer ID
18:52 - Partitioning, segmentation, metadata storage for Q
21:09 - Data storage, consumption, and fault tolerance
25:47 - Replicating messages in Kafka
28:57 - Faster interview questions highlight advantages of depth analysis
29:18 - System design interviews short summary, follow pattern
31:20 - Check-in with interviewer helps prepare for interview
Watch more system design videos here:
- Meta engineering manager answers a rate limiter interview question: • System Design Mock Int...
- Google SWE answers an algorithms interview question: • Google Software Engine...
- Google TPM answers Tiktok system design interview question: • System Design Mock Int...
- Flipkart EM “Design Amazon Prime Video” system design interview question: • System Design Intervie...
👉 Subscribe to our channel: bit.ly/exponentyt
🕊️ Follow us on Twitter: bit.ly/exptweet
💙 Like us on Facebook for special discounts: bit.ly/exponentfb
📷 Check us out on Instagram: bit.ly/exponentig
📹 Watch us on TikTok: bit.ly/exponenttikttok
ABOUT US:
Did you enjoy this interview question and answer? Want to land your dream career? Exponent is an online community, course, and coaching platform to help you ace your upcoming interview. Exponent has helped people land their dream careers at companies like Google, Microsoft, Amazon, and high-growth startups. Exponent is currently licensed by Stanford, Yale, UW, and others.
Our courses include interview lessons, questions, and complete answers with video walkthroughs. Access hours of real interview videos, where we analyze what went right or wrong, and our 1000+ community of expert coaches and industry professionals, to help you get your dream job and more!

[@tryexponent]

Make sure you're interview-ready with Exponent's system design interview prep course: bit.ly/3MWBqxs

[@samirvasani2186]

this video explains more on how Kafka works then a pure distributed system..Content can be improved by mentioning a use case so that viewers can corelate the problem and the solution.Thanks

[@tryexponent]

Hey samirvasani2186, thanks for the feedback!

[@soumikdc]

Although it was good explanation of distributed queue, it hardly seemed like an interview. Usually the interviewer asks multiple counter question and takes the discussion into different direction as per his/her choice but here there was no such thing. This could have been more interactive.

[@tryexponent]

Hey soumikdc! Thanks for the feedback. Appreciate it!

[@RicardoBuquet]

This guy is all over.. I would remove this video from your list. And the interviewed seems to be watching videos on youtube. There was barely no questions. This is not how interviews goes.

[@DebasisUntouchable]

the system design mock interview was good, but there are things were not fine,
1) It started with distributed queue, but you ended doing pub-sub
2) 100 bytes and initial calculation of topics and messages was not convincing,
3) You said there is a tradeoff between latency and throughput, that is not so, you are not thinking in distributed way.
4) You said, WAL, Consistent Hashing based on buyer_id and then Leader-Follower architecture, I guess you are bit confused.
5) And a segment of 10KB? that means for a message of 100 bytes, you will have 100 messages in a segment? It does not make sense right to have so less segment size.
6) And the consumer only reads from Leader not from replica?
7) Leader syncs to replica, you didn't say whether its async or sync replication.
On the positive side, I liked your handling of meta-data and cluster service inclusions.

[@afraz-khan]

less than 15 mints to design and explain this huge and complex system? chill brah :) but anyway, a greate video and very fluid approach to design. was helpful. thanks guys.

[@tryexponent]

Glad you liked it, Afraz!

[@Deepakkumar-dy1en]

Great content❤
About leaders and followers, i think there should be leaders for each topic plus partition instead of just topic.

[@raw_tech_with_tom]

A distributed message queue. I think that is different from a message stream. I see more of message stream like kafka design than an actual queue like rabbitmq.

[@jmitesh01]

segment size could be of 1 GB as in Kafka(vs here taken as 10KB which is quite small).

[@user-of1vc]

Which application/website is being used to draw diagrams in the video?

[@tryexponent]

Hey user, the whiteboard application being used here is Whimsical. They have a free and paid version so do check them out!

[@rajeshkishore7119]

Overall it was good but implementation has lot of complexity, could not find any thoughts on implementation for any of feature thoroughly

[@amylovescoffee4759]

Is this a typical PM interview or more for engineers?

[@tryexponent]

Engineering! These system design questions are usually asked in engineering management and other technical roles

[@ranjan19able]

which editor he is using to explain system design?

[@tryexponent]

Hey ranjan19able! If you are talking about the whiteboard, it's called "Whimsical"!

[@qutaibabs1]

what whiteboarding softare he is using ?

[@tryexponent]

Hey qutaibabs1! The whiteboard used here is called "Whimsical"

[@qutaibabs1]

love it @@tryexponent

[@LustreSucre]

What app was used in this video?

[@tryexponent]

Hey LustreSucre! The whiteboard tool used here is called "Whimsical".

[@pioneerx9066]

The interviewer didn't define what he meant by distributed. This just turned into a pub-sub topic system.

[@happy..1907]

I don't use eBay