Whatsapp Telegram System Design on AWS | Chat Messaging System Design Interview

แชร์
ฝัง
  • เผยแพร่เมื่อ 26 ส.ค. 2024

ความคิดเห็น • 120

  • @willchesson
    @willchesson ปีที่แล้ว +4

    Group messages: create a database instance just for group messages given the number of users. Each group chat has a unique identifier with an array of all the IDs of the member of the group. Similar to the explanation here, a message from one of the members hits the SQS, then is processed by the backend, which then sends the message to the other members of the group. The only real difference is the database instance dedicated to group messages and that the message itself is sent to various recipients.

  • @cloud_architector
    @cloud_architector 3 ปีที่แล้ว +20

    would be great to have system design udemy course, guys like the comment if you agree on that

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +3

      Since you were the first student on one of my Udemy courses, I will give this some thought Denys! Will keep you posted.

    • @cloud_architector
      @cloud_architector 3 ปีที่แล้ว +1

      @@cloudwithraj btw I have passed my aws pro architect, thanks to your videos too!!! I am continue now to learn and practice, so such course would be helpful for me

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      @@cloud_architector Nice Denys, congrats! AWS pro is the hardest cloud cert exam right now. Keep rocking my friend!

  • @rodoherty1
    @rodoherty1 2 ปีที่แล้ว +1

    Great video, Raj. I had to pause it to read up on how websocket servers can scale/failover while still maintaining connection to the clients. Once I had that straight in my head, the rest of the video made a lot more sense. More System Design videos, please :-)

  • @sathyasn6281
    @sathyasn6281 3 ปีที่แล้ว +2

    Just a try ...For Group chat, most of the functions same. But the DB will hold a separate table for the groups which has all the users who are added part of the group. The primary key will be the group name...The app should read from that table in case of group messages.. Still, it can use most of the same infra.

    • @motivewave001
      @motivewave001 2 ปีที่แล้ว

      Lets say a group id. Since the name can be duplicate and it would be hard to query the groups.

  • @vijaynaidun
    @vijaynaidun 3 ปีที่แล้ว +3

    Thanks for the video Raj. I would like to see more of these system designs with cost effectiveness as well.

  • @mikedqin
    @mikedqin 2 ปีที่แล้ว +3

    Hi Raj, thanks for sharing your idea on design. I have a question: Why did client app put messages directly to SQS behind the API Gateway. I know API Gateway can directly connect to SQS. In my opinion, the flow should look like this: Mobile App -> API Gateway -> ELB -> EC2 in ASG -> Save to Database, then send the message to SQS (Queue for 1:1) or SNS (Topic for 1:M), then Another EC2 instance as a consumer to get messages from SQS or SNS, and deliver the message through WebSocket to client app, then update the database status, and ACK to message delivery. Isn't it better flow? Thanks,

    • @TylerBundy260
      @TylerBundy260 8 หลายเดือนก่อน +1

      I know this comment is super old, just replying with my thoughts since I'm trying to improve my systems design process.
      I'd do API -> SQS without involving EC2 for a few reasons:
      1. You remove a point of failure sending items directly to the queue instead of waiting on and relying on the EC2 resources being available.
      2. You can use the queue length to scale your compute - you'd need to scale based on resource utilization otherwise.
      3. You mentioned having multiple sets of EC2 instances - seems like overkill, in my opinion.
      Less is more - having the SQS queue directly behind the API lets you offload more of your responsibility onto Amazon's platform. Since you're already paying for the queues, you might as well take advantage of their reliability. :)

  • @UmeshChandraDani
    @UmeshChandraDani ปีที่แล้ว

    Hi Raj, Please also make a system design video on e-commerce sites such as amazon, Flipkart etc.

  • @AlimHAli-eg7gz
    @AlimHAli-eg7gz 11 หลายเดือนก่อน

    Hey Raj... did you ever a do a WhatsApp Group chat system design video? If not - I love to see one!

  • @faisalnaseer798
    @faisalnaseer798 2 หลายเดือนก่อน

    really enjoyed it ... and subscribed for more

    • @cloudwithraj
      @cloudwithraj  2 หลายเดือนก่อน

      Awesome, thank you!

  • @guruprasadseeryadavenugopa351
    @guruprasadseeryadavenugopa351 2 ปีที่แล้ว

    Thank you very much for taking so much time to compile this video. For a message to be unique , the table should hold source, destination and message. So for a group message, the destination could be a set of destination ids( all members in group) instead of one destination id.

  • @saeeduchiha5537
    @saeeduchiha5537 2 ปีที่แล้ว +1

    Would be great to follow up with a database design video for chat applications. Great video, thanks.

  • @dvskr1
    @dvskr1 2 ปีที่แล้ว

    We can design table schema like this
    sender_id
    receiver_id( for group message received I'd type is group ,for individual type is single..groupid has 1 to many table having user details part of group...
    In LLD we can implement composiit pattern to read receipts, message deliveries ( or just fanout to send to groups but for individual buser only one user would exists)

  • @promptEngineerr1
    @promptEngineerr1 3 ปีที่แล้ว

    Thanks, raj, this series is really helping to understand different services uses as well.

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      Thanks Rahul brother 🙏

  • @ajitsalvi2062
    @ajitsalvi2062 2 ปีที่แล้ว

    Thanks. Its give good view to start thought process while designing.

  • @abdulrahmansaif354
    @abdulrahmansaif354 3 ปีที่แล้ว

    Wonderful Raj. The system design series helped me a lot in understranding the use cases of different AWS services. I wish if you could keep them on coming.

  • @NicoGanam
    @NicoGanam 3 ปีที่แล้ว +1

    Thank you again, I really like this kind of videos.

  • @frankiemak6066
    @frankiemak6066 ปีที่แล้ว

    very clear and useful info. Thanks for making this video. What is your amazon ID BTW? I am in Prime video.

  • @amitabhanath6913
    @amitabhanath6913 2 ปีที่แล้ว

    This is great Video, It really helped, Hats off. I have Two Question & will wait for your answer please...
    1- Assume per second 1 billion messages are being transferred.
    Will it create 1 billion queues? And incase of one queue & say its in FIFO order,
    once Alice, bob record is processed , then only the next record will process.
    This will make the last person/message(last in 1 billion) other party will receive it after 1 hour or so. How this will be taken care?
    2- If 2 user and A is served by server1 & server2 then how the communication/Co-ordination happens between the servers?

  • @anupbadharudheen2041
    @anupbadharudheen2041 2 ปีที่แล้ว

    I really like the way you explain keeping it very simple and starting from very scratch . Thank You so much . Now am a big fan of yours !! Hope you create some designs which also focuses on Networking , Edge networking , Usage of AWS backbone .

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว +1

      Thanks and welcome. I do plan to do some networking videos

  • @princevega1986
    @princevega1986 ปีที่แล้ว

    Great video

  • @mmu99
    @mmu99 2 ปีที่แล้ว

    OMG, how could I miss this awesome video from my cloud guru Raj..
    will explore your Udemy course soon guru..
    thank you! respect my authoritah :)))

  • @johnnybravo964
    @johnnybravo964 10 หลายเดือนก่อน

    A queue is not necessary. A user sends the message it goes to the backend and gets stored in the DB then it gets published on websocket to go to user. For unseen messages when the user goes to app it will do a request from server and get all messages that hasnt been seen yet

  • @jayateerthmirji1736
    @jayateerthmirji1736 3 ปีที่แล้ว

    Was looking for such an explanation since a long time . Very well explained. You earned a subscriber. Kindly make a video on system design/architecture of applications like Dunzo, swiggy or flipkart. Want to understand how DNS routing works specially when application is accessed globally.

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      Thanks Jayateerth! Will upload a retail online shopping design video in next couple weeks for sure, thanks for the suggestion.

  • @shivayaganti2263
    @shivayaganti2263 3 ปีที่แล้ว

    Hi Raj, all your videos so many useful yesterday I taken your udemy EKS course, please do video on VPC and Route53 how it's working in application level

  • @sayyednasir1752
    @sayyednasir1752 11 หลายเดือนก่อน

    in that case kinese service can be used for large client environment ?

  • @abknaveclem
    @abknaveclem 3 ปีที่แล้ว +2

    I have enrolled with your boto3 serverless course on udemy, It is quite thorough, I really liked it. Can you make a course on "System Design using AWS" on udemy? I like your way of explaining things. I hope this message gets out of the QUEUE and get an Acknowledgement...:D

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +6

      Thanks brother! I'll think about it. making a course takes lots of effort, since you have to design the chapters and lectures with gradual progression. I'll keep gathering feedback and if more folks want me to do this, will do it for sure. Thanks for watching and subscribing to my course!

    • @jetank2016
      @jetank2016 3 ปีที่แล้ว +1

      @@cloudwithraj I want that to sir jee. I have learnt a lot of AWS and hae done 4 certifications from practioner to pro level.. I have also cleared CKAD and CKA and Terraform associate.. I have also got formal training in design a microservice... All I want now is more videos and case studies like these so that I really know how things can be brought together

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      @@jetank2016 Thanks for the feedback bhai :). Amazing job on the certs - CKAA/D is super valuable right now.

    • @jetank2016
      @jetank2016 3 ปีที่แล้ว

      @@cloudwithraj Yes sir.. I have been in the industry for over 15 years now and sadly like most of us in India I just went to Project Management which is not a skill. So I have worked hard to upskills myself but just waiting for trainings that can help me put together all these learnings. You are the best trainer for me on TH-cam and udemy for these topics. God bless you

    • @cloud_architector
      @cloud_architector 3 ปีที่แล้ว

      @@cloudwithraj I would be glad to buy it

  • @vinodshalgar
    @vinodshalgar 3 ปีที่แล้ว

    Thanks Raj for this wonderful video. Your channel is near to 10k subscriber :) advance congratulations :)

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      Thank you so much Vinod 😀

  • @VaibhavPatil-rx7pc
    @VaibhavPatil-rx7pc 2 ปีที่แล้ว

    Hi can consider design, iot service having api with rate limit and process them in sequence using AWS services.

  • @snehap2127
    @snehap2127 2 ปีที่แล้ว

    You have thorough knowledge of the subject and you showcase with real time examples. Thats what i liked about your videos. Thanks for all these videos.
    Can we have more videos to describe projects using STAR method and also to explain challenges faced Vs solution provided for different domains like telecom, banking , e-com etc, please? ..

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Will definitely keep this in mind for future videos Sneha

  • @rightgateslearning
    @rightgateslearning 3 ปีที่แล้ว

    Thanks Raj, great explanation. I previously faced with a scenario like this and could not explain it very well. This would go along way for me. I'm currently working on your Serverless course on Udemy.

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      Thanks Oladele for supporting my Udemy course and watching this video. I am glad you found this video helpful.

  • @sayyednasir1752
    @sayyednasir1752 11 หลายเดือนก่อน

    Very Good technical videos.....Thank you..

    • @cloudwithraj
      @cloudwithraj  11 หลายเดือนก่อน

      Thanks for watching

  • @nightriderism
    @nightriderism 2 ปีที่แล้ว

    It is very very understandable... Raj Keep rocking..

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Thanks Selvan for the kind words 🙏

  • @totsubo2000
    @totsubo2000 2 ปีที่แล้ว

    Again great content. A general comment, your videos are great but you seem to gloss over the DB schemas/data models; you just point an arrow to a DB and don't really go into the details. Data models and schemas are something almost all system design interviewers will dig into.
    Some questions:
    1. You are using DynamoDB but have not defined the data model/table structures. Could you expand on this as I'm sure this is one of the first things the interviewer will ask :)
    2. How is the web socket connection between the Sender and the backend established? The Sender connects to the APIG, and the SQS is between the Backend and the APIG. In my mind the Backend has not direct link back to the Sender.
    3. How would you implement the feature that allows users to list all the conversations they have? I'm curious to see how this type of relationship is implemented in a noSQL database like DDB
    4. You hand-waved away a lot of the actual logic, putting it in EC2. I would have expected a micro services approach. Using EC2 makes it feel like a monolithic application.

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Great feedback, will keep this in mind for future

    • @ryanl7487
      @ryanl7487 2 ปีที่แล้ว +1

      For #2 - I believe the client is connected to the API Gateway via websockets. But internally between the APIGW and SQS, and between SQS and the backend compute - these are just standard (believe http) calls
      Agree for #1 - some simple data modeling for dynamodb would be good to see

  • @bandasree
    @bandasree 3 ปีที่แล้ว

    Nice video, very well explained. Yes, pls do follow-up video on group messages.

  • @babusivaprakasam9846
    @babusivaprakasam9846 2 ปีที่แล้ว

    Great video Raj. Much appreciated.

  • @mti2fw
    @mti2fw 2 ปีที่แล้ว

    Awesome video again! I would like to see a Google Authenticator app system design too :)

  • @heyitsaakash
    @heyitsaakash ปีที่แล้ว

    Hi, thanks for this it's really helpful...for the old messages that have already turned green or old messages/history is there a requirement for cache to be maintained in the phone or on the server side that need to be synced with devices

  • @dhirajjha8470
    @dhirajjha8470 3 ปีที่แล้ว +1

    Just one thing wanted to mention, For text message it's fine, but in what’s the app we send images and videos for that storage we need to consider AWS S3 to store those files.

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      You are correct. For images you need to resize if needed and save in S3.

  • @sivabenguluru1241
    @sivabenguluru1241 3 ปีที่แล้ว +1

    Well explained

  • @nehalverma1444
    @nehalverma1444 11 หลายเดือนก่อน

    Do we have video created for system design for Hotstar?

  • @ravikumarkumashi7065
    @ravikumarkumashi7065 2 ปีที่แล้ว

    Thanks for such an amazing vedio raj, i am definitely going to enroll some of your udemy courses!!
    you are amazing

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Thanks Ravikumar, glad you found my videos helpful

  • @phyominhtun2503
    @phyominhtun2503 3 ปีที่แล้ว

    Cool brother! Would like to see more architecture whiteboard session from you :D

  • @banji007
    @banji007 3 ปีที่แล้ว +2

    You talked about bringing code to the edge for last online. However Tina needs to go all the way to BE server to retrieve last online status of Bob. Can Tina subscribe to a SNS Topic to retrieve Bob Last online status. Can there be some other Architecture design ?

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      Good question Anirban! SNS is not ideal in this case coz it is push based notification. So for SNS to work, only when Tina opened Bob's messages in her app, Tina has to let Bob know that Bob needs to send the last online status, however Bob might not be online that time. Also SNS requires explicit acceptance from Tina (same way when you accept every time you set up SNS/email sub). And with growing users, phone to phone SNS connection becomes harder (who manages the topics, subs etc.). That's why Tina fetching the status from the backend is implemented. Hope this helps.

  • @sparker1170
    @sparker1170 ปีที่แล้ว

    Can you design us microservice architecture sir

  • @vinayswarup1664
    @vinayswarup1664 2 ปีที่แล้ว +1

    Can you please share a video on how to make search engines such as trivago on AWS

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Will keep in mind for future video Vinay

  • @debashishbhattacharya9449
    @debashishbhattacharya9449 2 ปีที่แล้ว

    @Cloud With Raj: Thanks for the video: I'm assuming for Group Messaging we have to use SQS Fanout method, and if we want specific order to deliver the message then SQS FIFO queue.

  • @dhirajjha8470
    @dhirajjha8470 3 ปีที่แล้ว

    Excellent explanation Dada ..!!

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      Thanks Dhiraj bhai. Glad you found it useful.

  • @ankurbhanot496
    @ankurbhanot496 3 ปีที่แล้ว

    hey raj, can you give some tips for whiteboarding during amazon aws interviews? like how to start and process?

  • @tango12341234
    @tango12341234 2 ปีที่แล้ว

    I read that lambda has a limit of 1000 concurrent invocations. How would it then be possible to use Lambda as a backend instead of EC2 especially with the amount of requests and users?

  • @shadyapp7416
    @shadyapp7416 ปีที่แล้ว

    What do you mean by backend here?

  • @agmos01
    @agmos01 2 ปีที่แล้ว

    Great video Raj!

  • @akram.n900
    @akram.n900 2 ปีที่แล้ว

    Great

  • @sachinjadhav878
    @sachinjadhav878 2 ปีที่แล้ว

    Nicely Explained

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Thanks for watching Sachin

  • @anshikagupta4931
    @anshikagupta4931 3 ปีที่แล้ว

    Hello, thanks for the detailed analysis, can you please also include a follow up video for group chats and how authentication using cognito etc can be integrated with this . would be super helpful

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      Will do in a future video Anshika, thanks for the suggestion

  • @ojaswiankursingh8508
    @ojaswiankursingh8508 3 ปีที่แล้ว

    Great Video..Quick question if we aren't going for Lambda for the compute layer, why did we chose Fleet of Ec2 with autoscaling and not ECS or Fargate? Keep making more system design videos 😀

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      Thanks Ojaswiankur for the kind words. I did this one on EC2 coz EC2 is still the most common compute layer. If you are comfortable with Lambda/Fargate, you can use them in the interview instead. I always suggest going with the compute layer that you are strongest in.

    • @ojaswiankursingh8508
      @ojaswiankursingh8508 3 ปีที่แล้ว +1

      @@cloudwithraj makes sense..We want people to be comfortable with this and then can take to next level as they feel comfortable with..great stuff..

  • @nishanthanr605
    @nishanthanr605 2 ปีที่แล้ว

    why we can go with Kafka instead of sqs?

  • @smita30apr
    @smita30apr 3 ปีที่แล้ว

    Hi Raj! You talked about use of microservices but you have only one database, microservices should have their own databases right? Like in this example, if you talk about 'Status' microservice then it should not talk to the main database directly right?

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      Great question Smita. For DynamoDB there is no concept of database holding tables. It's just multiple tables and each table can have separate scaling criteria. For microservices each API should "try" to interact with a non overlapping tables, but sometimes that's not feasible. Hope this helps.

  • @aishwaryamane9153
    @aishwaryamane9153 ปีที่แล้ว

    I completed aws course i interested in job

  • @jetank2016
    @jetank2016 3 ปีที่แล้ว

    Hi Raj, Many thanks for taking pains and teaching us. I have query, you are talking about microservices here.. However, in the diagram you are using EC2 has the back end where the application code is and using dynamo DB as DB. I am confused as to where does the microservices fit in here?

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      Great question Ankush. In this case you run one microservice in one EC2 and put it under it's own auto scaling group. This way each microservice can scale independently since they are under separate Auto Scaling Groups. Sometimes enterprises will implement more than one similar microservice into one EC2. But idea is to enable code implementation and scaling irrespective of each other. Hope this helps.

    • @jetank2016
      @jetank2016 3 ปีที่แล้ว

      @@cloudwithraj Ok thanks Bhai... Actually.. Since Microservice is always linked to EKS/K8s in general and therefore, I have just forgotten that its still just an application that can be deployed on EC2 and doesnot have to be a container all the time :)

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      @@jetank2016 haha yeah, it is natural. Most enterprises move to EC2 first, then modernize to K8s. I purposely avoided K8s as backend to focus on the general design and not bogged by K8s stuff. If you are confident on K8s, you can draw the design with K8s as backend as well.

    • @jetank2016
      @jetank2016 3 ปีที่แล้ว

      @@cloudwithraj Agreed. Super thanks..

  • @AnhNguyen-vu7mc
    @AnhNguyen-vu7mc 3 ปีที่แล้ว +1

    if there are a million users, how does the backend know which websocker connection is for Tina?

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      Great question Anh. For each websocket connection, the connection id is stored with the end user's cellphone number. When the sender's message comes to the server along with recipient number, the backend looks up the websocket connection id and send the message appropriately.

  • @lakinduhewawasam8635
    @lakinduhewawasam8635 2 ปีที่แล้ว

    Wouldn’t it be better to use Lambda over EC2 ?

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      Like I always say, in interview go with the compute option you are most comfortable with, all have pros cons. Choose amongst Lambda, EC2, and EKS based on your comfort level.

    • @lakinduhewawasam8635
      @lakinduhewawasam8635 2 ปีที่แล้ว

      @@cloudwithraj Yeah, I agree. loving your videos btw. Recently designed a chat service for this project im working on and your design inspired me!

  • @vardhansolanki8573
    @vardhansolanki8573 3 ปีที่แล้ว

    Don't mind me. Just commenting to help the TH-cam algorithm

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว +1

      haha thanks brother! Much appreciated!

  • @TDnothing
    @TDnothing 3 ปีที่แล้ว +1

    try 1.25x play speed, thank me later

  • @hemantsingh5840
    @hemantsingh5840 2 ปีที่แล้ว

    bro wtf , this video is from 2021 but his hair style is from 2000s haha🤣🤣

  • @promptEngineerr1
    @promptEngineerr1 3 ปีที่แล้ว

    For Group messaging we can create room and add individual users to the specific room just like whatsapp. So if one message is in one room so that message will be part of that room and visible to the user in that room only. This high level idea, let me know how we can it scaleable. Look forward to the next video.

    • @cloudwithraj
      @cloudwithraj  3 ปีที่แล้ว

      You are at the right path Rahul. I'd push you a bit and tell me what ideas you have to make it scalable, then I'll comment

  • @cloudwithraj
    @cloudwithraj  2 ปีที่แล้ว

    My Rocking Systems Design course is out now in Udemy! Limited time discounted link to get my courses for $9.99 - bit.ly/3Eku9RH

    • @laxmisuresh
      @laxmisuresh 2 ปีที่แล้ว

      Thanks for sharing your knowledge and experience. The coupon code "LEARNSYSTEMDESIGN" is expired. Can we get new coupon-code Raj?

    • @cloudwithraj
      @cloudwithraj  2 ปีที่แล้ว

      @@laxmisuresh Yeah, try LEARNCLOUDGETLOVE. All my course discounts are here - bit.ly/3Eku9RH

  • @GopalRoy-nn6ft
    @GopalRoy-nn6ft 4 หลายเดือนก่อน

    What is DLQ ?

  • @samjones4327
    @samjones4327 2 ปีที่แล้ว

    Good Day Raj! Thank you very much for walking us through this awesome design! I'm just starting out in cloud, specifically AWS. I'm studying for the CCP exam currently and I was excited to be able to understand the services you outlined to make this architecture work! It all made perfect sense! I would love to see the group chat architecture solution for sure!! Again thank you so much, be well, be safe and cheers!