This is nice explanation. I would like to see design of digital wallet which cover transfer, topup and withdrawal usecase and talks about concurrency and data consistency for users..
I have seen a lot paid and free courses on youtube. But luckily I have came across your channel recently when searched parking lot problem. Keep up doing the good work. Hats off sir.🙌
i would choose eventual consistency in this case, because the exact number is not that important to most users. eventual consistency will make scale of likes count easier
13:47 if you use kafka and your consumer manually commit offsets, then after consumer died zookeeper will re-arrange topics and assign the topic to another consumer. new consumer will continue from last comimted offset. considering that, we may want to put all the updates regarding the batch into single transaction, rather than do atomic updates for every tweet.
yep, that's a great idea! kafka msg is permanant store, restart from the last offset location, another way is that we do a write ahead log, every msg, we consume, write it to a file system, we then, can read it when the consumer restart, then we don't have repeat the work.
You ask many what-if questions, but answer none. You also don't justify any of your choices (such as creating the 3rd table instead of updating tweets table). This is not a quiz game; the video is not self-contained.
your questions in this video is quite intuitive. i am sure by asking right questions many will be able to think in terms of solving problem while continuing watching.
Thanks for covering the breadth of the different solutions. You started with brute force and worked up to suggest some kind of an optimal solution. However, rather than telling the "WHY" part of choosing a design pattern - you are leaving it up to the viewers to interpret and decide. Would have definitely subscribed and liked your content for wholesome information and depth.
so I am not sure how things are concluded. is the cache going to hold counters for all of the history that ever exists? and there's two types of consumers? one that writes individual like records to DB and one that stream aggregate and update increments to the Distributed cache?
Great video! precisely what's needed for interviews, thank you sir. But I was left with the doubt of what to do If a consumer dies in the middle of processing a like. I guess the answer is that Messages in a queue can be re-tried?
A question about write-through cache to distributed cache at 14:09. will redis node die before responding to MQ, will a single record be counted multiple times in this case?
I want to thank you for your video , but i am a bit disappointed at the way you ended the video abruptly. You did not complete the video and close the question in discussion. It seems like you have a paid service somewhere you answer all these questions or are preparing that for future. Kindly share your thoughts.
regarding adding a field to. tweet table vs like count table, i would choose like count table, the reason is that it will reduce a lot writes/update the tweet, otherwise it could be a performance issue. I would like to hear what you think?
A question about the example sql queries done at 11:20: is the "SELECT ... FOR UPDATE" statement still needed here when the UPDATE query would trigger an exclusive lock on the Tweet_Likes_Count row for tweetId "abcdefghij" anyway?
Yes because select and update are two different statements so it is possible that multiple threads run select in parallel and get the same value for the count and then all try to update count = count + 1. In this case, they all will change the count to the same value although it may need to be updated to count + X where X is number of concurrent calls coming at that time.
@@ThinkSoftware In mysql at least, I believe an update statement that does a query like "count = count + 1" would be atomic. A "select ... for update" would probably help if you needed to do additional work with the selected value afterwards, or if you wanted to update the column manually using the current selected value
Hey...I loved your videos, they are really helpful. Special thing is you ask a lot of questions during explanation and open my mind to think beyond one way of thinking. My question is - where do you discuss these questions which you ask in your videos? Many thanks... wishing you good health and wealth 😊
06:50 A service should be responsible for single task, so we must create a separate service for tweet likes. Assumption is that tweet create service have a less load but tweet like service has huge load so we can scale those services separately
Every tweet will have the tweet_like_count field. Shouldn't we add the tweet_like_count field to the existing Tweets table? What are the pros of creating a new table?
Let me know if you find this video useful. And please don't forget to like the video and subscribe to this channel. Thanks :)
This is nice explanation. I would like to see design of digital wallet which cover transfer, topup and withdrawal usecase and talks about concurrency and data consistency for users..
I have seen a lot paid and free courses on youtube. But luckily I have came across your channel recently when searched parking lot problem.
Keep up doing the good work. Hats off sir.🙌
i would choose eventual consistency in this case, because the exact number is not that important to most users. eventual consistency will make scale of likes count easier
13:47 if you use kafka and your consumer manually commit offsets, then after consumer died zookeeper will re-arrange topics and assign the topic to another consumer. new consumer will continue from last comimted offset. considering that, we may want to put all the updates regarding the batch into single transaction, rather than do atomic updates for every tweet.
yep, that's a great idea! kafka msg is permanant store, restart from the last offset location, another way is that we do a write ahead log, every msg, we consume, write it to a file system, we then, can read it when the consumer restart, then we don't have repeat the work.
You ask many what-if questions, but answer none. You also don't justify any of your choices (such as creating the 3rd table instead of updating tweets table). This is not a quiz game; the video is not self-contained.
your questions in this video is quite intuitive. i am sure by asking right questions many will be able to think in terms of solving problem while continuing watching.
Thanks for covering the breadth of the different solutions. You started with brute force and worked up to suggest some kind of an optimal solution. However, rather than telling the "WHY" part of choosing a design pattern - you are leaving it up to the viewers to interpret and decide. Would have definitely subscribed and liked your content for wholesome information and depth.
excited!
Excellent and thought provoking, I like your teaching style where you answer and give thought provoking questions.
so I am not sure how things are concluded. is the cache going to hold counters for all of the history that ever exists? and there's two types of consumers? one that writes individual like records to DB and one that stream aggregate and update increments to the Distributed cache?
Great video! precisely what's needed for interviews, thank you sir. But I was left with the doubt of what to do If a consumer dies in the middle of processing a like. I guess the answer is that Messages in a queue can be re-tried?
yah , data can be retrieved from message queues.
really liked your crisp explanation
Thanks 🙏
one corner case that you have not taken up sir
what if a person again clicks on the like button to dislike the tweet
how will you remove the like?
A question about write-through cache to distributed cache at 14:09. will redis node die before responding to MQ, will a single record be counted multiple times in this case?
I want to thank you for your video , but i am a bit disappointed at the way you ended the video abruptly. You did not complete the video and close the question in discussion. It seems like you have a paid service somewhere you answer all these questions or are preparing that for future. Kindly share your thoughts.
I do have a paid service but some of the questions could be topics for more videos.
@@ThinkSoftware yeah overall good start and everything but the last bit was a big disappointment, to be honest :(
th-cam.com/video/_b6FniEjiR4/w-d-xo.html
This video is a great supplement to the current video!!
regarding adding a field to. tweet table vs like count table, i would choose like count table, the reason is that it will reduce a lot writes/update the tweet, otherwise it could be a performance issue. I would like to hear what you think?
thanks for the comment 🙂
This is super helpful. Thanks
A question about the example sql queries done at 11:20: is the "SELECT ... FOR UPDATE" statement still needed here when the UPDATE query would trigger an exclusive lock on the Tweet_Likes_Count row for tweetId "abcdefghij" anyway?
Yes because select and update are two different statements so it is possible that multiple threads run select in parallel and get the same value for the count and then all try to update count = count + 1. In this case, they all will change the count to the same value although it may need to be updated to count + X where X is number of concurrent calls coming at that time.
@@ThinkSoftware In mysql at least, I believe an update statement that does a query like "count = count + 1" would be atomic. A "select ... for update" would probably help if you needed to do additional work with the selected value afterwards, or if you wanted to update the column manually using the current selected value
It all depends on DB transaction isolation level. With serializable isolation level, it may work but not with other isolation levels.
Appreciate 👍for sharing. Approach with likes in separate service looks better for me.
Thanks for the comment 😊
Hey...I loved your videos, they are really helpful. Special thing is you ask a lot of questions during explanation and open my mind to think beyond one way of thinking. My question is - where do you discuss these questions which you ask in your videos? Many thanks... wishing you good health and wealth 😊
Thanks for comment 😊. I am looking at various options for that.
Thanks
06:50
A service should be responsible for single task, so we must create a separate service for tweet likes.
Assumption is that tweet create service have a less load but tweet like service has huge load so we can scale those services separately
Watched. --
Every tweet will have the tweet_like_count field. Shouldn't we add the tweet_like_count field to the existing Tweets table? What are the pros of creating a new table?
Wish that you say less about stuff everyone already known