Great explanation of the fan-out approach! One challenge I see with this solution is when a post has thousands of followers (not quite a celebrity's reach, but still substantial). The task worker would need to make thousands of individual database calls, which could keep it busy for an extended period. Additionally, any failure during this process might lead to missed timeline updates. If the worker restarts from Kafka, it would need to redo all those DB writes, as it wouldn't have a record of which ones succeeded before the crash. Is this a real issue, and if so how could it be managed effectively?
Really neet content as always. i think the last cold start question is more about how to generate a user's full timeline when they first join Twitter. Like what would twitter do if the newcomer have not followed anyone yet... They can't be showing a blank page, right? I was thinking maybe having the newcomer select a few topics/categories that they are interested in, and from the TopK analysis of all the tweets, just show them some top clicked tweets. Or maybe straightup recommend some celebrities when the newcomer joins twitter. After all, fetching celebrity is a poll based mechanism.
One more question, if celebrity 1 follows celebrity 2, how to build celebrity 1's timeline? poll based method as everyone else? Or maybe build some sort of smaller social circle within Twitter just for the celebrities?
why the followerDB is ready heavy? it will have the same number of read request as tweets get created. only difference is that the processing is going to take time for those read queries. that does not mean the read calls are going to be huge.
NOTES & FILES:
- excalidraw file: excalidraw.com/#json=wOCs42zY3ba3_UWsCA1W2,-ihYDBv-ek0brC1Ix7dWWw
- screenshots of the excalidraw file: imgur.com/a/iARP5so
- text file: pastebin.com/m9DiQwNg
Great explanation of the fan-out approach! One challenge I see with this solution is when a post has thousands of followers (not quite a celebrity's reach, but still substantial). The task worker would need to make thousands of individual database calls, which could keep it busy for an extended period. Additionally, any failure during this process might lead to missed timeline updates. If the worker restarts from Kafka, it would need to redo all those DB writes, as it wouldn't have a record of which ones succeeded before the crash. Is this a real issue, and if so how could it be managed effectively?
Really neet content as always. i think the last cold start question is more about how to generate a user's full timeline when they first join Twitter. Like what would twitter do if the newcomer have not followed anyone yet... They can't be showing a blank page, right?
I was thinking maybe having the newcomer select a few topics/categories that they are interested in, and from the TopK analysis of all the tweets, just show them some top clicked tweets. Or maybe straightup recommend some celebrities when the newcomer joins twitter. After all, fetching celebrity is a poll based mechanism.
Starts at 13:06
Thank you for your work
One more question, if celebrity 1 follows celebrity 2, how to build celebrity 1's timeline? poll based method as everyone else? Or maybe build some sort of smaller social circle within Twitter just for the celebrities?
why the followerDB is ready heavy? it will have the same number of read request as tweets get created. only difference is that the processing is going to take time for those read queries. that does not mean the read calls are going to be huge.
Yes, you're right. However, I believe query patterns can also help identify the best database choice, not just read/write traffic.
Thankyou❤
Thank you
Very nice ❤