I would really appreciate it if anyone can answer this question. When they say that the data from the write api comes in and it goes to army of redis instances and does a fan out write, how does it actually find where (in which redis instance) the user timeline for the followers is present? Does that also come from the social graph service? If so, how does social graph service keep track of which redis instance has which users timeline? Basically, I am trying to understand how do we know that we need to query a particular redis instance for this user's timeline.
I am not sure about this case but most of the databases use sharding (horizontal partitioning) where they partition the data based on some key using hashing (consistent hashing mostly). The social graph service must be returning user ids of the followers to the fan out nodes. The fan out nodes must then be determining the exact redis instance by applying the same hashing algo to the follower's user ids. Again, It's a guess.
I believe they use consistent hashing. He also mentioned they have a replication factor, which means data would be present on at least replication factor number of instances. There are multiple strategies on how this replication is done within nodes in the ring. I guess that's a discussion for a different day.
Redis stores integers in their integer representation, so for string values that actually hold an integer, there is no overhead for storing the string representation of the integer. redis.io/commands/incr#:~:text=Redis%20stores%20integers%20in%20their,string%20representation%20of%20the%20integer.
code and code ?? REAL WORLD samples ? Demos are cool, but it is time to talk about the real world! The Succinctly series This frustration translated into a deep desire to produce a series of concise technical books that would be targeted at developers working on the Microsoft platform. We firmly believe, given the background knowledge such developers have, that most topics can be translated into books that are between 50 and 100 pages. This is exactly what we resolved to accomplish with the Succinctly series. Isn’t everything wonderful born out of a deep desire to change things for the better?
The best article ever
I would really appreciate it if anyone can answer this question. When they say that the data from the write api comes in and it goes to army of redis instances and does a fan out write, how does it actually find where (in which redis instance) the user timeline for the followers is present? Does that also come from the social graph service? If so, how does social graph service keep track of which redis instance has which users timeline? Basically, I am trying to understand how do we know that we need to query a particular redis instance for this user's timeline.
I am not sure about this case but most of the databases use sharding (horizontal partitioning) where they partition the data based on some key using hashing (consistent hashing mostly). The social graph service must be returning user ids of the followers to the fan out nodes. The fan out nodes must then be determining the exact redis instance by applying the same hashing algo to the follower's user ids. Again, It's a guess.
As per my understanding, they use a hash lookup using the user ID to locate the redis instance that holds the particular timeline
I believe they use consistent hashing. He also mentioned they have a replication factor, which means data would be present on at least replication factor number of instances. There are multiple strategies on how this replication is done within nodes in the ring. I guess that's a discussion for a different day.
Thanks everyone! That cleared it up.
One question! How it stores an element in a list with twitterid, userid and bits to get only 8+8+4 bytes? redis use string as data type...
Redis stores integers in their integer representation, so for string values that actually hold an integer, there is no overhead for storing the string representation of the integer.
redis.io/commands/incr#:~:text=Redis%20stores%20integers%20in%20their,string%20representation%20of%20the%20integer.
It would be great if could share the presentation.
Gilfoyle?
He has exactly my thoughts.
09:07 Redis, memchache
How old is this talk?
The slides say its from 2013
code and code ??
REAL WORLD samples ? Demos are cool, but it is time to talk about the real world!
The Succinctly series
This frustration translated into a deep desire to produce a series of concise technical books that would be targeted at developers working on the Microsoft platform.
We firmly believe, given the background knowledge such developers have, that most topics can be translated into books that are between 50 and 100 pages.
This is exactly what we resolved to accomplish with the Succinctly series. Isn’t everything wonderful born out of a deep desire to change things for the better?