1 more thing to add; sorted set, internally is implemented with skip lists and hashtables in Redis. Thats why adding value to sorted set is O(logN) on average and searching is O(logN + M). If you bring this up, congratulations you passed your interview
On the contrary, if I brought up this statement in an interview after having seen it on TH-cam, I just blew the interview. Any interviewer when faced with a candidate that is trying to impress by throwing out irrelevant buzzwords, will ask "oh yeah, tell me more about how to implement skip lists".
@@abhijit-sarkar couldn't agree more. Mentioning such points on your own always brings a follow up question, now I have to also read how skip lists works.
@@bhawanasahu3000 i think if you know how it works under the hood and what implications that has on time complexity that is extremely good regardless if you have further deep knowledge of those further implementations..
That was too awesome! I'm using redis too much recently and have some core concepts that will help at daily basis. If you post more topics of that, just announce and you'll got a viewer. Great content.
Hi i just want to say your work is truly fantastic and you are an excellent educator! I started watching your videos to prepare for system design interviews and your explanations and examples are top notch. I'm motivated to continue watching not just to prep but also to expand my knowledge and improve my craft. Eagerly awaiting the great work you'll do next!
Very informative and helpful video as always. I would appreciate if the visualization of the internals could be a bit more detailed and in sync with what you describe, rather than just a couple of boxes. Regardless, this is a refreshing take to step away from the usual system design videos to dive into technologies that are useful in both interviews and real life. Hope to see more of this in the future.
Great content. This is a great resource for learning before i apply Redis to my own projects. One thing I was wondering was if your content would be able to be added into Spotify so I could listen along on the go?
I really enjoy your content on the website, the videos as well as posts on reddit. Can you make another video on behavior/leadership principles interview next? I would like to learn how to tell a convincing story about my leadership potential.
Wait .... I thought I clicked on a Dua Lipa music video. 😀. Just kidding. Stefan I love this - clear, concise, articulate and I learned something. Keep 'em coming.
It's a great deep dive. I'd like to give a suggestion, Can you guys add chapters to the video, so that, if I want to rewatch again about certain portion of the video then that would be easy thing to do.
You should have chapters on this one! If you look at the description "Key Moments" will take you to the relevant sections. LMK if we're not doing something else we should be doing!
@@hello_interview I see that, you missed to add "00:00" time stamp in your "Key moments", that's causing YT to not auto chaptering your video. I have checked couple of other videos in the channel, videos which are having 00:00 in their key moments, respective chapters are added.
Loved it! Want to see more on how communication between service on global scale like using kafka with async or consistent connection around this please, if you get the idea, Sorry i am bad at explaining
At 22:55 how does hashing tweet IDs help if we're looking for the most liked tweets overall? Or were you referring to getting top liked tweets for specific keywords when splitting that key across multiple nodes?
Hi, you talked about append a random number to solve the hotkey issue. But when you want to retrieve the data, you won't know the key because you append a random number.
How Redis maintains this cached data internally? does it maintains in some kind of tables or files? If I want to store 10GB of some relational data tables data or query output, should we need approximately same out of RAM for REDIS setup? Could you please provide the details how to understand the internals
Redis is an in-memory datastore, if you want to store 10GB of data in Redis, you need 10+GB of RAM, there's no paging out to disk. When operations happen they are added to an AOF (append-only file), but that's the full extent the disk is involved for garden variety Redis.
@@hello_interview sorry, what's the approach? I think somewhere else in the video talks about it, but I also somehow did not understand that. Is it roughly the following? Generate X random suffixes. Maintain a map on the (Redis') client side of . Whenever we want to do some sort of "write", we get a random suffix from our list and append it to our key in the Redis command. Whenever we want to do some sort of "read", we get _all_ random suffixes (X of them) from our set, make X reads from Redis, and aggregrate on the client side. ---- Or maybe it's sometimes the opposite way. For example, in a cache, I'm guessing we'd want to write our value to each and every one of the key+randomsuffix. Then for a read, we can (randomly) pick one of the suffixes. ---- Maybe the missing link (for me and others similarly confused) is how/where the random suffixes are managed. There's also the question of how we (automatically?) identify a hotkey, or develop an algorithm that generates (or increases) the number of suffixes depending on some sort of access rate.
You can use Kafka as pub/sub! Differences in durability (Redis in memory, Kafka persists to disk), which then has consequences on other things like ease of disconnection, etc. But they can both be used.
Redis pub/sub is really dumb, at-most-once delivery of messages. You can think of it as a hashmap from a topic (string) to a socket connection to a subscribing server with all the associated problems (messages might get lost?) and benefits (incredibly lightweight, low-latency etc.) Kafka is more full-featured. Kafka has message persistence, you can make guarantees on delivery, etc.
The problem of hot key is clear but solution of hashtag is very difficult to understand. do we have to handle that all in the client? like figure out what are the hot keys?? and what will adding hashtag do really?
@@hello_interview Geohashing algorithm is too complex. I just cant take in all this diagrams. I also dont understand how proximity systems can search something inside the circle using square blocks of geohashes
@@maxvettel7337 Gotcha! I'll earmark this for follow-up. The nice thing is you don't necessarily need to know about the geohashing internals to use something like Redis' proximity search in practice!
Can i book sys design coaching sessions on hello interview or is it only for mock interviews?I'm a senior engineer and I've just started preparing do you recommend to jump straight into mocks?
@@hello_interview Awesome, thanks so much for the quick response. I really appreciate what you guys are doing, this is a great resource and y'all rock!
In a cache setup you expect that sometimes you'll have a cache miss and have a way of handling it. If we have a key `foo` that is particularly hot, we can start appending a number 1-9 `foo1`, `foo2`, `foo3`, ... Each of these keys will reside on a different node/shard in the cluster. If the key is actually hot, all the keys will be written eventually and the reads will be spread across the nodes in the cluster.
@@hello_interview if we append this to all keys then it would reduce the effective size of our cache and would have a lot of cache misses. If we do only do this for keys that are "hot", then how do we detect if that key is hot since if we use a redis cache to keep track of how many times a key has been hit, we run into the same issue since that one redis partition is getting hit hard keeping track of the amount of times that key has been accessed.
@@goodmew123I guess you would have to know beforehand. Also, there might be some heuristic for that, for example, if the number of likes on a recent tweet is high, then many people will be querying that number of likes.
In practice? Substantially faster and cheaper if you can accept some of the (many) compromises. In an interview setting it can be helpful to have a few multifaceted tools vs having to know all about Kafka (video on that soon). If your business was entirely built around append-only logs I probably wouldn't take Redis as an acceptable substitute, though it works in a pinch.
For hot key you really need to have caching in the service that's hitting Redis, unless you have a ton of hot keys a small LRU cache should work. Although this trades off some consistency unless you have a real short TTL :) - Dynamically sharding keys adds complexity to the application layer - Dynamically scaling read replica nodes seems wasteful (you dont need to duplicate the whole node) All solutions seem non optimal.
14:40 the nice thing about this stream is if we have got something we need to make sure all of the items of the stream are processed, the redis give us a bunch of premitives to work with. What does this mean?
I've been waiting for videos on deep dives. More deep dives please!! These rock
🫡
1 more thing to add;
sorted set, internally is implemented with skip lists and hashtables in Redis. Thats why adding value to sorted set is O(logN) on average and searching is O(logN + M). If you bring this up, congratulations you passed your interview
On the contrary, if I brought up this statement in an interview after having seen it on TH-cam, I just blew the interview. Any interviewer when faced with a candidate that is trying to impress by throwing out irrelevant buzzwords, will ask "oh yeah, tell me more about how to implement skip lists".
@@abhijit-sarkar couldn't agree more. Mentioning such points on your own always brings a follow up question, now I have to also read how skip lists works.
@@bhawanasahu3000 i think if you know how it works under the hood and what implications that has on time complexity that is extremely good regardless if you have further deep knowledge of those further implementations..
Can't thank you enough. You made system design so much easier to understand.
That was too awesome! I'm using redis too much recently and have some core concepts that will help at daily basis. If you post more topics of that, just announce and you'll got a viewer. Great content.
Hi i just want to say your work is truly fantastic and you are an excellent educator! I started watching your videos to prepare for system design interviews and your explanations and examples are top notch.
I'm motivated to continue watching not just to prep but also to expand my knowledge and improve my craft. Eagerly awaiting the great work you'll do next!
You are best channel for system design interview. Such a indepth technical understanding, I really appreciate you making these videos.
Echoing the existing sentiment in that this content is incredibly informative and well put together. Really appreciate your work
Glad it's been helpful! More soon.
Really cool deep dive, updated with time as well. Thanks a lot for making this. Looking forward to more!
Very informative and helpful video as always. I would appreciate if the visualization of the internals could be a bit more detailed and in sync with what you describe, rather than just a couple of boxes. Regardless, this is a refreshing take to step away from the usual system design videos to dive into technologies that are useful in both interviews and real life. Hope to see more of this in the future.
Good feedback, thank you! Will try to make the visuals more engaging in the future.
Just love the way you add practical aspects! Thankyou for the thoughtful, concise and articulate deep dive! Was definitely a high ROI on time :)
So glad it was useful!
Insanely good explanations !
Loving the consistency of uploads! :)
would love to see a deep dive on Kafka as well
Kafka is in the works! Written resource will be posted to the website this week or early next!
Great content. This is a great resource for learning before i apply Redis to my own projects. One thing I was wondering was if your content would be able to be added into Spotify so I could listen along on the go?
I really enjoy your content on the website, the videos as well as posts on reddit. Can you make another video on behavior/leadership principles interview next? I would like to learn how to tell a convincing story about my leadership potential.
Absolutely loved it !!!
Can we have one deep dive on SQL vs NoSQL, a topic always asked in system design interview.
We actually have an opinion on that very topic, check this out: www.hellointerview.com/learn/system-design/in-a-hurry/key-technologies#core-database
Wait .... I thought I clicked on a Dua Lipa music video. 😀. Just kidding. Stefan I love this - clear, concise, articulate and I learned something. Keep 'em coming.
Lol. The intro is a banger, no :)?
Couldn't focus, too handsome
The GOAT has arrived and has something important to say!
@@hello_interview Gigachad onlyfans allocator (of) toes, that's very kind of you
Hey Jordan looks like you are getting your life back.
@@coledenesik unclear at best
It's a great deep dive. I'd like to give a suggestion, Can you guys add chapters to the video, so that, if I want to rewatch again about certain portion of the video then that would be easy thing to do.
You should have chapters on this one! If you look at the description "Key Moments" will take you to the relevant sections. LMK if we're not doing something else we should be doing!
@@hello_interview I see that, you missed to add "00:00" time stamp in your "Key moments", that's causing YT to not auto chaptering your video. I have checked couple of other videos in the channel, videos which are having 00:00 in their key moments, respective chapters are added.
Great content. Really liking it. Can you please do a deep dive on TiDB, Rocks DB and Cassandra too
Cassandra is soon!
Great watch! A similar deep dive video for Kafka would be amazing 🙏
Kafka is in the works! Written resource will be posted to the website this week or early next!
@@hello_interview amazing!
Fantastic video, thank you!😊
this video is awesome! exactly what I expected!
So glad to hear it!
Excellent video. Please also make video on mock interviews
Mock interviews are what we do! What is it you want to see?
Loved it! Want to see more on how communication between service on global scale like using kafka with async or consistent connection around this please, if you get the idea, Sorry i am bad at explaining
Enjoyed both your videos on Kafka and Redis. Can you please make a video on CQRS?
Really nice! Ty!
So is Redis pub/sub a type of hashmap? Can you explain how that's implemented inside redis?
so in terms of consumer groups subscribing to topics in a pub/sub system...
can you summarize when to use Redis vs Kafka?
Can we have a deep dive on Apache Flink please ?
We'll put it on the list!
Thank you very much
At 22:55 how does hashing tweet IDs help if we're looking for the most liked tweets overall? Or were you referring to getting top liked tweets for specific keywords when splitting that key across multiple nodes?
The latter!
Hi, you talked about append a random number to solve the hotkey issue. But when you want to retrieve the data, you won't know the key because you append a random number.
How Redis maintains this cached data internally? does it maintains in some kind of tables or files?
If I want to store 10GB of some relational data tables data or query output, should we need approximately same out of RAM for REDIS setup?
Could you please provide the details how to understand the internals
Redis is an in-memory datastore, if you want to store 10GB of data in Redis, you need 10+GB of RAM, there's no paging out to disk. When operations happen they are added to an AOF (append-only file), but that's the full extent the disk is involved for garden variety Redis.
Can you please do deep dives on Kafka and Spark too ? I love your videos
Written deep dive on Kafka coming in a couple days!!
10:00 didn't quite catch how adding random suffix would solve hot node problem
Splits the hot shard into multiple, so you can distribute that load across nodes.
@@hello_interview sorry, what's the approach? I think somewhere else in the video talks about it, but I also somehow did not understand that.
Is it roughly the following?
Generate X random suffixes. Maintain a map on the (Redis') client side of .
Whenever we want to do some sort of "write", we get a random suffix from our list and append it to our key in the Redis command.
Whenever we want to do some sort of "read", we get _all_ random suffixes (X of them) from our set, make X reads from Redis, and aggregrate on the client side.
----
Or maybe it's sometimes the opposite way. For example, in a cache, I'm guessing we'd want to write our value to each and every one of the key+randomsuffix.
Then for a read, we can (randomly) pick one of the suffixes.
----
Maybe the missing link (for me and others similarly confused) is how/where the random suffixes are managed.
There's also the question of how we (automatically?) identify a hotkey, or develop an algorithm that generates (or increases) the number of suffixes depending on some sort of access rate.
@@hello_interview but how will you query the key then, as at read time you don't know what random suffix got added
@@hello_interview What if the new key conflicts with any keys added later?
@@Xiao-p9m How will it conflit, we append the certain value every single time so there wont be a conflict
What is the difference between Redis Pub/Sub and Kafka? From the other video, what I understand is Kafka is queue and event driven as well.
You can use Kafka as pub/sub! Differences in durability (Redis in memory, Kafka persists to disk), which then has consequences on other things like ease of disconnection, etc. But they can both be used.
Redis pub/sub is really dumb, at-most-once delivery of messages. You can think of it as a hashmap from a topic (string) to a socket connection to a subscribing server with all the associated problems (messages might get lost?) and benefits (incredibly lightweight, low-latency etc.)
Kafka is more full-featured. Kafka has message persistence, you can make guarantees on delivery, etc.
Thank you.
One can imagine the implementation of the geo-spacial index using 2d indexing data structures - quad tree, R-tree and others.
Thanks so much for the video. I would be really interested in deep dive on elastisearch/opensearch
That one's next!
The problem of hot key is clear but solution of hashtag is very difficult to understand. do we have to handle that all in the client? like figure out what are the hot keys?? and what will adding hashtag do really?
Solid value.
Please make a video on Elastic Search
Re-upload, sorry! Stefan is a TH-cam noob.
Facts.
🤣
I have never clicked so fast!
Amazing Content.. Can't thank you enough!! Any Plans for DSA Mock Interviews 👀
What are you looking to see?
@@hello_interview DSA Mock Interviews with FAANG Candidates..
Great video, what tool you're using for the drawing? I was looking for something simple and easy similar to this one, thanks.
Excalidraw
It would be nice to see a Deep Dive video about location database. Pretty hard to understand for me
Which part is most confusing? Geohashing? Quad trees? Where to use them?
@@hello_interview Geohashing algorithm is too complex. I just cant take in all this diagrams. I also dont understand how proximity systems can search something inside the circle using square blocks of geohashes
@@maxvettel7337 Gotcha! I'll earmark this for follow-up. The nice thing is you don't necessarily need to know about the geohashing internals to use something like Redis' proximity search in practice!
Can i book sys design coaching sessions on hello interview or is it only for mock interviews?I'm a senior engineer and I've just started preparing do you recommend to jump straight into mocks?
Send us an email at support@hellointerview.com and we might be able to make something work. Have some products planned here for people in your shoes.
@@hello_interview done
Would a sorted set implementation be appropriate for a priority queue?
For many, yes. Depends a bit on the scalability requirements.
@@hello_interview Awesome, thanks so much for the quick response. I really appreciate what you guys are doing, this is a great resource and y'all rock!
at 9.38, don't understand appedning a random number to evenly distribute request across all the nodes in redis ? how does it work ?
In a cache setup you expect that sometimes you'll have a cache miss and have a way of handling it. If we have a key `foo` that is particularly hot, we can start appending a number 1-9 `foo1`, `foo2`, `foo3`, ... Each of these keys will reside on a different node/shard in the cluster. If the key is actually hot, all the keys will be written eventually and the reads will be spread across the nodes in the cluster.
@@hello_interview if we append this to all keys then it would reduce the effective size of our cache and would have a lot of cache misses. If we do only do this for keys that are "hot", then how do we detect if that key is hot since if we use a redis cache to keep track of how many times a key has been hit, we run into the same issue since that one redis partition is getting hit hard keeping track of the amount of times that key has been accessed.
@@goodmew123I guess you would have to know beforehand. Also, there might be some heuristic for that, for example, if the number of likes on a recent tweet is high, then many people will be querying that number of likes.
Nice! Can you also do Deep Dive on Graph Databases(eg, neo4j) as well. please!
Will add it to the list! Typically, graph dbs are not as commonly used at scale.
But how will you handle the at most one delivery of redis in the case messaging?
Check out our Whatsapp guide for one example of this: www.hellointerview.com/learn/system-design/answer-keys/whatsapp
Why would I want to use Redis stream over something like Kinesis or Kafka streams
In practice? Substantially faster and cheaper if you can accept some of the (many) compromises. In an interview setting it can be helpful to have a few multifaceted tools vs having to know all about Kafka (video on that soon).
If your business was entirely built around append-only logs I probably wouldn't take Redis as an acceptable substitute, though it works in a pinch.
Kafka next, please!
This week!
For hot key you really need to have caching in the service that's hitting Redis, unless you have a ton of hot keys a small LRU cache should work. Although this trades off some consistency unless you have a real short TTL :)
- Dynamically sharding keys adds complexity to the application layer
- Dynamically scaling read replica nodes seems wasteful (you dont need to duplicate the whole node)
All solutions seem non optimal.
Yes! No free lunch.
Why not talking about cache invalidation when using as a cache?
Finding it difficult to understand stream and consumer group
Could Even cover Redis again?
Nah, this explanation is killer. Couldn’t do it better if I tried!
-Evan
@@hello_interviewPlease
@@hello_interview at least we can highlight the knowledge point . Have some summary on the diagram
14:40 the nice thing about this stream is if we have got something we need to make sure all of the items of the stream are processed, the redis give us a bunch of premitives to work with. What does this mean?
I agree with you . This is the first out of ten video I felt I learned nothing after watching in this channel
I see , I like.
this video is too high level..need more simpler deep dive on redis if possible
Which part did you miss? Maybe we can point you in the right direction.
4:24 - Did I hear a piano? Are you a fellow Pianist my friend?
You may have, but if you did it's a 5 year old banging keys not me :)
Format reminds me of acloudguru
I missed the co founder now
do a deepdive on ethereum latest version, l2 rollups
I had to struggle with the accent. Can I pls request to speak a bit slower and maybe a bit more clear accent, if possible. Thanks!
brah why do you use so much big words. lmao slow down with the vocabs damn.