NOTES & CORRECTIONS: • *Excalidraw file:* excalidraw.com/#json=z6OfXNuA1htMbSEfXeixq,8-DMBtQczaZfr2m-PpCNmQ • *Screenshots of the excalidraw file:* imgur.com/a/C37N2hh • *Text files:* • *overview.md* - pastebin.com/HgmV8dRi • *indexing.md* - pastebin.com/KtuELqhJ • the excalidraw link and screenshots above contain the corrected version of the drawing for the "write heavy" scenario
thanks a lot! found ur channel a month ago or so and trying to watch at least one video every day (from the first video and now im on tinder), great content!! thanks for ur work
Your videos are so good that I selfishly don't want the world to know in order to have less competition during the interviews 🤪 . Thanks for these wonderful videos. Please keep the good work going.
You don't need linearizability for rate limiting. There are CRDT counters. Also, a seriazable transaction wrapping around a read-counter and increment-counter works.
FYI, I don’t intend to ever paywall my content. Even if I convert all of this stuff into a book (as I’ve said on a few occasions now is definitely a long-term interest with all this stuff), I’d prefer to do something like mmds.org and have something freely available online as well. I’ve already got a repo up where I occasionally dump all my files (github.com/systemdesignfightclub/SDFC), and I’m interested in giving it an open source license - I’m just split between the MIT license or doing something like what ElasticSearch did
I made a couple others already (including a really cleaned up 100 second long video on consistent hashing that I referenced in this video). They typically don't do as well as my problem coverage, and I do prefer to cover content that isn't already out there -- for example, I have the only coverage on the entire internet for at least 6 problems, and possibly the only thorough coverage on at least another 12. On the other hand, I feel that system design concepts has tons of phenomenal and comprehensive content for all skill levels, where it might be a waste of time to try to duplicate the efforts of those authors. So I prefer to just point people to the best resources for concepts (and I really do have a little more than 20 physical books on basically anything even tangentially related to system design): github.com/systemdesignfightclub/SDFC#roadmap--useful-resources
I really, really don’t want to ever “paywall” any of my content. I know a few people have asked for a donation button. I think my current best idea for any further monetization might be some kind of “merch” but I’m honestly more worried about complicating my tax situation than scoring an extra couple grand per year 😅
I think most companies actually use either medium or Ghost. I think these two tools internally do something similar to jekyll and other similar static site generators, where they take a markdown file and convert it to HTML... and then, you can just stick that HTML, CSS, and JS up on either a CDN or do something similar to what I did in the craigslist video with serving static files directly off of nginx. In fact, if you wanted to really just "roll it from scratch", it'd still look very similar to what I did in my coverage of craigslist -- they'd most likely still be doing something that just entails a very simple CRUD server for creating, editing, previewing the blog posts, and then "compile it" into HTML just like other static site generators and send it off to something like S3, nginx, or a CDN for hosting 😊 Here's a link to the craigslist video if you're interested in that content: th-cam.com/video/ZLpMtEXizOw/w-d-xo.html
Thanks for the great video! Can you explain further why a global secondary index shouldn't be used in favor of the read-optimized views that you mentioned? I would imagine that maintaining the read-optimized views is essentially writing to a new version of the table, which is what a GSI does, except one would have to manage themselves perhaps (at least with DynamoDB I believe).
Wow. You’re right. That’s fascinating! I didn’t cover graph databases themselves because (to my understanding) they don’t actually use a different type of index that’s more optimal for graph data, they purely have a different SQL syntax that makes it easier to work with graphs - however, they seem much more well suited to OLAP use cases from stories of hands-on experience that I’ve heard, which makes what you’ve said all the more interesting. Vector indices seem to traditionally become more necessary for OLTP scenarios. So, seeing a vector index on something more useful for OLAP, particularly *graph* databases which are already so incredibly niche, just begs the question (imo) of who they think is going to be using that feature
I’m definitely being hard on redis as a product. I just don’t know why they didn’t fork it and spin pub/sub or most of those other features off into separate products. It’s a risk to the core caching functionality (too many distractions typically results in a loss of focus where the core features “rot” and become less competitive) Plus, it gives silver bullet vibes, which is just not what I think most experienced engineer like to see in an infrastructure component. It’s probably considered a bad smell
@@SDFC Nevertheless it is an amazing video, I shared with all my friends. Thanks for putting so much effort in bringing all this knowledge together and making it easier to digest too :)
Yes, that was a clumsy mistake 😥 I updated that in the files that were posted to the pinned comment and video description. Thanks for picking up on that! I think I'm at least happy that it was something a bit more obvious and non-essential than something like the key-range query drawings 😅
NOTES & CORRECTIONS:
• *Excalidraw file:* excalidraw.com/#json=z6OfXNuA1htMbSEfXeixq,8-DMBtQczaZfr2m-PpCNmQ
• *Screenshots of the excalidraw file:* imgur.com/a/C37N2hh
• *Text files:*
• *overview.md* - pastebin.com/HgmV8dRi
• *indexing.md* - pastebin.com/KtuELqhJ
• the excalidraw link and screenshots above contain the corrected version of the drawing for the "write heavy" scenario
I think the "Absolutely Everything That I Know About X" can make a really good series, would love to see you continue this!
thanks a lot! found ur channel a month ago or so and trying to watch at least one video every day (from the first video and now im on tinder), great content!! thanks for ur work
Your videos are so good that I selfishly don't want the world to know in order to have less competition during the interviews 🤪 . Thanks for these wonderful videos. Please keep the good work going.
You're a legend
You don't need linearizability for rate limiting. There are CRDT counters. Also, a seriazable transaction wrapping around a read-counter and increment-counter works.
Master class Alert 💙
FYI, I don’t intend to ever paywall my content.
Even if I convert all of this stuff into a book (as I’ve said on a few occasions now is definitely a long-term interest with all this stuff), I’d prefer to do something like mmds.org and have something freely available online as well.
I’ve already got a repo up where I occasionally dump all my files (github.com/systemdesignfightclub/SDFC), and I’m interested in giving it an open source license - I’m just split between the MIT license or doing something like what ElasticSearch did
@@SDFC I wouldn't mind seeing a donation button in there somewhere, your passion for teaching is admirable, you're doing the lord's work.
amazing 🎉
Hey, will you make videos about system design fundamental concepts?
I made a couple others already (including a really cleaned up 100 second long video on consistent hashing that I referenced in this video). They typically don't do as well as my problem coverage, and I do prefer to cover content that isn't already out there -- for example, I have the only coverage on the entire internet for at least 6 problems, and possibly the only thorough coverage on at least another 12.
On the other hand, I feel that system design concepts has tons of phenomenal and comprehensive content for all skill levels, where it might be a waste of time to try to duplicate the efforts of those authors.
So I prefer to just point people to the best resources for concepts (and I really do have a little more than 20 physical books on basically anything even tangentially related to system design):
github.com/systemdesignfightclub/SDFC#roadmap--useful-resources
honestly really love these videos i'd pay for this. no one really goes into detail into this i feel.@@SDFC
I really, really don’t want to ever “paywall” any of my content.
I know a few people have asked for a donation button.
I think my current best idea for any further monetization might be some kind of “merch” but I’m honestly more worried about complicating my tax situation than scoring an extra couple grand per year 😅
Thanks for the video!
I would love a video on Content Management Systems (CMSs) like for example Wordpress or Drupal for companies with big websites.
I think most companies actually use either medium or Ghost.
I think these two tools internally do something similar to jekyll and other similar static site generators, where they take a markdown file and convert it to HTML...
and then, you can just stick that HTML, CSS, and JS up on either a CDN or do something similar to what I did in the craigslist video with serving static files directly off of nginx.
In fact, if you wanted to really just "roll it from scratch", it'd still look very similar to what I did in my coverage of craigslist -- they'd most likely still be doing something that just entails a very simple CRUD server for creating, editing, previewing the blog posts, and then "compile it" into HTML just like other static site generators and send it off to something like S3, nginx, or a CDN for hosting 😊
Here's a link to the craigslist video if you're interested in that content:
th-cam.com/video/ZLpMtEXizOw/w-d-xo.html
@@SDFC thanks for the fast response. Will definitely look into it :)
Great content!
more videos about database pls ♥
Thanks for the great video! Can you explain further why a global secondary index shouldn't be used in favor of the read-optimized views that you mentioned? I would imagine that maintaining the read-optimized views is essentially writing to a new version of the table, which is what a GSI does, except one would have to manage themselves perhaps (at least with DynamoDB I believe).
Broooooooo share those valuable files and work
Done! 😁 you can find them in either the pinned comment or the video description for the links to the excalidraw file, screenshots, and text files
Neo 4j is also a vector db
Wow. You’re right. That’s fascinating!
I didn’t cover graph databases themselves because (to my understanding) they don’t actually use a different type of index that’s more optimal for graph data, they purely have a different SQL syntax that makes it easier to work with graphs - however, they seem much more well suited to OLAP use cases from stories of hands-on experience that I’ve heard, which makes what you’ve said all the more interesting.
Vector indices seem to traditionally become more necessary for OLTP scenarios. So, seeing a vector index on something more useful for OLAP, particularly *graph* databases which are already so incredibly niche, just begs the question (imo) of who they think is going to be using that feature
Poor guy redis getting roasted so much. I think redis has insecurity and wants the parents to acknowledge him, so he does everything to impress them.
I’m definitely being hard on redis as a product.
I just don’t know why they didn’t fork it and spin pub/sub or most of those other features off into separate products.
It’s a risk to the core caching functionality (too many distractions typically results in a loss of focus where the core features “rot” and become less competitive)
Plus, it gives silver bullet vibes, which is just not what I think most experienced engineer like to see in an infrastructure component. It’s probably considered a bad smell
@@SDFC Nevertheless it is an amazing video, I shared with all my friends. Thanks for putting so much effort in bringing all this knowledge together and making it easier to digest too :)
do you recommend this book for beginners?
Web Scalability for Startup Engineers by Artur Ejsmont
Thanks ..
Baby come back
read heavy- write heavy diagram looks the same?
Yes, that was a clumsy mistake 😥 I updated that in the files that were posted to the pinned comment and video description. Thanks for picking up on that!
I think I'm at least happy that it was something a bit more obvious and non-essential than something like the key-range query drawings 😅
No discord?
discord.gg/system-design-fight-club-993670645718204479
REDIS