Latency Numbers Programmer Should Know: Crash Course System Design #1
ฝัง
- เผยแพร่เมื่อ 3 ต.ค. 2022
- Weekly system design newsletter: bit.ly/3tfAlYD
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.
Correction: The label at 4:46 should be inter-zone networking, not intra-zone. It is the network round trip between availability zones. The closed caption and verbal explanation are correct.
Still a very good metrological read, thankyou.
Thanks a lot for the video! Also for the correction! Very video useful indeed! It puts things into perspective. As a man who writes code myself, I agree with you that the more important idea of the video is "Developing a sense of the relative orders of magnitude difference" instead of knowing exact numbers (probably in ~5 years time these numbers will change again, but the magnitude between them will pretty much stay the same unless some scientifical&technological breakthrough occurs)
Just started seeing all of the videos you have in the System Design Fundamentals. Very useful! I hope everything related do systems design from your channel is in this playlist and I am not missing any of them, as they are pure gold haha! Starting seeing you guys as sort of a second freecodecamp channel. This is great and I hope it will stay free.
Kudos to you again and keep it up! God bless!
Another interesting intuition is to put those times on a human scale by setting 1ns CPU time = 1s human time. At this conversion ratio, a clock cycle is ~0.1-0.2s, which is more or less instantaneous for humans. This could be, for example, the time to add 2 small numbers that you already have in memory. If you need to read them off of a page in front of you (L1/L2 cache), that could take a few seconds. Going further, a system call takes a few minutes, context switching is an hour or two, HTTP request takes several hours, you need to wait a day to read something from SSD, and when writing it's gonna be in the mail for around a week. Seek time of HDD is like 2 months, if you need data over the network from another continent that's a few years, and a TLS handshake is like a decade. And God forbid you need to transfer 1GB over network in same cloud region - that's like 3 centuries.
Can I somehow spotlight this comment? This is exactly the kind of perspective most people need to get a grasp of these numbers. I usually do a simmilar thing when teaching kids about the age of the universe and how long people have been on our planet.
The simile I've heard for data access puts you in the shoes of a craftsman on a work site.
Register is the tools in your hands, Cache your toolbelt, RAM your toolbox, SSD your car, HDD back at the warehouse and so on.
Awesome analogy, thanks!
alternately, i was thinking it could be helpful to realize the screen refresh rate as the upper limit of that time scale... 60 Hz makes the screen visible every 6 ms (doubting my math now😬)
Shows just how incredibly fast computers have made humanity
in the UI if something takes longer than 100ms it becomes noticeable by the user. like if you click a button and it takes more than 100ms to react to the click the action doesn't feel instantaneous anymore
3+ sec page load times are problematic too
@@raianmr2843 WDYM? My users accept anything as long as it is bellow one minute
@@raianmr2843 might aswell be offline then
Indeed the brain is pretty tolerant at simply correlating the signals together bellow that threshold. It has to be as it has to deal with both the physical lag of the input itself and the fact that your senses each have their own neural processing latency that varies depending on which sense is being activated ~20-40ms for Visual or ~8-10ms for auditory for example. Of course this does all assume you are not dealing with a user with sensory processing disorder as one of the symptoms is breaking this ability to correlate sensory input from different sources.
As a programmer when I click on one of my buttons, I know it is time to get a new coffee because it will take a while.
Your content is awesome. The amount of knowledge per time is fantastic. Only facts no bs. And the visualizations are great. Thank you
truly amazing
Have you consider to record videotutorial (or maybe whole course) about how to create such amazing visual effects for presentation? Your team very good at it, and those knowledges would be priceless
that doesn't make sense for him
Communication, pitching is super important task of architect, so I concur about the idea for the visual effect empowered presentation sharing clips
yes I was also wandering how do you make such a wonderful videos
GIMP animations, with face overlay and pan & zoom effects from Shotcut? You might learn more about video production from video production channels.
> Latency Numbers Programmer Should Know
Average Python guy who use openCV for croping images: Thanks, it's so helpful!
I literally love "working with you". Your videos and newsletters (and books) make my data science programming so much more effective and significantly easier to deploy to production. Thank You - you've taken my career to an entire new level.
This is a sheer jewel. Although I knew most of this by experience and interest, having a summary like this is incredibly eye opening. This should be standard curriculum in the first year of computer science education.
This channel deserves to be a lot more popular. It is head and shoulders above almost all other IT channels I have encountered on TH-cam.
I love the way each of the videos is short and rich with useful info that helps to expand your software engineering knowhow 👌
The many round trips of the TLS handshake is something that is actively being improved upon.
TLS 1.3 currently supports 1-RTT handshakes and initial support for 0-RTT handshakes.. which would basically remove the delay of the TLS handshake all together as it can be performed simultaneously to an actual content request call.
This is one of the best channels ever.
I share your videos with all of my developer friends. Great for a refresher, summary, or introduction to these concepts. Keep up the excellent work!
I am still astonished that content of this quality can be *free. It gives me a lot of hope for the world. Thanks you much and keep it the great work
Your content is highest quality! And very insightful!
Fascinating to put this into perspective! Specially access speed differences between registers, cache and memory. Also context switching for threads
You're doing good work. I think this video will be helpful for anyone in computer science whether a student, professional or someone who is just curious. Also the visualisations are great. Thanks.
The knowledge density of content on this channel is through the roof. The books are great too. Impatiently waiting for more videos! Keep it up!
indeed! stuffs like branch mispredict penalty, MD5 Hash etc. are complete brand new concepts for a CS bachelor like me, gonna take at least half an hour to digest all details he mentioned in a single video. This channel is like the low-level design version of Fireship while both channel rock the same dense and style loll
Awesome video. This was important to get the time intuition of different processes.
Another great video.
Have you created content that shows the impact of this latency on system designs? Examples:
* Accessing memory by reference instead of cloning the content.
* Single threaded, vs multi-threaded applications
* Async-await vs multi-threaded applications
* Impact of microservices and containerization
* Impact of serverless
I often feel that latency and the cost of communication is forgotten in the scale-out discussion. I have personally seen examples where a containerized microservice architecture was 40-200 times slower than a non-containerized approach. Have you seen the same?
Amazing video! Loved the visuals and the explanation!
I learned these things in the 1990s, so nice to have an update. Wish you had mentioned cost of an ordinary subroutine call.
Awsome video, both explanations and animation are fabulous! Thanks a lot!
Thank you! I have bought both your books!
This video really helped me. Thanks a lot!
Thank you. Very important and helpfull context
Useful information. A bit of connected data: 10 milliseconds of difference between two notes sounds fine, and imparts a certain "feel", but 30ms starts to sound "off", like one is early or late.
Great content as usual, and amazing presentation to explain the same.
Loved it 👍
This channel is incredibly handy
I've been lucky enough to enjoy two simultaneous careers in the 35 years I've been working professionally, one in enterprise database design & troubleshooting, and the other I'm an RF mixed signal electronics engineer: I go all the way down to pico & sometimes 100s of femto seconds for the electronics side.
One of the most useful facets of having experience in two apparently largely mutually exclusive skillsets is when it comes to performance & troubleshooting, whether it's knowing what looks about right, or knowing what's reasonably achievable. In my database work, I usually talk in terms of orders of magnitude improvements, but in my electronics work it's more often just ~10% or 20% improvements, as we're already hitting the state of the art.
The same dependencies on both experience and underlying understanding are required. In terms of analytical skills, it's pretty much identical: you split a given challenge into manageable chunks, identify the area(s) where a problem lies, re-engineer accordingly., and put it back together.
The two are similar in so many ways, even down to Heisenberg uncertainty, such as sticking in instrumentation into code or probing a high speed interface: both can substantially interfere with the result.
"...such as sticking in instrumentation into code or probing a high speed interface: both can substantially interfere with the result..." - Guilty!
Your videos are amazing. Thanks!
Excellent video. I definitely have intuitively known some of these numbers for a long time being just a super nerdy programmer with a gamedev hobby. However, I came up a while ago not knowing a lot of the cloud numbers, and L3 cache didn't exist in CPUs and maybe not even L2 in the earlier days. No surprises really exist other than SSDs being a lot faster than HDDs. I've known intuitively that network transfer rates (bandwidth, not latency), but now even latency of network access is faster than an HDD. In otherwise, a locally connected HDD is _slower_ than an intrazone networking transfer if the server is reading from main memory or significantly faster IO storage.
Simply thank you very much for your videos!
Fascinating, thanks. It had not occurred to me before that a (relatively) slow hash algorithm has the advantage regarding resistance to brute force trials. I'd love to see a comparison of server response latencies; I'm guessing a liteSpeed server using ssd storage located a continent away would respond faster than a local apache server with a moving disk.
It always amazes me how few design based on these factors that can affect performance at such huge levels.
Very nice 👍 Helpful to have this kind of intuition while debugging performance issues in complex distributed systems
Wow greate explanations, thank you
This is great - going straight to my bookmarks, thankyou :)
Also important 10-100ms the time between two image in animated image. For a game you should typically aim at getting everything done in 16ms to hit 60fps. Or finish all calculations in the 1-10ms range to be able to scale to higher frequency displays.
~100ms is also the maximum amount audio and video can be out of sync before people start noticing it. Already at ~200-300ms it becomes difficult to hold an audio conversion, as people start missing queues about when it is their turn to talk.
Thank you for this very interesting video!
This is awesome content !!! Thanks.
Glad you had enough time available to make this video.
Great content and presentation. Keep up the good work!
Awesome one - Thanks a lot
Very nice video! Thanks!
Greate video, thanks for making it! I think this knowledge is useful when reasoning about software, for instance if you have a simple task that takes a second, you might have made a mistake somewhere, like a n+1 query. If you don't have this intuition you might think: Oh, it's just a second. But a second is literally an eternity for a computer. If your operation takes a second, you better know why!
amazing videos It must took you a lot of effort to combine the resources and put it in a simple way , for viewers they need to take notes and then understand each things mentioned in the video :)
Would have loved to see a similar video with the comparison to how tech has evolved , anyway great video 🎉
simple yet elegant.
Just amazing stuff. Excellent!!!
Thank you! It is really helpful)
Nice videos ...thank you.
thanks for the info good representing and good editing
Great summary of meaningful latencies. I think it would have been nice to include human reaction to in 100-200 ms range. This is the minimum time humans need to actually react any new information, not matter how little semantics that information contains. An example would be ability to press a mouse button after something appears on the screen.
Another good latency to include would have been 10-20 ms for the minimum time to get something on the screen on most computers.
1 ns = 10^-9 s
1 us = 10^-6 s
1 ms = 10^-3s
Register < L1/L2/L3 caches < memory < SSD < Harddisk < intranet < internet < TLS
Thank you for sharing!
main memory
10-100us
- Network proxy HTTP request routing
- Read 1 MB sequentially from main memory
- Read 8 KB page from SSD
100-1000us
- SSD write
- Intra-zone networking round-trip
1-10ms
- Memcache/Redis get round trip (~1 ms)
- Inter-zone networking round-trip (~5 ms)
- HDD seek time (~5 ms)
10-100ms
- US east to west coast or Europe network round-trip
- Reading 1 GB from main memory
100-1000ms
- bcrypt a password (~300 ms)
- TLS handshake (~250-500 ms)
- US west coast to Singapore network round-trip
- 1 GB sequential read from SSD
>1s
- 1 GB network transfer in same cloud region (~10 s)
Brilliant Many Thanks
I would have loved to see gpu read and write latencies on here but I really loved this video
Another good important point PCI-E devices roundtrip is about 500ns
Great video! I'm curious about how you got the estimations for these operations.
Human visual reaction time is about 200ms. An interface response faster than this has diminishing returns.
What a very fascinating thing to talk about.
trap -> kernel space -> trap -> user space, not accounting for processing time of the system call itself)
- MD5 hash of 64-bit number
1-10us
- context switch between Linux threads (does not account for bringing large # of pages for the new thread, which would extend time further)
- copy 64 KB memory to new location
10-100us
- typical amount of time for network proxy to process a http request (nginx)
- read 1 MB of sequential main memory data
- read 8K page of SSD
100-1000us
- SSD write latency for 1 page
- intra-zone networking latency [100us]
- memcache/redis get operation as measured by client [1ms]
1-10ms
- inter-availability zone of modern cloud provider networking latency [5ms]
- hard disk seek time [5ms]
10-100ms
- us-west coast to us-east coast latency, us-east coast to europe latency
- read 1 GB of sequential main memory
100-1000ms
- bcrypt a password [300ms]
- TLS handshake [250 - 500ms, depends on distance between machines]
- us-west coast to singapore latency
- read 1 GB of sequential SSD data
1s
- transfer 1 GB over cloud network within same region [10s]
100 to 1000ms: User perception of a "fast response" on the internet is around this range. (Google says 200ms.) That means if you want your page to feel responsive, a response better be delivered and rendered to the user before 200ms have passed after a mouse click or screen tap.
1s to 10s: Range of spindown time of spinning hard drive.
10s to 100s: TikTok and other short format videos are in this range. User attention becomes this 15s limit, meaning users get bored super quickly. (Especially young users.) Also, interestingly, if a system does not respond in this amount of time, users might think your program has locked up and try force closing/rebooting/etc. Along the same lines, most UNIX and Linux systems send SIGTERM to all process, followed by 5 seconds wait, followed by SIGKILL. (Your program has 5 seconds to clean things up before it's forcefully killed.)
100s to 1000s: TCP maximum retransmission time is 120 seconds by default. If DRAM is cooled, the contents can be retrieved in this amount of time after power is removed.
As always great video
Sound like a full course for me!
It is worth mentioning 100ms border of a human perception. Any operations faster then this we treat as "instant". Anything longer - as having some delays.
These videos are cool! I miss the videos where you explain more basic concepts such as TLS or other protocols. I found those the most interesting
More to come! Do you have any topics in mind?
@@ByteByteGo mqtt would be quite interesting
@@ByteByteGo a video about cache hierarchy (L1 to L7) might be really cool :)
@@ByteByteGo RabbitMQ, ampq, celery.
@@ByteByteGo Push vs. Pull model for getting events
Coming from a game dev background, I'm used to thinking of latency in clock cycles, and anything more than a few hundred is "too long".
your vids are so perfect
Great video!
How minute your discussions have been, Sir
Great video, thanks a lot!
I love the last one, takes 10 sec to transfer 1GB over network within same region .. then some webpages take several seconds to load a little bit of text and a few pictures.. I mean there are of course many things at play here, it's not only the transfer, which is considered here, but still we have a lot to improve on in general when it comes to web. And I don't only mean the programming I mean choice of server OS and so on. Some of the extra latency can be due to hundreds and thousands of processes being run concurrently on the same servers . So if the kernel interrupts every n mikroseconds, it takes the amount of nano/microseconds for every process to go from userland to kernel land which adds up until the scheduler gets back to the process again.. And storing and loading the process state in it self also requires cyckles aka time to do every time. Then there are more stuff going on as well.. (I didn't remember the numbers presented in the video but there all there in the video :) )
We need less bloaty things even though hardware gets better. Sometimes today surfing the web is almost the same experience as the early 2000-s..
Most of that is taking bids to sell ads by now lol
@@OatmealTheCrazy lol yeah! There's also that!
I really love your content
I liked the video. See you in 10 years for the new version !
100ms -- the time at which a human being regards a response from a computer as "instantaneous" -- for example, if the user presses down on a key "X", and then an "X" appears on the screen within 100 milliseconds, then the human being judges that the computer responded "instantaneously" -- VERY IMPORTANT for programmers to know!
I would add that at 10-100ms is the range which many consider to be "real time", you should have latency under 15-30ms if you want to attain 60-30fps due to some complex command like computer vision.
thank you 🙏 ❤
I'd be interested to see thread switching and kernel call latency comparisons between Linux and Windows.
good point 🤔. but I think Linux is some 10% - 30% faster
Excellent, can be use for some software development. You have PDF version of this video, like cheat sheet ?
Btw, if you don't mind, request video for "Acceptance latency number for user/client" so any software development including game, desktop, and of course web can have general idea how long they can ask user to wait. Thank you.
You really picked up publishing pace ☺️👍🏻
Same pace since the beginning in June. Once a week on Tuesday.
@@ByteByteGo It has become a highlight of my early week! Thank you! 👍🏼😎✌🏼
Great video!, how do you create those animations for your videos?
great video
A good sister video would be to review these numbers for an embedded system, maybe an ARM microcontroller.
Interesting!!!
good video！
Some of the ratios between some obf the numbers have been surprisingly constant over many years such as memory access latency expressed in CPU clock cycles.. Sure, the exact values are changing but these exact values rarely matter.
You have to put into perspective the distance light travels in these times. In 1 ns light only travels ~30cm. With a speed of electricity of ~2/3 c that means it is physically impossible to access information further than 10cm from the CPU in 1ns. That means anything the CPU has to do in 1 clock cycle of let's say a 5 GHz CPU has to be physically within 2cm (a bit less than an inch for the americans). More realistically, it has to be within 1cm as any logic also takes time and electrical connections are rarely point to point.
And this might soon or already now need a 100ps-1000ps range as clock speeds are getting faster again.
Great❤️
The beauty of computer science in one video
Awesome.
Subscribed 👍
Hey, nice plaque!
Hey, thanks! 🤣
this is why for low latency software you do not use any databases... your system is a database except its better, because unlike a general purpose database your system is focused to do specific things
I would find it cool to add perceived latency on the ui. there are papers on this, like when does a ui feel "laggy". For example a button has some highlight effekt on mouse over and if the effekt takes too long to start it feels laggy. I have in the back of my head the mark of 50ms for those things, but could be wrong.
It's typically closer to 100ms in most people, bear in mind there is around a 20-30ms difference in neural processing delay between visual and auditory signals so a normal brain needs to be able to integrate at this level. Touch signals can be even slower you are looking at closer to 40ms (Which is at the slow end of visual processing time) for a signal from the toes to reach the brain before processing can even begin for an adult (~2m @ 50-60 m/s). If it wasn't able to do it over windows at least in the 50ms range or so then coordinating anything involving multiple senses would be significantly hampered eg using vision and proprioception to manipulate objects with precision etc.
The illustrative presentation makes it a whole lot easier to understand 🔥
I have question on the 100-1000ms latency level. You said "it is slow enough to render brute force password cracking ineffective." Does that mean the latency for brute force password cracking is slower?
That means that a single thread can only hash around 5 passwords per second, even if you use something like an EPYC CPU with 96 cores you'll only be able to test about 1000 passwords per second. And that is ungodly slow, taking multiple years to crack even simple passwords using a rainbow table.
Hash algorithms that are disgned for consistency and equality checks like SHA are much faster. SHA-256 can hash 2GB per second per core. And can be implemented for GPUs.
You can test orders of magnitude more passwords in the same time
We should expect SSD latency to reach that of ramdisk accesses that don't fit into CPU cache during the 2020s as F-RAM and FeFET memory become large and cheap.
Minor nitpick: bcrypt has a variable workload parameter, which allows one to freely configure how long it will take to calculate a hash for a given amount of data on a given hardware. Thus, saying a slow hash-function takes between 100 and 1000 ms isn't really correct. It can take however long it is configured to take.
Good point. Appreciate the clarification.