Node JS - Writable Streams & Backpressure

In this video we will be talking about Writable streams & Backpressure, which are used basically everywhere just like Readable Streams.
If you wish to support me, here is my patreon page:
www.patreon.co...
Github: github.com/web...

[@AkimboFennec]

For people who might not have understood the high watermark. What happens under the hood in a a computer is the following thing: you have a RAM, which is where all of the variables of a program live. The files, since they are persisted, are stored in a Disk, which is physically separate from RAM memory. Now, when you want all of the bits from one file to another file, you are essentially copying the bits from one area on disk to another area on disk. But computer hardware does not allow you to copy data directly from disk location to another disk location. Instead what you do is firstly get the bits from the source file into the RAM memory, and then these bits are again sent back to the disk, but this time to a destination location on the disk. Now, since RAM is limited amount of memory, if you were to try and copy the files directly, without using Streams, you run into memory issues. For example, let’s say that you only have 1 Gigabyte of RAM, which is about 1 billion memory cells number from 0 and so on. And let’s say that you are trying to copy a file of size 0.6 gigabytes. If you were to NOT use the streams, you would need to copy 0.6 Gigabytes of bits from disk into RAM, thus occupying 60 percent of RAM space, leaving other programs running on your computer with much less ram memory. This will be detrimental to performance of your computer since you will force other programs to store their variables on disk, since RAM does not have enough memory, and Disks are a lot slower than RAM. Now with Write and Read Stream, what you do is simply allocate a certain number of bytes on the RAM. Let’s say you decided to allocated 1 megabyte of space for this stream, which is 1 million memory cells. This means that you allocated a fixed amount of space on the ram, and it will not change. Now the way copying files works with streams is like this. You begin moving bits from the source location on the disk to this 1 megabytes sized space on RAM. Whenever 1 million bits have been sent to RAM, the stream is now full, and can begin writing these 1 million bits back to the disk, but instead of writing them to the source location on disk, it writes them to some other destination location on disk, which is a destination file. High water mark just specifies how many bytes of storage do you want to reserve on RAM. Key point: in order to copy all the data from one file to another, you copy the data from Disk to Ram, and then from RAM back to Disk, since computers can’t copy data from disk to disk without going through RAM. High water specifies the size, in bytes, of this temporary RAM storage that is used to harbor these bits before they are being sent to the destination file. Hope this comment helps clarify some stuff.

[@syedhaider0916]

Thank you brother, feel bad that you don't have atleast 50k subscribers based on the quality stuff and the way of teaching.

[@steveansahababio8917]

The best explanation of backpressure ever. Thanks man

[@vibhugautam5093]

Wow man this is something really good one of the unique concepts I learned in JS. Loved your work!!

[@ayikkathilkarthik4312]

WOW what an awesome explanation .
Thanks 🙏

[@iHariPatel]

simple and understandble, Thank You

[@starterdev]

Merry Christmas

[@WebDevJourney]

merry christmas

[@mohamedmegg2796]

super!! this video worth more likes than you can imagine, man ur the best ❤

[@viktorpoliushko2015]

Great explanation of write streams!

[@mykun8737]

thanks so much! hoping you release more videos about what behind in node js, it can be worker thread

[@hagiwararyunosuke3371]

happy new year and thanks for the stream serie !

[@sylvaingagnon6263]

Really good explanation! Thanks!

[@agagoua]

I d suggest, instead of adding the `highWaterMark` property on the `writable stream`, to put it in the `readable stream`.
So, we can control the chunk size that we read and easily find the point where the `writable stream` does not `backpressure`.
The smaller `highWaterMark` e.g. {highWaterMark: 500}, the more time it takes to download the file and the less memory it consumes. Just need to play with that property, so to find the right size. :)

[@anishsharma2856]

can you please share how can we download excel file in browser using with exceljs in stream?

[@hoshankumaran]

While it's true that adjusting the `highWaterMark` property on the `readable stream` can help control the chunk size and potentially manage backpressure, it's important to note that the `highWaterMark` property is actually associated with the `writable stream` rather than the `readable stream`.
The `highWaterMark` property determines the maximum amount of data that can be buffered in a writable stream before it starts applying backpressure to the readable stream. When the amount of buffered data exceeds the `highWaterMark`, the writable stream signals the readable stream to pause, indicating that it should slow down or stop producing more data until the writable stream catches up.
By adjusting the `highWaterMark` property on the writable stream, you can control the point at which backpressure is applied. A smaller `highWaterMark` value would result in more frequent backpressure signals, causing the readable stream to slow down and reducing the amount of data buffered. This can be useful when you want to balance memory consumption and download time.
However, directly adjusting the `highWaterMark` property on the readable stream would not have the desired effect of controlling the chunk size and backpressure. It's the writable stream that dictates when backpressure should be applied, and the readable stream responds accordingly.
So, to control the chunk size and manage backpressure, you should focus on adjusting the `highWaterMark` property on the writable stream rather than the readable stream. Experimenting with different `highWaterMark` values can help find the right balance between memory consumption, download time, and efficient data transfer.

[@sirkartik]

@@hoshankumaran This is generated using ChatGPT, isn't it?

[@RamKumar-es3mp]

Great content please put rxjs series

[@trishulcurtis1810]

Great video!

[@uaudith]

what happens if the read stream does not respect the back pressure

[@AkimboFennec]

If the read stream does not respect, what will happen is that your RAM memory usage will get very high, which will impact the performance of your application, and computer. If you want I could explain it with a much greater detail.

[@kapilbhise8273]

Can you please tell which of your series is a MERN stack series?

[@WebDevJourney]

This is a series where we did (somewhat) a twitter clone using the MERN stack

[@flamess007]

Bro...thank you so much

[@WebDevJourney]

again, you are very welcome

[@flamess007]

@@WebDevJourney I just subscribed good content! Btw I have a node certification in 3 weeks :S

[@WebDevJourney]

@@flamess007 hey man thanks a ton, also congrats on getting a certification, didn't know you can get one, might need to look into that.

[@flamess007]

@@WebDevJourney send me a message plssss

[@WebDevJourney]

@@flamess007 again here is your message.

[@mykun8737]

i don't really understand the difference between data stream flowing into buffer memory and data stream flowing through stream pipeline, hoping you can explain it to me

[@WebDevJourney]

this is part of a series, hopefully the first video will explain what you are having trouble with.
th-cam.com/play/PLrwNNiB6YOA18XANsFe0CnizlhYKjJT0f.html

[@abelmarkos5753]

greet video
could you tell us what's your plan for this amaizing chanel in 2021

[@WebDevJourney]

will be my next video after this series

[@SyranoProductiones]

std out ...nice