When I first got into linux all these words "tar", "gzip", "tarball" and their commands were a mystery which took me way longer than 100s to figure out. This video is gold for getting started. Putting all this in perspective in less than 2 minutes is just awesome.
I also like how they are not afraid to get "funny" with it. You have "more" and then "less" ("less is more"), you have weird names like 🔫zip, and tarball, and to get help with something you call the "man"
It's also ridiculously hard to zip/tar things with CLI compared to a GUI. Archiving and compressing being separate commands, weird flags you have to learn and footguns like accidentally zipping every file in a folder individually (happened to me several times).
One interesting difference with zip is that in zip each file is compressed and then archived, whereas in tar.gz you archive all the files and then compress the whole thing. Advantage: Cross-file compression (i.e. repeated patterns across files are used for compression) Disadvantage: You can not extract a single file and leave the rest alone.
That's what's nice in RAR, it has a flag for that. If you want to compress a huge library of pictures or videos for example, you might want to have them independently compressed. But a large database or text-like files or things like a complete game / program you might want to compress the whole archive.
The main use for archives that I see is not for saving my own data, but for distributing collections of files. And normally when you get one of those you want to extract everything.
The actual compression algorithm (deflate) is the same between zip and gzip. Zip adds the archiving around it, while gzip just adds a small header and a checksum, so it's lighter.
You're all lucky newer implementations of Tar stripped all the tape-related stuff. Reading through the man page you'd spend an hour to find how to create, or add to an archive, while in the process learning how to rewind the tape, seek the next archive on tape, activate a jukebox-like tape swapper mechanism, wipe a tape, check whether there's enough free space after the last archive to fit yours, switch to a different track on a multi-track tape, set tape drive rpm, and a myriad of others.
@@stickyfingies what "in the days"? Tape drives are used a lot right now in datacenters, like google and amazon, they are great for long term data storage
This man is so considerate, he even included the -v flag (which for starters stands for verbose) just so you can see your files being compressed, truly a born teacher. Here's an explanation of the flags used by him: -c = stands for create, it's the one you'd use the most. -z = tells tar to compress using gzip. There's many other ones you can choose from, which imo gzip being the best. -v = verbose, tells tar to display output in the terminal. -f = stands for filename, usually a directory. You can read most, if not all options using the --help command, with two hyphens. Even if you use windows, using the terminal makes your life way faster if you get used to it. I highly recommend giving a shot to the lifestyle.
@@Gesepp95 Most of the stuff you can use in linux is included in windows as aliases. They should have similar, if not the same functions that what you're used to has.
Oh yeah, compression really played a hand when I implemented a redis caching in the backend. It significantly reduced memory usage and network traffic to cloud redis. And gzip is actually really fast too (we save string to redis as key val).
Nah, you can save different data types as val but not a file if I remember right. What I did is JSON stringified data and compress the string after. Issue with this approach is that Json parse is not enough to convert values like functions, but that's okay cos you shouldn't be caching functions inside redis anyway. Oh and Json parse is crazy fast to in es5
I had to write my own implementation of the Huffman encoding algorithm for a computer science class back in the day. Compression and data encoding is a really interesting subject! It's an interesting scientific problem to figure out how to represent lossless information in the last amount of bits possible.
While an undeniably good developer, Jeff really shines with his ability to synthesize and condense down information. These videos are snappy, engaging, and eloquent. Thank you a million, Jeff.
tarballs sounded so intimidating to me and i thought it was something complicated lmfaoo i always wondered. its so freaking simple. i just made gzipped and gunzipped a couple files and then made a tarball out of them. shrank 250kb to 80kb. Cool stuff lol Just goes to show you how you can get in your own way sometimes. And also how good Fireships videos are.
I usually use zstd (Zstandard) for files when the portability of gzip or zip is not required. Also lz4 is pretty cool to use in applications because of its ultra-low compression and decompression overhead.
@@lawrencedoliveiro9104 I wouldn't say so. Give tar.zst or 7z created with zstd/brotli/lz4 patch to 100 people, I guess one of them will be able to decompress it. When I experimented with LZ4 i found it's useless for anything, but sparse data (such as segmentation labels where index does not change often) and then it's perfect - something like run-length compression. Am I wrong about LZ4? One I stumbled on lossy compression of scientific data that had maximum allowed error and that was likely using some kind of predictor (basically it compressed n-dimensional arrays of integers or floating point), but I have no idea what was the name and if it was free (I think it was not).
The key point not mentioned here is that information content is a statistical concept. The closer your data stream looks like to random noise, the less compressible it is. It is inherent in the design of compression algorithms that they produce a data stream that looks very much like random noise.
I really love these kind of videos. Short & sweet. You really feel like you've learned some fundamental (and interesting!) things in a very short amount of time
Wat. Living in the era of winrar and 7zip, it never occurred to me that archiving was an intermediary step for zipping multiple files, I always assumed the compression program would handle it on it's own. I've never been able to figure out the purpose of tar until this video. So that's what .gz and tar.gz meant.
It also nicely fits with the UNIX philosophy of having small programs doing a single focussed job. tar does the archiving while gzip does the compressing. :)
Yes, it is definitely becoming more popular. Blender has removed the option for gzip compression when saving its documents (though it can still read existing gzip-compressed ones) in favour of zstd.
Have watched a lot of your 100 seconds explanation and they are great but I given the nature of this one I thought you might have squeezed it into 50 seconds!!! Keep up the great work and thx!
In my 4th year of computer engineering and i never thought to actually figure out what tar stood for/ came from It makes it so much easier to remember what it does
One if not the most undervalued algorithm for a dev. I solved a couple of difficult performance issues only by using gzip at the right moment. You can read a gzip stream in a memory efficient manner as well and you don't need level 9 compression to benefit good compression ratio.
@Fireship 1:35 You're incorrect about the compression ratio. A ratio of 90.3% means that the size of the compressed file equals 90.3% of that of the original file, which means that the lower the ratio, the more compressed it is.
Can you do some "in 100 seconds" videos on different IDE's and build tools? Would be really cool to see a condensed explanation of VSCode, WebStorm, Postman, etc.
VS Code is a general purpose code editor that plays nice with all languages, as long as there's an extension for one. WebStorm, if I recall correctly, is a highly specialized IDE for writing JavaScript and related stuff, TypeScript, etc. I presume. It has specific features to work with JavaScript built into it that you either can't get on VSCode or need a ton of extensions to replicate. Postman is an app that's used to test API endpoints and develop API systems.
@@shlongchad6159 I've used all of these specific pieces of software and know them well, just figured they would be a good starting point if Fireship decides to branch out to development tools.
RAR is my favorite one! It's the only compression tool that can contain redundant information for restoration. It saved me a lot of troubles several times so I always add redundant info.
This saved me in production. I saw a massive reduction of 70% in case of large nested json lists having similar keys. It reduced 400+KB data size to approx 100+KB in my case
p sure you can use -a instead of -z with tar and it will determine which compression algorithm to use based on the extension of the archive file name you provide
Uuuuh what kind of web game are you doing, and what kind of data loading are you doing that it's slow? Usually you wouldn't do this kind of gzip for serving data on the web, maybe use something like nginx, or whatever framework you're using for inbuilt gzip options. For example, if you're using nginx already, you just need to write the directive gzip on You can even vary between the sizes you want, etc. I also know next.js just has an option to compress. You'd probably want to use those instead of manually gzipping your files
you shouldn't handle the gzipping of your stuff in your game code, but in the web server serving the game files. it does gzipping on the fly. and of course set it up to cache static files so it's not constantly gzipping every request.
gRPC in 100 Seconds please!! We’ve started using gRPC at our company and love it, mostly for the type enforcement and reduced payload size thanks to sending information over a binary stream instead of human readable, inefficient-to-parse JSON
![How are Images Compressed? [46MB ↘↘ 4.07MB] JPEG In Depth](http://i.ytimg.com/vi/Kv1Hiv3ox8I/mqdefault.jpg)
This is exactly what 100 Seconds is for. I'm never going to do a full course on gzip but it's nice to know what it's about.
No need, there's the manpage
Or the tldr
you will at some point face gzip and wonder what the magical parameter string means it's just a matter of time
In 100 seconds he managed to mess up the name of one of the inventors...Mark Adler, not "Alder".
There's "full courses" on gzip? What? 😄
I've tried to read book about compression once and I regret it.
When I first got into linux all these words "tar", "gzip", "tarball" and their commands were a mystery which took me way longer than 100s to figure out. This video is gold for getting started. Putting all this in perspective in less than 2 minutes is just awesome.
I also like how they are not afraid to get "funny" with it. You have "more" and then "less" ("less is more"), you have weird names like 🔫zip, and tarball, and to get help with something you call the "man"
@@kte4423 but man isn't originally funny. I think its short for manual (like a computer manual/book)
@@daniel_2 Sure, gunzip makes sense too. I just find it funny that it's called "gun" (zip)
It's also ridiculously hard to zip/tar things with CLI compared to a GUI. Archiving and compressing being separate commands, weird flags you have to learn and footguns like accidentally zipping every file in a folder individually (happened to me several times).
SO THAT'S WHAT THE TAR EXTENSION IS, was always curious about that whenever i had to do something in linux, but never actually searched it up
Wow this was my exact reaction as well. I was always so confused about the .tar.gz, makes so much sense now
sigh, fucking normies, 5y.o I was compiling arch.
Yeah, it's helped me to finally know why the files end in their composite extension like .tar.gz instead of just .tar or .gz or something else
TApe aRchive
@@fakezpred No, it's Tape ARchive. Do you even Unix?!
One interesting difference with zip is that in zip each file is compressed and then archived, whereas in tar.gz you archive all the files and then compress the whole thing.
Advantage: Cross-file compression (i.e. repeated patterns across files are used for compression)
Disadvantage: You can not extract a single file and leave the rest alone.
That's what's nice in RAR, it has a flag for that. If you want to compress a huge library of pictures or videos for example, you might want to have them independently compressed. But a large database or text-like files or things like a complete game / program you might want to compress the whole archive.
Couldn't one just gzip [insert glob pattern here] and then tar the individually gzipped files if non-solid compression is necessary?
The main use for archives that I see is not for saving my own data, but for distributing collections of files. And normally when you get one of those you want to extract everything.
@@Winnetou17 7z is the new rar.
The actual compression algorithm (deflate) is the same between zip and gzip. Zip adds the archiving around it, while gzip just adds a small header and a checksum, so it's lighter.
You're all lucky newer implementations of Tar stripped all the tape-related stuff. Reading through the man page you'd spend an hour to find how to create, or add to an archive, while in the process learning how to rewind the tape, seek the next archive on tape, activate a jukebox-like tape swapper mechanism, wipe a tape, check whether there's enough free space after the last archive to fit yours, switch to a different track on a multi-track tape, set tape drive rpm, and a myriad of others.
that seems fun
@@daleryanaldover6545 but probably counter-productive too
What? ...huh?? was Tar actually used in the days of magnetic-strip-tape memory?
@@stickyfingies what "in the days"? Tape drives are used a lot right now in datacenters, like google and amazon, they are great for long term data storage
@@symix. I had no idea; crazy
This man is so considerate, he even included the -v flag (which for starters stands for verbose) just so you can see your files being compressed, truly a born teacher. Here's an explanation of the flags used by him:
-c = stands for create, it's the one you'd use the most.
-z = tells tar to compress using gzip. There's many other ones you can choose from, which imo gzip being the best.
-v = verbose, tells tar to display output in the terminal.
-f = stands for filename, usually a directory.
You can read most, if not all options using the --help command, with two hyphens.
Even if you use windows, using the terminal makes your life way faster if you get used to it. I highly recommend giving a shot to the lifestyle.
Where can I learn about the basic Windows equivalents to core GNU functions?
@@Gesepp95 Most of the stuff you can use in linux is included in windows as aliases. They should have similar, if not the same functions that what you're used to has.
@@sum1179 Thank you!
not gonna lie, i assume when most people write tar -cvf that they literally just learned it as a package deal and haven't looked into it.
I remember the flags as "Compress Ze V--king Files"
It’s refreshing when a video focuses on information density
Oh yeah, compression really played a hand when I implemented a redis caching in the backend. It significantly reduced memory usage and network traffic to cloud redis. And gzip is actually really fast too (we save string to redis as key val).
So it is string key and gzip file as the Val?
Had the same issue, using compressed GZip strings and Redis type Hash, the size used is 1/10 of what it was before
Nah, you can save different data types as val but not a file if I remember right. What I did is JSON stringified data and compress the string after.
Issue with this approach is that Json parse is not enough to convert values like functions, but that's okay cos you shouldn't be caching functions inside redis anyway.
Oh and Json parse is crazy fast to in es5
What did I just read 😂
fireship your insane! im binging your entire videos like high quality paid courses and its actually helping me alot in my assignments, kudos
I had to write my own implementation of the Huffman encoding algorithm for a computer science class back in the day. Compression and data encoding is a really interesting subject! It's an interesting scientific problem to figure out how to represent lossless information in the last amount of bits possible.
While an undeniably good developer, Jeff really shines with his ability to synthesize and condense down information. These videos are snappy, engaging, and eloquent. Thank you a million, Jeff.
tarballs sounded so intimidating to me and i thought it was something complicated lmfaoo i always wondered. its so freaking simple. i just made gzipped and gunzipped a couple files and then made a tarball out of them. shrank 250kb to 80kb. Cool stuff lol Just goes to show you how you can get in your own way sometimes. And also how good Fireships videos are.
I love that your channel is pure information, no duplication. Perfect compression 👌🏼😌
These 100 second videos are in a way gzipped version of all the knowledge acquired over a long time, isn't it? 🙂
😂😂😂
more like jpeg lol
It would need brotli. Then it would maybe actually be 100 seconds
I usually use zstd (Zstandard) for files when the portability of gzip or zip is not required.
Also lz4 is pretty cool to use in applications because of its ultra-low compression and decompression overhead.
zstd is widespread enough that you can consider it portable nowadays. And tar supports it, too.
btrfs uses zstd transparent compression.
Not by default.
@@MenacingPerson but it does in fedora.
@@lawrencedoliveiro9104 I wouldn't say so. Give tar.zst or 7z created with zstd/brotli/lz4 patch to 100 people, I guess one of them will be able to decompress it.
When I experimented with LZ4 i found it's useless for anything, but sparse data (such as segmentation labels where index does not change often) and then it's perfect - something like run-length compression. Am I wrong about LZ4? One I stumbled on lossy compression of scientific data that had maximum allowed error and that was likely using some kind of predictor (basically it compressed n-dimensional arrays of integers or floating point), but I have no idea what was the name and if it was free (I think it was not).
omg I actually understand tar gz now. this is fantastic.
Always enjoy content designed around re-enforcing the basics. Your 100 second format is a great way of presenting it.
The key point not mentioned here is that information content is a statistical concept. The closer your data stream looks like to random noise, the less compressible it is. It is inherent in the design of compression algorithms that they produce a data stream that looks very much like random noise.
Jeff is still my favourite tech youtuber
I would loved to have this video 5 years ago when i deep dived into linux. It's excellent ❤️
Nice! Super informative! Been doing this over 12 years and just this is a ton of news to me.
Yessss, please do more videos like this.
Linux commands like these are made easier to grasp like this.
Do the tar command next!
those 100 seconds series is fking AMAZING !!!!
I really love these kind of videos. Short & sweet. You really feel like you've learned some fundamental (and interesting!) things in a very short amount of time
Wat. Living in the era of winrar and 7zip, it never occurred to me that archiving was an intermediary step for zipping multiple files, I always assumed the compression program would handle it on it's own. I've never been able to figure out the purpose of tar until this video. So that's what .gz and tar.gz meant.
It also nicely fits with the UNIX philosophy of having small programs doing a single focussed job. tar does the archiving while gzip does the compressing. :)
Thank you Jeff for another amazing computer science explanation. I think you should make a video about data transformation with DBT.
I thought he was out of ideas, now I see he is only getting started. THIS is the good stuff
I learned more in 2 minutes TH-cam video understanding how GZIP works with Linux than an entire month at college
Thnk you for making this video!
Left out the standard for most *nix packages in 2022... zstd.
The whole linux ecosystem pretty much uses zstd now. It's insanely good.
Never heard of zstd. Should I use ?
Yes, it is definitely becoming more popular. Blender has removed the option for gzip compression when saving its documents (though it can still read existing gzip-compressed ones) in favour of zstd.
zstd would beat out brotli on the web if it were implemented with a similar dictionary
Have watched a lot of your 100 seconds explanation and they are great but I given the nature of this one I thought you might have squeezed it into 50 seconds!!! Keep up the great work and thx!
this probably is the epitomy of how to much knowledge can be squezed into a juicy video
In my 4th year of computer engineering and i never thought to actually figure out what tar stood for/ came from
It makes it so much easier to remember what it does
Keep everything abstract until you aim for using it. This is the reason I follow Jeff's tutorial.
I didn’t realize how much I wanted this video!
It's a friday night, I'm absolutely hammered, and now ecstatic to know why it's called a TAR file. So glad I found this channel
The best explanation I found about tar is in this 100 seconds video. Thank you!
It is not freaking 100 seconds!
Just about time when I am currently working/understanding gzip for my nginx config. Good intro, adds 2yrs of exp in my resume...
One if not the most undervalued algorithm for a dev. I solved a couple of difficult performance issues only by using gzip at the right moment. You can read a gzip stream in a memory efficient manner as well and you don't need level 9 compression to benefit good compression ratio.
Top-notch quality content as always! PS. Didn't expect to see the Snowman in the vid 😎
You are the master td;lrizer of computer concepts! Awesome channel.
Thanks for providing us with the command to archive/compress tarball files, my brain can't wait to forget it 🙃
I've learned so much from these 100 seconds videos, thanks a billion
Oh yes, the newest fireship video. Great as always!
huge fan of this series
I love your videos! I can't get enough of them.
I definitely support these kinds of videos where we discover some of Linux commands.
An eleguant way to talk about the things we use everyday, thanks.
I've never had that feeling of wow a video dropped right at the perfect time, until now
@Fireship 1:35 You're incorrect about the compression ratio. A ratio of 90.3% means that the size of the compressed file equals 90.3% of that of the original file, which means that the lower the ratio, the more compressed it is.
I was also confused about the ratio formula whether it is calculated as (new)/(original) or (original-new)/original.
Ratio + Ls + Gzip
best video for this format yet
Those 100 seconds videos are just fire!🔥
I WISH I watched this video when I first started using and learning Ubuntu years ago. This would have been a life changer for me
"{Linux_Command} in 100 seconds" is the series that the world needs.
-czvf and -xzvf are the only tar flags I know, and im happy to see it basically used in this video
i dont think you need -z while eXtracting anymore
also try czavf instead
This is the perfect one.. The perfect material for 100 seconds IMO..
Tar has always been a mystery to me. Thanks for explaining it so clearly!
These videos are so helpful and revealing for stuff I’ve always wondered about thank you!
This is awesome !!! I was always wondering why we use the combination on tar and gzip. Now I know !!!
Can you do some "in 100 seconds" videos on different IDE's and build tools? Would be really cool to see a condensed explanation of VSCode, WebStorm, Postman, etc.
VS Code is a general purpose code editor that plays nice with all languages, as long as there's an extension for one.
WebStorm, if I recall correctly, is a highly specialized IDE for writing JavaScript and related stuff, TypeScript, etc. I presume. It has specific features to work with JavaScript built into it that you either can't get on VSCode or need a ton of extensions to replicate.
Postman is an app that's used to test API endpoints and develop API systems.
@@shlongchad6159 I've used all of these specific pieces of software and know them well, just figured they would be a good starting point if Fireship decides to branch out to development tools.
He has videos about vscode and vim i think
how to exit vim in 100 seconds
@@KerbiterAnd it’s just “ESC :qa!” and then 99 seconds of silence?
I liked this video more than all the language videos in this channel.
Funfact, the like button glows when Jeff say's like and subscribe
yes i thought i was the only one see it
“Ending in dot Jeezy” that took me out. Lol. You’re goated for that one 😂.
Really nice video, please do every stuff we encounter on linux like that!
wish one day i can make pro and clear videos like yours great job always (im not a programmer i just enjoy the videos)
RAR is my favorite one! It's the only compression tool that can contain redundant information for restoration. It saved me a lot of troubles several times so I always add redundant info.
Awesome video as always, I would love a more thorough video on how gzip actually works internally
The information in this video is pretty compressed, too.
Really helpful all this terminology is demystified 🙌🏿
once i get my next paycheck i'm buying lifetime pro on your website, keep up the great work man
Great! Thanks Jeff! Now I can add gzip to my resume!
Finally the tarball hits the ground and gzips. Enlightenment.
This saved me in production. I saw a massive reduction of 70% in case of large nested json lists having similar keys.
It reduced 400+KB data size to approx 100+KB in my case
The way I always remember the flags to gzip+tarball is thinking in Arnold's voice "Compress Ze Files!!!" -czf
This dude actually made gzip interesting 😭
Very inspiring! Your content is really great and has helped shape me as a new TH-camr and also data scientist/software engineer. Thanks a lot!
This is critical information for a new Linux user. I just made some connections myself 😅
Throwing my hat's away. Thanks you, this was really helpful. ❤
Great content, as always !
OMG, it really worked. Thank you so much!!
p sure you can use -a instead of -z with tar and it will determine which compression algorithm to use based on the extension of the archive file name you provide
Loved that bonus of "what is tar". That's why the .tar.* extensions.
i would love to see a 100 seconds video about nim
Thanks! I had been trying to make a web game load faster. People told me to use gzip, but I had no idea how! Your video came at the perfect time
What sort of web game?
Uuuuh what kind of web game are you doing, and what kind of data loading are you doing that it's slow?
Usually you wouldn't do this kind of gzip for serving data on the web, maybe use something like nginx, or whatever framework you're using for inbuilt gzip options.
For example, if you're using nginx already, you just need to write the directive
gzip on
You can even vary between the sizes you want, etc.
I also know next.js just has an option to compress.
You'd probably want to use those instead of manually gzipping your files
you shouldn't handle the gzipping of your stuff in your game code, but in the web server serving the game files. it does gzipping on the fly. and of course set it up to cache static files so it's not constantly gzipping every request.
These videos nowadays feels like blessings 💜
Thanks for making video on gzip. Huge thanks. Also could you please make video on... other tools and protocols.
VISUAL BASIC IN 100 SECONDS , WOULD WANT TO SEE THAT ONE FOR SURE
I love your content man. 💜
gRPC in 100 Seconds please!! We’ve started using gRPC at our company and love it, mostly for the type enforcement and reduced payload size thanks to sending information over a binary stream instead of human readable, inefficient-to-parse JSON
Thanks, now I have more questions then when I started.
Next video topic request: Whats Tar and the different flags it uses.
Great content, would watch again.
Thank you jeff for this very informative video!
Short and informative 👍
Hey Jeff, could you please do a video on the Nim programming language?
I love this channel.
Your like && subscribe tag is always on point. Loved it getting zipped at the end!
🤣🤣
Thank you for making this.
THANK YOU SO MUCH FOR THIS!
You can easily do compression with numbers: Instead of writing ASCII digits, decompose the number in bytes and write the bytes to file.
This is a really impressive video.