Thanks Wesley, I'm glad it was helpful! In my next video I'm going to share some tips for increasing container security, and multi-stage dockerfiles are useful there as well! (they make it possible to use a distroles image to reduce the attack surface area! github.com/GoogleContainerTools/distroless)
Hey bro, will u please debug this bitbucket alpine image failed to copy files: failed to copy directory: Error processing tar file(exit status 1): Container ID 166536 cannot be mapped to a host ID
We tend to use kitchen sink style docker containers that are large, take a long time to build, and are hard to move around. Good tips, I'm inspired to be more selective of our images!
There is certainly a balance to strike. For a data science I see where it can be nice to have lots of libraries/packages pre-installed. This is kind of the approach that jupyter has taken with their docker stacks images: For example the jupyter/datascience-notebook (jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-datascience-notebook) is nearly 4GB! It all comes down to the use case. If you hardly ever need to rebuild/push/pull (e.g. add a new library or bump a version #) then its not too bad to have large images. As always... "It depends" 😁
Could you please explain a bit more what youre actually doing in the multi stage approach and what step it is that reduces the size? Also, what editor are you using? Thanks
Yeah! For a language that can be compiled into static binary, using a multi-stage dockerfile where the first stage has all of the build dependencies and then the second stage starts from SCRATCH and simply copies the build binary and runs it is a great way to go for the ultimate minimal image!
I am building docker image for Shcrodinger Suite, which has lot of dependent tar file installations. I have used multistage build but still need to optimize size more.... How can I compress the dependencies which are being installed?
Do you have a link to a repo and/or Dockerfile you can share? Also, TH-cam comments are clunky for technical discussions, if you want to hop into the channel discord it would probably make for a more productive discussion! discord.gg/ExrENts5CT
Thank you very much its short and sweety. why couldn't we use alpine image in the first FROM itself, here we are copying /app to alpine don't we require the binaries at run time.
I think you are talking about the Golang example. We needed to use the golang:1.14-alpine image in the first stage because it contains additional dependencies required to build the application. We can then use the smaller alpine:3.12 image in the final stage because we dont need them at runtime! Does that answer the question?
Awesome video! I don't know a whole lot about writing Docker files, but why would someone ever not want to start with the slim base image in the first place?
Great question! In general, it is just a matter of convenience. The full image will often have utilities installed that are helpful when building/configuring your application that the slim container doesn't It will vary from one image to the next, but a few examples for the node base images are curl, wget, and git which are pre-installed in the full image but not in the slim image. If you needed to use them in the slim image you would need to first `apt update` then `apt install` them in the Dockerfile. -- This is why multi-stage builds are so great! You get all the convenience when writing your Dockerfile along with the smaller size, improved workflows (see my previous video for tips on using multi-stage images for debugging), and added security (my next video will include info about this 🙂)! Sorry for the super long answer -- hopefully that makes sense!
True that python is interpreted, but there are often cases where it is much easier to configure things in a full-sized image (with a bunch of utilities preinstalled) than working with a lightweight image from the start. For example, if you need to clone a git repo or download a binary it can be nice to have git and curl preinstalled 😁 Also, if your code is leveraging components written in other languages (wiki.python.org/moin/IntegratingPythonWithOtherLanguages)... then you might have a build stage for the C++ (or other) code and copy the built executables into your final python-based stage!
@@DevOpsDirective Well done. I always do it live with my talking because matching the speed I find hard really hard! you did it quite well! It flowed well. Also good content :)
Thanks @JT -- I'm actually working on a follow up to this one in which I build a fully functional web server container image in just 6.3 kB. Stay tuned! 😁
Thank you so much!!! i reduced my docker image from 900+MB to only 11MB for my discord bot!!! EDIT: after cleaning the cache it didnt work anymore so i updates some things and now the size is 67MB instead of 11MB
@@DevOpsDirective i used nodejs:12 but now im using alphine with npm+nodejs12, after a lot of crashes (i didnt cleaned the cache, so it looked it worked with 11mb but it didnt after an hour when all caches are cleaned) its now 67MB but thats still 7.25% of that it was before and for me still worth it!
Simple video, explaining the benefits of multi stage docker files. Great job
Thanks Wesley, I'm glad it was helpful!
In my next video I'm going to share some tips for increasing container security, and multi-stage dockerfiles are useful there as well! (they make it possible to use a distroles image to reduce the attack surface area! github.com/GoogleContainerTools/distroless)
Also, hello to all of the HN folks who found my channel through this post! news.ycombinator.com/item?id=23952024
Welcome to the channel! 👋👋👋
Hey bro, will u please debug this
bitbucket alpine image failed to copy files: failed to copy directory: Error processing tar file(exit status 1): Container ID 166536 cannot be mapped to a host ID
We tend to use kitchen sink style docker containers that are large, take a long time to build, and are hard to move around. Good tips, I'm inspired to be more selective of our images!
There is certainly a balance to strike.
For a data science I see where it can be nice to have lots of libraries/packages pre-installed. This is kind of the approach that jupyter has taken with their docker stacks images:
For example the jupyter/datascience-notebook (jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-datascience-notebook) is nearly 4GB!
It all comes down to the use case. If you hardly ever need to rebuild/push/pull (e.g. add a new library or bump a version #) then its not too bad to have large images.
As always... "It depends" 😁
Amazing video. Never tried using multi stage docker builds! I need to try this tomorrow. Thanks!
Oh man -- multi-stage dockerfiles are a game changer! 👌
If you hit any snags, post them here -- happy to help debug!
Great now o want to go back to all my previous projects to implement these techniques.
Thanks for the tips!
Nice! You are welcome, @dalmiro!
Could you please explain a bit more what youre actually doing in the multi stage approach and what step it is that reduces the size? Also, what editor are you using? Thanks
can we use multi-stage with ruby on rail app ?
Wuuuut??? I just learned about this hold up, how did u reduce the size like that???
We can also use from scratch to make it even smaller, but we have to install the compilers as well
Yeah! For a language that can be compiled into static binary, using a multi-stage dockerfile where the first stage has all of the build dependencies and then the second stage starts from SCRATCH and simply copies the build binary and runs it is a great way to go for the ultimate minimal image!
I am building docker image for Shcrodinger Suite, which has lot of dependent tar file installations.
I have used multistage build but still need to optimize size more....
How can I compress the dependencies which are being installed?
Do you have a link to a repo and/or Dockerfile you can share? Also, TH-cam comments are clunky for technical discussions, if you want to hop into the channel discord it would probably make for a more productive discussion! discord.gg/ExrENts5CT
Thank you very much its short and sweety. why couldn't we use alpine image in the first FROM itself, here we are copying /app to alpine don't we require the binaries at run time.
I think you are talking about the Golang example. We needed to use the golang:1.14-alpine image in the first stage because it contains additional dependencies required to build the application.
We can then use the smaller alpine:3.12 image in the final stage because we dont need them at runtime!
Does that answer the question?
It was one of my interview question and I messed up already. Hope I would be able to answer it in my next if asked. Haha
Next time for sure!
Awesome video!
I don't know a whole lot about writing Docker files, but why would someone ever not want to start with the slim base image in the first place?
Great question!
In general, it is just a matter of convenience. The full image will often have utilities installed that are helpful when building/configuring your application that the slim container doesn't
It will vary from one image to the next, but a few examples for the node base images are curl, wget, and git which are pre-installed in the full image but not in the slim image. If you needed to use them in the slim image you would need to first `apt update` then `apt install` them in the Dockerfile.
--
This is why multi-stage builds are so great!
You get all the convenience when writing your Dockerfile along with the smaller size, improved workflows (see my previous video for tips on using multi-stage images for debugging), and added security (my next video will include info about this 🙂)!
Sorry for the super long answer -- hopefully that makes sense!
@@DevOpsDirective It does indeed make sense! Thank you :)
I wish you applied the multi stage for the node too
Great tutorial!
Thanks, @Dirk!
That's great! Unfortunately I'm using Python so I am not able to use a multistage build (as python isn't compiled)
True that python is interpreted, but there are often cases where it is much easier to configure things in a full-sized image (with a bunch of utilities preinstalled) than working with a lightweight image from the start. For example, if you need to clone a git repo or download a binary it can be nice to have git and curl preinstalled 😁
Also, if your code is leveraging components written in other languages (wiki.python.org/moin/IntegratingPythonWithOtherLanguages)... then you might have a build stage for the C++ (or other) code and copy the built executables into your final python-based stage!
Thanks! Simple
Glad it helped!
Nice! Awesome video!
Thanks Marek -- Trying out a new style where I recorded the audio first and then did a screen record to match it!
@@DevOpsDirective Well done. I always do it live with my talking because matching the speed I find hard really hard! you did it quite well! It flowed well. Also good content :)
My trick is playing it back extra slow when doing the screen recording. It's annoying to do, but makes the edit much easier!
Great stuff, thank you.
Thanks @JT -- I'm actually working on a follow up to this one in which I build a fully functional web server container image in just 6.3 kB. Stay tuned! 😁
Thank you so much!!! i reduced my docker image from 900+MB to only 11MB for my discord bot!!!
EDIT: after cleaning the cache it didnt work anymore so i updates some things and now the size is 67MB instead of 11MB
Woohoo! Pinning this comment 😀
What language are you using/which techniques did you use to shrink the image?
@@DevOpsDirective i used nodejs:12 but now im using alphine with npm+nodejs12, after a lot of crashes
(i didnt cleaned the cache, so it looked it worked with 11mb but it didnt after an hour when all caches are cleaned)
its now 67MB but thats still 7.25% of that it was before and for me still worth it!
Yeah, from 900 --> 70MB is still pretty good!