I'm fully onboard with the Nix way. It's a big improvement! But one thing I'm still concerned with when discussing "reproducibility years into the future"-- and that's the source files. If you can't get derivations from the binary cache, often the fallback is building from source, and those sources can still disappear from the internet. The github repo could be deleted, or a server could change a base url, etc. It would be great to be able to vendor sources when that's desired.
Yes a flag that takes a snapshot of all the source dependency repos at the commits used and bundles them up along the lock file so that you can always rebuild from source, assuming no bootstrapping issues, would be awesome. Then a flag to point to this bundle to use in place of the regular repos.
If you are dealing with legacy near obsolete software, you are likely better suited to mirroring the source and creating your own pkg cache. You don't have to grab packages from nixpkgs you could manage the packages yourself. Re-using already built pkgs.
I've been looking at some enhancements to the nix to container conversions where one would have greater control over the layers and how they were composed. Another angle is to teach the container ecosystem about "nix layers" that are known to be non-conflicting, and thus don't have the performance overhead of overlayfs, limit to 128 layers, can be processed in parallel, etc.
I am so glad you are following up with Nix again, I have not made the time yet to pour into a deeper understanding. I am curious about the size of programs if they need to bring along their own dependencies, how much Nix relies on internet access, and how it would work if one wanted to modify a package to use an additional library (say for a specific sensor in a logging app).
When in the container world; the dependencies will be there no matter what. The nice thing about Nix-based and built containers is that their runtime is automatically minimized. This removes all the build environment pieces from the container leaving only that which is necessary for runtime.
Very useful, thanks. After trying it for a few months I finally switched to NixOS in all my development enviroments. I'm still fine-tuning all the dev tools and stuff, and this video is a treasure trove of cool ideas and precious information.
I'm glad you're enjoying it. Screencasts can easily become quite dull, so when I'm talking about software, hopefully I can keep it interesting with lots of hands-on action and visual interest.
awesome content, explained a lot of things I did not grok before by reading blog posts. I would suggest to setup you editor with syntax highlighting, it helps reading the nix code for the untrained eye. Please make more videos
Nice video. What I really haven’t seen so far is packaging something like a Django or Drupal application and put that for example into a container. Especially if one doesn’t use tools like poetry2nix but rather runs poetry directly. Most of the stuff is just, here is how to package this basic C library.
I really want to like the nix approach, but the declarative nature of a Dockerfile is so much more readable and accessible. The Nix approach seems to overcomplicate things significantly. Maybe this cool for weekend projects, but I see the risk of wasted developers hours trying to understand the intricacies of nix, that would be quite straightforward and well documented in Docker. Add the really bad documentation of nix and you realize that any CTO pushing nix in a productive environment will be cursed by devs that want to work on code not waste their time on their tooling. There’s no wonder Shopify abandoned its nixos experiments.
For dev environments, tooling, languages, libraries, etc. You don't need to create a Docker container to accomplish things locally. For example, you can setup direnv and put a flake.nix in your project repo. When you clone the repo and change to the project directory, the flake will be run to install just what you need for the project. When you exit the project directory, the environment self-destructs and the /nix/store packages are dereferenced. When you go back into the project it will load the already installed dev env near instantly. This allows to have various incompatible environments that would not play together nicely and switch between them. You can also just spin up a nix-shell with the tooling you need. Meanwhile, building containers with Nix is still viable and useful to spin up containers in Kubernetes, etc.
hi @matthewstott3493 , i tried that but it downloads the tools each time - probably due to the nix.gc runs. i don't know, but it downloads some 400 MiBs every run after few days. i don't have that kind of network all the time. > _"put a flake.nix in your project repo"_
this is so funny... I was following along (~6 months after this video was launched) and the Dockerfile build completely failed and I gave up trying to get it to work. But the nix way worked (after a simple syntax fix of adding a missing ')' to the flake.nix file. After that nix worked "one-shot" basically. Wow, docker confirmed not reproducible but nix is!
Great content, great style. I do wonder how long Nix is going to keep Flakes marked as experimental … but I agree they’re the future. I need to dive in more though. As to the idea these images can be built for years to come, someone else in the comments touched on this but if the cache were no longer around Nix automatically goes back to sources (e.g. GitHub) … but for absolute certainty of longevity maybe you could cover automating the process of cloning sources or caches onto your own infrastructure internally and building from there? Exact replica and reproducible builds are a huge thing where finance is concerned of course.
its kinda over complicated, i dont understand the benefits of flakes if its not nixos configuration.nix, you can just pin the package tree and if you really need add overlays. ive been building my containers with nix for the last 6 years ;)
I'm all for nix, weirdly it's better at building docker images than dodcker. However, it should be mentioned that you CAN specify a hash instead of a tag in dockerfiles.
Hello! The rec binding allows for attributes to reference their sister attributes within an attribute set. For example, rec { name = { some = thing; }; default = name; } In this example "default" references its sister attribute "name". TH-cam won't let me post links, but use a search engine to find "Language Constructs - Nix Reference Manual" and search in the page (Ctrl+F) for "rec" to read the technical documentation.
It sounds cool on paper, but I don't think it's worth it. Man, you go from 8 lines of readable statements to more than 60 lines of gibberish. Really the syntax is ungodly . Thanks for the video.
Yes quite large. It would be interesting to see what is being pulled in. For example, the JRE might be pulling in the graphical desktop stack. It is usually possible to override that sort of thing.
So it is just matter if you trust ubuntu or if you trust nix 🤣Cause dockerfile is clearly more readable and you admit it. This comparison example is not good. Like you can easily solve it by uploading your build docker image somewhere - and trust yourself that you not gonna delete it nor change it - that is when tagging and sha1 hashes comes handy, cause tag latest feels not stable. Nix is nice for that development thingy but as a docker container replacement I think no.
No it’s not about trust with Nix. With the Flake lock file you get the exact same version. When you rebuild the Ubuntu image you might have the base image pinned but what happens in the RUN steps can be different every single time. Unless you know how to exactly pin every apt package etc. With Nix this is already built in and each package brings its own isolated dependency tree which helps further with reducing problems. In short if you rebuild the Nix image a few months later you get the exact same image including all the bugs that were present at that time. When you rebuild the Ubuntu image you will get a similar image but at minimum a lot of the packages got changed because of security updates. The changes might be small in an LTS but they are there and you hope that it still works.
you might still need to run something old, when you are trying to modernize it and you want to compare how did the old thing worked. also, not everything runs on the public internet, so security vulnerabilities might not be a problem.
I'm fully onboard with the Nix way. It's a big improvement! But one thing I'm still concerned with when discussing "reproducibility years into the future"-- and that's the source files. If you can't get derivations from the binary cache, often the fallback is building from source, and those sources can still disappear from the internet. The github repo could be deleted, or a server could change a base url, etc. It would be great to be able to vendor sources when that's desired.
True. That's why nix has the concept of substitutors. URL fetchers also allow for a list of mirrors. Finally there is work in progress for IPFS.
@@MarkRuvald that sounds awesome!
Yes a flag that takes a snapshot of all the source dependency repos at the commits used and bundles them up along the lock file so that you can always rebuild from source, assuming no bootstrapping issues, would be awesome. Then a flag to point to this bundle to use in place of the regular repos.
A transparent caching feature would be amazing. Allowing you to cache on a set of servers on-prem so you can manage the lifecycles yourself.
If you are dealing with legacy near obsolete software, you are likely better suited to mirroring the source and creating your own pkg cache. You don't have to grab packages from nixpkgs you could manage the packages yourself. Re-using already built pkgs.
I've been looking at some enhancements to the nix to container conversions where one would have greater control over the layers and how they were composed. Another angle is to teach the container ecosystem about "nix layers" that are known to be non-conflicting, and thus don't have the performance overhead of overlayfs, limit to 128 layers, can be processed in parallel, etc.
I’m sure I’m not alone in saying anything you find in that process would be of great interest to the wider community.
I can see why people really love this channel. Great explanations. Cheers!
I am so glad you are following up with Nix again, I have not made the time yet to pour into a deeper understanding. I am curious about the size of programs if they need to bring along their own dependencies, how much Nix relies on internet access, and how it would work if one wanted to modify a package to use an additional library (say for a specific sensor in a logging app).
Yeah I'll definitely do a few more videos about Nix, and I'll keep these topics in mind.
When in the container world; the dependencies will be there no matter what. The nice thing about Nix-based and built containers is that their runtime is automatically minimized. This removes all the build environment pieces from the container leaving only that which is necessary for runtime.
Very useful, thanks. After trying it for a few months I finally switched to NixOS in all my development enviroments. I'm still fine-tuning all the dev tools and stuff, and this video is a treasure trove of cool ideas and precious information.
I'm really enjoying this new style of video.
You're surprisingly good on camera imo :)
I'm glad you're enjoying it. Screencasts can easily become quite dull, so when I'm talking about software, hopefully I can keep it interesting with lots of hands-on action and visual interest.
thanks so much for the detailed video. saved to use and review later. thanks again. more videos on Nix please are always welcome
awesome content, explained a lot of things I did not grok before by reading blog posts. I would suggest to setup you editor with syntax highlighting, it helps reading the nix code for the untrained eye. Please make more videos
Nice video. What I really haven’t seen so far is packaging something like a Django or Drupal application and put that for example into a container. Especially if one doesn’t use tools like poetry2nix but rather runs poetry directly. Most of the stuff is just, here is how to package this basic C library.
I really want to like the nix approach, but the declarative nature of a Dockerfile is so much more readable and accessible.
The Nix approach seems to overcomplicate things significantly.
Maybe this cool for weekend projects, but I see the risk of wasted developers hours trying to understand the intricacies of nix, that would be quite straightforward and well documented in Docker.
Add the really bad documentation of nix and you realize that any CTO pushing nix in a productive environment will be cursed by devs that want to work on code not waste their time on their tooling.
There’s no wonder Shopify abandoned its nixos experiments.
Thank you for this video. Glad I found it :)
Thanks for showing a highly informative, yet no boiler plate, nix tutorial
For dev environments, tooling, languages, libraries, etc. You don't need to create a Docker container to accomplish things locally. For example, you can setup direnv and put a flake.nix in your project repo. When you clone the repo and change to the project directory, the flake will be run to install just what you need for the project. When you exit the project directory, the environment self-destructs and the /nix/store packages are dereferenced. When you go back into the project it will load the already installed dev env near instantly. This allows to have various incompatible environments that would not play together nicely and switch between them. You can also just spin up a nix-shell with the tooling you need. Meanwhile, building containers with Nix is still viable and useful to spin up containers in Kubernetes, etc.
Thanks for the tip. Just checked out direnv and its really neat.
hi @matthewstott3493 , i tried that but it downloads the tools each time - probably due to the nix.gc runs. i don't know, but it downloads some 400 MiBs every run after few days. i don't have that kind of network all the time.
> _"put a flake.nix in your project repo"_
this is so funny...
I was following along (~6 months after this video was launched) and the Dockerfile build completely failed and I gave up trying to get it to work.
But the nix way worked (after a simple syntax fix of adding a missing ')' to the flake.nix file.
After that nix worked "one-shot" basically. Wow, docker confirmed not reproducible but nix is!
I am very happy to see you again. good luck.👍
You start building your app now, you finish it in 2099 with this stack.
Great content, great style. I do wonder how long Nix is going to keep Flakes marked as experimental … but I agree they’re the future. I need to dive in more though.
As to the idea these images can be built for years to come, someone else in the comments touched on this but if the cache were no longer around Nix automatically goes back to sources (e.g. GitHub) … but for absolute certainty of longevity maybe you could cover automating the process of cloning sources or caches onto your own infrastructure internally and building from there?
Exact replica and reproducible builds are a huge thing where finance is concerned of course.
its kinda over complicated, i dont understand the benefits of flakes if its not nixos configuration.nix, you can just pin the package tree and if you really need add overlays.
ive been building my containers with nix for the last 6 years ;)
I'm all for nix, weirdly it's better at building docker images than dodcker. However, it should be mentioned that you CAN specify a hash instead of a tag in dockerfiles.
how do you access the Website from this Video? Tried a few times, will not let me in. Carlo ( PS no register link)
This very very useful on how convince a manger to pay you a huge bill for you consulting for something that probably he don't need. 😅😅😅
Love Nix
Can't you just replace docker with flakes? I thought that was possible.
14:47 use cases: deployment vs IDE
7:55 what is that "rec" attribute?
Hello!
The rec binding allows for attributes to reference their sister attributes within an attribute set.
For example,
rec {
name = { some = thing; };
default = name;
}
In this example "default" references its sister attribute "name".
TH-cam won't let me post links, but use a search engine to find "Language Constructs - Nix Reference Manual" and search in the page (Ctrl+F) for "rec" to read the technical documentation.
See 17:25 in the video.
@@matthewboyea3860 thanks a lot
what about how to update nixos? thanks!!!
It sounds cool on paper, but I don't think it's worth it. Man, you go from 8 lines of readable statements to more than 60 lines of gibberish. Really the syntax is ungodly . Thanks for the video.
number of layers simply unacceptable ;((
hey otl, it's pronounced Scipio, like the roman dictator.
I build a jenkins docker image with nix, it wss 1.7GB 😆😆
Yes quite large. It would be interesting to see what is being pulled in. For example, the JRE might be pulling in the graphical desktop stack. It is usually possible to override that sort of thing.
So it is just matter if you trust ubuntu or if you trust nix 🤣Cause dockerfile is clearly more readable and you admit it. This comparison example is not good. Like you can easily solve it by uploading your build docker image somewhere - and trust yourself that you not gonna delete it nor change it - that is when tagging and sha1 hashes comes handy, cause tag latest feels not stable. Nix is nice for that development thingy but as a docker container replacement I think no.
No it’s not about trust with Nix. With the Flake lock file you get the exact same version. When you rebuild the Ubuntu image you might have the base image pinned but what happens in the RUN steps can be different every single time. Unless you know how to exactly pin every apt package etc. With Nix this is already built in and each package brings its own isolated dependency tree which helps further with reducing problems.
In short if you rebuild the Nix image a few months later you get the exact same image including all the bugs that were present at that time.
When you rebuild the Ubuntu image you will get a similar image but at minimum a lot of the packages got changed because of security updates. The changes might be small in an LTS but they are there and you hope that it still works.
I hope to hell you won’t be building these images years from now.
why is that?
@@richtigmann1 Security vulnerabilities are a thing.
you might still need to run something old, when you are trying to modernize it and you want to compare how did the old thing worked.
also, not everything runs on the public internet, so security vulnerabilities might not be a problem.
8:55 i added (writeTextDir "tmp/nginx_client_body/.placeholder" "") to contents, because nginx seemd to refuse to work without the tmp directory