To be honest, if you are self-hosting in a closed home network and somebody gains access to exploit many of these vulnerabilities, you probably have way way bigger problems than these vulnerabilities.
The idea of a reverse proxy with TLS offloading is normally to expose your services, so this is by design something meant to be on the outside of your network. This makes the chances of any exploit significantly higher, as people will be randomly scanning for these.
@@t3ddi ok yeah fair point for this particular case. But this is usually all mentioned in the context of a home lab where you usually have something like NPM mostly to give your services a readable domain, that's it.
@@Squitdoogenz I wouldn't go that far. It's just a concern Christian has. Yes, security is important, but we also have to be aware in what environments all these tools are running. If I run these tools in my closed homelab environment, I can make some compromises and not worry about certain things.
Tuesday is security day. More security stuff please! Some projects (like NPM) are so popular that one doesn’t consider checking up on the security track record. Thanks for reminding us. Just put out my own video on securing a VPS/Linux server - the lazy way for beginners :)
Just like you did videos about configuring Nginx Proxy Manager, now it would be interesting to do one on migrating to Nginx only. And maybe another one to move on to Traefik.
Since NPM is based on Openresty, which in turn uses Nginx as a base migrating to Nginx is a matter of studying the NPM repo and just adapt the Nginx configs which NPM uses. This way you get rid of all the NPM specific code and still get the benefit and ease of use. I created an Ansible role, which I based off NPMs configs since I liked the modular setup of the config, but wanted to be able to setup my whole infrastructure as code, which NPM doesn't support. This way I have the ease of use NPM offers, but don't have to deal with NPM itself. It took me quite some time to reverse engineer and customize everything and it still isn't perfect yet, but I'll keep working on the role until it supports all features of NPM that I might need.
I’m not sure I understood the advisory correctly but wouldn’t an attacker need to be authenticated to exploit this vulnerability? They used a JSON Web Token for the POST request that delivers the payload. Still bad but not as bad as exposing it to everybody. Also usually I don’t have such admin panels accessible to the open internet anyways (like most people I would hope) so in this case an attacker would need to have access to your home network as well as be authenticated in npm. By the time they are in your home network you probably have other concerns as their access to a docker container. But the general point made here is still valid
I was wondering the exact same thing. I haven't looked into the exploit, but it didn't sound like something exploitable directly from a proxied entry like a wordpress instance.
Love the focus on security with this video. With all the threats and constantly emerging vulnerabilities, it's a moving target and easy to forget the threat is real. Hope to see more videos touching on security.
One thing you forgot to mention is that Version 3 of NPM is being worked on now, and that it is a complete rewrite from JavaScript to a completely new Go backend and new JS frontend. This might also make it less likely to work on Version 2 as it will soon, eventually, be outdated. Version 3 has been in development for over two years.
It all depends on who has supposed to have access. If you are the only one, it's simpler to leave open only the VPN port and set all services inside to listen only on the tunnel interface, not physical ones. Some problems are mitigated in that case, even if the software is bugged.
Came here to say this exactly, why are any of these exposed anyway? There's no reason to pour forward anymore and expose the stuff to the world Setup a zero trust / access scenario and make sure there are no ports exposed to the world, along with a very strong password
The problem with NPM is that he has an open source project that he is basically developing on his own. He doesn’t accept many pr and is away months without any notice. I tried to get in contact with him but even after months he doesn’t respond. The team is a one man team.
For projects i want to rely on, eg. reverse proxy, i only use projects that have many people as part of the team. Even if the single developer speeds a significant proportion of their personal time on the project, you never know when they might give up on it.
The problem with traeffic is that it has access to the docker socket. This means a rce vulnability in traeffic = root on host! With NPM the attacker still has to escape the container.
i am not sure what the point of the video was. people who selfhost are probably technical literate enough and (should) know what to expect from FOSS projects. is it really a surprise that small projects are often maintained by few people in their sparetime? the reason those projects are open source is often so that people can contribute - not that people can just blindly trust and use them.
Thank you for the discovery of caddy. When I've read "easier than NPM", I said what ? But yes it is easier to configure. Not with a web UI but still easier. ;-)
A vulnerability in a self hosted project not exposed to the internet is still far less dangerous than say, a python library you don't know the true contents of, or random docker images, etc... Unless the CVE is that the project actively downloads malware or exfiltrates data, you have to just be careful about how you use small projects. Good luck combing through all the libraries most dev ops guys use.
So personally I don’t use nginx proxy manager but I agree this is a problem with smaller projects, less people using and testing against it means exploits might remain around a lot longer. But I don’t think this means small tools can’t be secure as it all depends on your threat model. As you also have to realize that NPM runs in a docker container if you use the recommended setup so the “Remote arbitrary code execution” is inside of a docker container so you would then need to escape that. It’s still definetley a bad thing but in these sorts of cases docker while not make it secure adds a layer of security which can help while these exploits are patched
I abandoned NPM. they suddenly introduced a retrocompatibility issue for Arm7. It was never solved, even it was reported by many. It was a nice project. It's a pity
This is a good discussion. Basically you trade cost for support. It really boils down to that. If you pay a company, they can afford to quit their "day jobs" and fix the bugs that inevitably come up.
This might be another video, but SaaS soluitions also have a similar impact; although, not as bad. I wouldn't say M$ or Google would be in that list, but there are quite a few smaller SaaS companies who are feature driven and not security driven. I've had several run-ins with smaller SaaS (but 'cost-effective') companies who would drag feet, or literally work against you when reporting or pushing for a security fix (even as a paying customer). NPM has somewhat of an excuse because they rely on donations, but it can be frustrating to see a paid solution do this. Also, one 'pro' to consider for self-hosting is you have more freedom to implement mitigations if there's a critical CVE on the loose, with a SaaS, you're at the mercy of the companies willingness to fix it in time. This helped me reconsider NPM and look at others like Taefik, thank you!
Been meaning to move everything off of NPM and into traefik. This video was the kick I needed. Moved everything over today! With the exception of homeassistant it's moved over just not resolving lol. Thanks for the great content!! :edit homeassistant resolves just fine. Just took forever!
Any tips or tutorials, I was going to migrate to it a couple of months ago but I found it overly complex for what I had at the time but since I now have Nextcloud, homeassitant, Vaultwarden and a bunch of other stuff I might slowly migrate to it. The main thing was that it looked like I needed a lot more work compared to NPM for a simple proxy?
The problem with traeffic is that it has access to the docker socket. This means a rce vulnability in traeffic = root on host! With NPM the attacker still has to escape the container. This is the main reason I'm sticking with NPM.
Wow thanks Christian! I am currently using cloudflared tunnel docker on my Synology NAS to connect to a nextcloud and mattermost docker, but I am having issue with SSL, as both nextcloud and mattermost have not certificate. I will try to use traefik, thanks for the tip!
What is a "small" project? Downloads or installations surely aren't the best indicator. Contributors may be a better indication but how many makes a project "big"?
i did the same thing (move from NPM to Traefik) for a similar reason - came across a bug in NPM and it didn't get resolved (don't know if it has since) but am never going back now - Traefik is far superior IMO - good vlog though
once i contacted Jenkins maintainers after trying more than 10 of their container images all with 100s of vulnerabilities,, they simply reply that they do not have a single Jenkins container image without security vulnerabilities
Reminder that any container with docker socket access effectively has zero sandbox. If an attacker can talk to your socket they have *root* access to your entire machine with all the same power you have. That includes kernel capabilities.
my skill level is just good enough to follow along TH-camrs like you are. I installed some Docker containers in MacOS, like Portainer, Pihole, Yellyfin, also tried to install Nextcloud behind Nginx and a self-signed cert, but all for experimenting and learning without expsing them to the Internet. 2 days ago I accidentally stumbled over Docker Scout within Docker Desktop which provides analysis of locally installed Docker images. Result: not a single image without vulnerabilites. Portainer shows 101 vulnerabilities etc. For a non-developer without IT qualification this feels quite concerning. Especially now I am thinking about my Home Assistant installation which is also built on Docker as far as I know
There is no 100% of security. You‘ll have to deal with it. Never put important data on DIY devices with open ports, manual/no updates, or without IT security monitoring and professional support. My home lap is also a result of „half knowledge“ in networking and IT security. So, I trust the most important data to big companies like Apple, because they can’t afford any vulnerabilities at all. With 2FA, encryption and so on. I would NEVER use a self-hosted vault for my important passwords, as long as Apple‘s keychain will do it better and in an even more secure way.
yesss!!! this is a really good point on this topic. THANKS A LOT Christian!!!!! these are my thoughts on this: traefic for external reverse proxy and nginxproxymanager for internal things (as it is easier to configure with a GUI instead of yaml)
I can understand the concerns you are raising, however: Having worked with and around many large enterprise organizations, I can say the people working there are no more competent that those working on small projects. I would say the real issue with small projects is reliable funding. If users don't pay for the use of the projects, the people working on those projects won't be able to devote time to them (they need food, clothing and shelter, just like the rest of us. If working on the project isn't providing those, then they must spend their efforts elsewhere). Large companies can cheat with respect to this, by turning the user into a product. That is the tradeoff you accept for "free stuff".
Thats interesting I was thinking about exactly the same thing few days ago. I'm actually using this project in homelab too. But the thing about vulnerability is exposure of control panel to the wide web. If Login and password will be exposed and attacker dont have access to your login page/protocol then you are safe. The other thing would be if you would use it on some vps to expose multiple dockers from one ip. Then this is big no no because you are forced to expose login page and some secure solutions need to be used. I actually learned haproxy that dont have any login implemented.
This is another reason to use Kubernetes instead of docker. Many companies are behind the software being used in production. Ingress-nginx is used way more than nginx proxy manager.
The most dangerous parts of the homelab are the ones that can be accessed externally without a login. Your reverse proxy and authentication system, or non-cloud VPN, are probably the most dangerous. Everything else requires a login to be able to access and is less risky. Just expect these critical services will have 0days. - Auto-update these services frequently, or have a process to get alerted when there new versions are released - Choose products written in memory safe languages. (Caddy & Traefik are both good choices) - Use sandbox techniques (AppArmor/SELinux) to further restrict system access. - Don't forget to block outbound network access from these services.
The other thing is, nobobdy will write malware for smaller projects. It's just not worth it for a few thousand users, after all you still have to find all servers which run the certain programm.
You make some good points, but big software is not the end all be all. How many libraries out there put out by large companies that are not actively maintained? They may not have any developers assigned.
But with self hosting i can spin up a bunch of vm's and isolate them for much cheaper than in the cloud. This way they cant spread. This definitely brings up the point to host one service per instance to mitigate the blast radius if something gets compromised. The difference for a vps is its on someone else's hardware and internet connection. Still need to make sure what your running is secure and patched.
The problem is also that many people report bugs without providing enough data to reproduce the bug or just entirely ignore issue templates that repo maintainers set up and just write random things which are not helpful. If people that reported bugs would invest some effort into bug reports, they potentially wouldn't be such a tedious thing to go through and resolve. I do understand that project maintainers are burned out in general and all the reported low effort crap they have to go through doesn't help at all.
Ha, the OSS project I'm a part of has almost 500 open issues 😂 People need to get over themselves regarding the issue count. Security-related bugs are fixed immediately, everything else is fixed whenever someone feels like it.
Proxy manager is a time saver but I would never consider it for production as there are many small details that nginx need to be adapted that the nice little gui has. I totally apreciate the proxy manager even exists and its free!! they should charge for it and fix issues :)
ปีที่แล้ว +1
It was ok npm, but i merged to pfsense and haproxy, faster and more reliable
Christian, thank you. I have tried nginx and use it internally in the lab... this makes me reconsider using it altogether. It was an easy tool but i guess that comes with consequences.
Thanks for the great channel. I'm just starting out with a Linux server (Ubuntu) running off a laptop for my "home lab". If you were to restart with roughly $2000 - $3000, what would you go for? It would mostly be test environments, but could potentially host initial production databases and applications. Could you do a vlog on this, unless the solution is stupidly straight-forward? Thanks.
Thank you so much for this. and i have believed that it is safe so after this, you have started to doubt. unfortunately, it's the only thing I've found that is user-friendly. traefik also talked a lot about but it is light years for me to understand how to do with it. Do you know any other alternative with ui that you would recommend and maybe even make a video about.
If you want a generic gateway, that's probably a vpn. I know VPNs have come in for a lot of hate, but running strong security at the application layer is so much harder, as you have to secure every application, and applications focus on features, not security.
I disagree with the point of view that self hosting is less secure than using a large service because the large service is a MUCH bigger target and will be attacked far more often. However, the inherent design of what's evolved in the self hosting space (a large number of tools exposing numerous web interfaces) is completely idiotic as all of those interfaces creates a huge attack surface. There is a massive history of exploits in web interfaces. This really should be consolidated into ONE interface, with one entry point that can be throughly tested.
That’s not what I said. I said when you’re selfhosting it’s better not to throw each and every small project in your stack without researching how well it’s maintained.
Find different project like NPM that has a good GUI - so its veeery easy to manage without digging into config files - potentially exposing more then you want xD NPM is just easy to learn xD in fact WAF is a good way to secure external assets. Layer7 is needed today.
Did it tho? The CVE for nginx proxy manager was rather minor as it required admin access to the service and then that said admin doing the attack. This is a problem for shared hosting for sure, but not really for self hosting scenario as you already have way bigger rights. Either way, something that should be fixed. But HAProxy actually has had critical level of CVE's. Looks like jailed instance of nginx proxy manager would have been safer option if past CVE's is what we care about.
To start with, I like most of your videos. They're informative. This one though... Sorry. We should all be security focused, and each piece of software you expose to the internet is an attack vector, we can't deny that. That being said, you fail to mention a few things: 1) the biggest use case for NPM - was the CVE on the biggest use case? I don't really think so. 2) more than 1000 issues open, yeah true, but the vast majority aren't security related. 3) using traefik in homelab scenarios is like using RHEL these days... you'll be booted, you just don't know when. Try doing a video on HAProxy for example. Much more useful than that...
One should be looking at the complexity of the codebase and NOT the size of the teem developing it. Bigger codebases are inherently more difficult to secure as more code = larger attack surface. A small teem developing a simple tool should be perfectly fine.
The point is that small teams mostly lack the skills and experience how to tackle security problems. Bigger projects aren’t protected against security vulnerabilities, but the chances are higher there are people and processes in place for handling security.
Unpopular opinion: This is inducing paranoia in people who don't really care to read about the vulnerabilities and understand them. As if using big companies are usually not patient zero. Just wait long enough until your business dies the next time aws goes down. And BTW big companies rely on small projects en mass. Self hosting is the most important gate for newbies to learn. As for alternatives to NPM - Traefik specifically has been proven for over 4 years to be missing fundamental core features that require plugins or some honestly stupid workaround to maybe resolve the missing features problems. The documentation is atrocious as well. I want to keep the tone positive, so i wont say anything about the way they respond to issues or feature requests. Literally any other proxy/ingress is better than traefik specifically. But trefic feels sponsored man. Lastly - dont ever count on any software to be completely exploit proof - just be smart about how you use it.
This is personally, why I use the real deal and not hobbyist's projects that try to make things easier (I'd say that it makes it harder). Less complexity means that CVEs like this are far less likely to happen, it's not very common to have a severe CVE in something like vanilla Nginx.
I mean yes, I understand that the nginx proxy manager team doesn't have the time to take care of all of the problems of the tool but nobody should let such a critical think as the nginx manager login page exposed to the public internet
One mistake to avoid is that you shouldn't presume that thousands of "reported" bugs or vulnerabilities should be seen as verifiable, reproducable, and mass-exploitable. Many of theses could be one-offs, or just users who have misconfigured something, or are using other poorly engineered software. Also, you can understand how a lot of for-profit companies might want to attack a small FOSS project. One which gives away something for free which the attacker wants to be able to charge people for. So, in an effort to stir up doubt and harm that project's rep, someone might file a bunch of false and very scary sounding reports, all the while deleting all the reports on their own site, and not fixing them. I'm not saying that's the case here, but it's something to take into consideration.
Those are open cases - 1000 of them, and not many have comments on them. Seeing how many cases are open and how few have comments, it looks like the developer(s) are not even looking at the cases. IQ3000 move: 'if you don't look at them then the issues cannot be verified'? Just because the issues have not been looked at doesn't mean all users have made a mistake or did something with a one in a trillion chance of happening again. I'm not saying all 1000 cases are actual issues because we have no idea. I think it's naive to assume there aren't any issues in the open cases.
Every script, application and service has open and unrealized exploits. Nothing on the internet is ever nor will ever be 100% secure. You should always keep backups. For this very reason. You’re never 100% safe, because where there is a will. There is a way.
Well to be fair that that nginx proxy manger security issue is a joke. Getting full sys access while being logged in? I mean comon i can steal all your admin passwords when i'm logged in. They would've probably fixed it quickly if the issue was criticial
I'm not fully agree with this one. Because how much of a safety risk is this bug in Nginx? Second Nginx is not a big program and has a lot lesser code. Than a OS like windows. So te comparison between windows and Nginx and a big company and a small company doesn't say much in my eyes. And bug what aren't a big big safety risk will be solved on a later time. This happens by big companies as wel. The best thing in my eyes is popular "Open Source Software" because then everyone can see the source code.
this is the reason why big companies will always have more sales than small ones - everybody is AFRAID the small company will simply vanish. Same for projects
In my experience, home-lab enthusiasts have better security than enterprise implementations. I worked for companies that had RSA hard-tokens on web-cams accessible to the Internet. I could go on and on.. nginx, docker, proxmox are not small tools, just small usage footprints in our hobby.
Love the focus on security with this video. With all the threats and constantly emerging vulnerabilities, it's a moving target and easy to forget the threat is real. Hope to see more videos touching on security.
To be honest, if you are self-hosting in a closed home network and somebody gains access to exploit many of these vulnerabilities, you probably have way way bigger problems than these vulnerabilities.
The idea of a reverse proxy with TLS offloading is normally to expose your services, so this is by design something meant to be on the outside of your network.
This makes the chances of any exploit significantly higher, as people will be randomly scanning for these.
@@t3ddi ok yeah fair point for this particular case. But this is usually all mentioned in the context of a home lab where you usually have something like NPM mostly to give your services a readable domain, that's it.
Yeah, this video is some pretty major shilly fud. Definitely calls into question the authenticity of the channel, imo.
nginx runs 35% of the Internet.
@@Squitdoogenz I wouldn't go that far. It's just a concern Christian has. Yes, security is important, but we also have to be aware in what environments all these tools are running. If I run these tools in my closed homelab environment, I can make some compromises and not worry about certain things.
I remember during the reveal of the HeartBleed vulnerability that it was brought to the forefront that OpenSSL was maintained by 2 people.
Tuesday is security day. More security stuff please!
Some projects (like NPM) are so popular that one doesn’t consider checking up on the security track record. Thanks for reminding us.
Just put out my own video on securing a VPS/Linux server - the lazy way for beginners :)
Just like you did videos about configuring Nginx Proxy Manager, now it would be interesting to do one on migrating to Nginx only. And maybe another one to move on to Traefik.
Since NPM is based on Openresty, which in turn uses Nginx as a base migrating to Nginx is a matter of studying the NPM repo and just adapt the Nginx configs which NPM uses. This way you get rid of all the NPM specific code and still get the benefit and ease of use.
I created an Ansible role, which I based off NPMs configs since I liked the modular setup of the config, but wanted to be able to setup my whole infrastructure as code, which NPM doesn't support. This way I have the ease of use NPM offers, but don't have to deal with NPM itself. It took me quite some time to reverse engineer and customize everything and it still isn't perfect yet, but I'll keep working on the role until it supports all features of NPM that I might need.
I’m not sure I understood the advisory correctly but wouldn’t an attacker need to be authenticated to exploit this vulnerability? They used a JSON Web Token for the POST request that delivers the payload. Still bad but not as bad as exposing it to everybody. Also usually I don’t have such admin panels accessible to the open internet anyways (like most people I would hope) so in this case an attacker would need to have access to your home network as well as be authenticated in npm. By the time they are in your home network you probably have other concerns as their access to a docker container.
But the general point made here is still valid
I was wondering the exact same thing. I haven't looked into the exploit, but it didn't sound like something exploitable directly from a proxied entry like a wordpress instance.
this...!
Love the focus on security with this video. With all the threats and constantly emerging vulnerabilities, it's a moving target and easy to forget the threat is real. Hope to see more videos touching on security.
Thanks! I’ll keep that in mind :)
One thing you forgot to mention is that Version 3 of NPM is being worked on now, and that it is a complete rewrite from JavaScript to a completely new Go backend and new JS frontend. This might also make it less likely to work on Version 2 as it will soon, eventually, be outdated. Version 3 has been in development for over two years.
Where is it? Xd canwe test it?
This especially goes for anything that faces the public internet, if you're using these kind of tools internally the risk is negligible.
It all depends on who has supposed to have access.
If you are the only one, it's simpler to leave open only the VPN port and set all services inside to listen only on the tunnel interface, not physical ones.
Some problems are mitigated in that case, even if the software is bugged.
Came here to say this exactly, why are any of these exposed anyway? There's no reason to pour forward anymore and expose the stuff to the world
Setup a zero trust / access scenario and make sure there are no ports exposed to the world, along with a very strong password
*port
The problem with NPM is that he has an open source project that he is basically developing on his own. He doesn’t accept many pr and is away months without any notice. I tried to get in contact with him but even after months he doesn’t respond. The team is a one man team.
For projects i want to rely on, eg. reverse proxy, i only use projects that have many people as part of the team.
Even if the single developer speeds a significant proportion of their personal time on the project, you never know when they might give up on it.
Yep, that is what I have seen too. Seems like a really crappy way to run such a popular piece of software
Great Video! Maybe im the only one wondering, but what is an alternative to NPM then? Traeffic or is it a similar story?
The problem with traeffic is that it has access to the docker socket. This means a rce vulnability in traeffic = root on host! With NPM the attacker still has to escape the container.
i am not sure what the point of the video was.
people who selfhost are probably technical literate enough and (should) know what to expect from FOSS projects.
is it really a surprise that small projects are often maintained by few people in their sparetime? the reason those projects are open source is often so that people can contribute - not that people can just blindly trust and use them.
I really like this vids, security is more important than ever!
Thanks!
been using Caddy for a month and honestly couldn't be happier , it was also easier to configure than NPM if you can believe it!
Thank you for the discovery of caddy. When I've read "easier than NPM", I said what ?
But yes it is easier to configure. Not with a web UI but still easier. ;-)
I love Caddy, but its limited in what it can do vs nginx
A vulnerability in a self hosted project not exposed to the internet is still far less dangerous than say, a python library you don't know the true contents of, or random docker images, etc... Unless the CVE is that the project actively downloads malware or exfiltrates data, you have to just be careful about how you use small projects. Good luck combing through all the libraries most dev ops guys use.
So personally I don’t use nginx proxy manager but I agree this is a problem with smaller projects, less people using and testing against it means exploits might remain around a lot longer. But I don’t think this means small tools can’t be secure as it all depends on your threat model. As you also have to realize that NPM runs in a docker container if you use the recommended setup so the “Remote arbitrary code execution” is inside of a docker container so you would then need to escape that. It’s still definetley a bad thing but in these sorts of cases docker while not make it secure adds a layer of security which can help while these exploits are patched
I abandoned NPM. they suddenly introduced a retrocompatibility issue for Arm7. It was never solved, even it was reported by many. It was a nice project. It's a pity
This is a good discussion. Basically you trade cost for support. It really boils down to that. If you pay a company, they can afford to quit their "day jobs" and fix the bugs that inevitably come up.
This might be another video, but SaaS soluitions also have a similar impact; although, not as bad. I wouldn't say M$ or Google would be in that list, but there are quite a few smaller SaaS companies who are feature driven and not security driven. I've had several run-ins with smaller SaaS (but 'cost-effective') companies who would drag feet, or literally work against you when reporting or pushing for a security fix (even as a paying customer). NPM has somewhat of an excuse because they rely on donations, but it can be frustrating to see a paid solution do this. Also, one 'pro' to consider for self-hosting is you have more freedom to implement mitigations if there's a critical CVE on the loose, with a SaaS, you're at the mercy of the companies willingness to fix it in time.
This helped me reconsider NPM and look at others like Taefik, thank you!
Been meaning to move everything off of NPM and into traefik. This video was the kick I needed. Moved everything over today!
With the exception of homeassistant it's moved over just not resolving lol. Thanks for the great content!!
:edit homeassistant resolves just fine. Just took forever!
Any tips or tutorials, I was going to migrate to it a couple of months ago but I found it overly complex for what I had at the time but since I now have Nextcloud, homeassitant, Vaultwarden and a bunch of other stuff I might slowly migrate to it.
The main thing was that it looked like I needed a lot more work compared to NPM for a simple proxy?
Glad it helped! Thanks a lot for sharing your feedback :)
The problem with traeffic is that it has access to the docker socket. This means a rce vulnability in traeffic = root on host! With NPM the attacker still has to escape the container. This is the main reason I'm sticking with NPM.
Wow thanks Christian!
I am currently using cloudflared tunnel docker on my Synology NAS to connect to a nextcloud and mattermost docker, but I am having issue with SSL, as both nextcloud and mattermost have not certificate.
I will try to use traefik, thanks for the tip!
The statistic show opensource get patch fast on active projects. Kernel get patch with a week, Microsoft left one reported bug 2 years.
I moved from it few weeks ago , you just made me notice more issues . thx for sharing
What is a "small" project? Downloads or installations surely aren't the best indicator. Contributors may be a better indication but how many makes a project "big"?
i did the same thing (move from NPM to Traefik) for a similar reason - came across a bug in NPM and it didn't get resolved (don't know if it has since) but am never going back now - Traefik is far superior IMO - good vlog though
once i contacted Jenkins maintainers after trying more than 10 of their container images all with 100s of vulnerabilities,, they simply reply that they do not have a single Jenkins container image without security vulnerabilities
i'd pay for any good tool out there rather than using Jenkins💩
Reminder that any container with docker socket access effectively has zero sandbox. If an attacker can talk to your socket they have *root* access to your entire machine with all the same power you have. That includes kernel capabilities.
Good thing I have NPM inside a container inside a dedicated VM then
my skill level is just good enough to follow along TH-camrs like you are. I installed some Docker containers in MacOS, like Portainer, Pihole, Yellyfin, also tried to install Nextcloud behind Nginx and a self-signed cert, but all for experimenting and learning without expsing them to the Internet. 2 days ago I accidentally stumbled over Docker Scout within Docker Desktop which provides analysis of locally installed Docker images. Result: not a single image without vulnerabilites. Portainer shows 101 vulnerabilities etc. For a non-developer without IT qualification this feels quite concerning. Especially now I am thinking about my Home Assistant installation which is also built on Docker as far as I know
There is no 100% of security. You‘ll have to deal with it. Never put important data on DIY devices with open ports, manual/no updates, or without IT security monitoring and professional support. My home lap is also a result of „half knowledge“ in networking and IT security. So, I trust the most important data to big companies like Apple, because they can’t afford any vulnerabilities at all. With 2FA, encryption and so on.
I would NEVER use a self-hosted vault for my important passwords, as long as Apple‘s keychain will do it better and in an even more secure way.
yesss!!! this is a really good point on this topic. THANKS A LOT Christian!!!!!
these are my thoughts on this:
traefic for external reverse proxy and nginxproxymanager for internal things (as it is easier to configure with a GUI instead of yaml)
Thanks for sharing!! Well said!
I can understand the concerns you are raising, however: Having worked with and around many large enterprise organizations, I can say the people working there are no more competent that those working on small projects. I would say the real issue with small projects is reliable funding. If users don't pay for the use of the projects, the people working on those projects won't be able to devote time to them (they need food, clothing and shelter, just like the rest of us. If working on the project isn't providing those, then they must spend their efforts elsewhere). Large companies can cheat with respect to this, by turning the user into a product. That is the tradeoff you accept for "free stuff".
Yeah it’s not so much about the competence, but about resources and time
Thats interesting I was thinking about exactly the same thing few days ago. I'm actually using this project in homelab too. But the thing about vulnerability is exposure of control panel to the wide web. If Login and password will be exposed and attacker dont have access to your login page/protocol then you are safe. The other thing would be if you would use it on some vps to expose multiple dockers from one ip. Then this is big no no because you are forced to expose login page and some secure solutions need to be used. I actually learned haproxy that dont have any login implemented.
Thanks for sharing your feedback :)
This is another reason to use Kubernetes instead of docker. Many companies are behind the software being used in production. Ingress-nginx is used way more than nginx proxy manager.
The most dangerous parts of the homelab are the ones that can be accessed externally without a login. Your reverse proxy and authentication system, or non-cloud VPN, are probably the most dangerous. Everything else requires a login to be able to access and is less risky.
Just expect these critical services will have 0days.
- Auto-update these services frequently, or have a process to get alerted when there new versions are released
- Choose products written in memory safe languages. (Caddy & Traefik are both good choices)
- Use sandbox techniques (AppArmor/SELinux) to further restrict system access.
- Don't forget to block outbound network access from these services.
The other thing is, nobobdy will write malware for smaller projects. It's just not worth it for a few thousand users, after all you still have to find all servers which run the certain programm.
You make some good points, but big software is not the end all be all. How many libraries out there put out by large companies that are not actively maintained? They may not have any developers assigned.
Hi Christian! Thank you very much for you video! Really good. Regarding password manager, have you tried vaultwarden?
Thanks for this warning.
Treafik is great, but their docs are absolute trash and by this date I still don't know how to properly set up with Docker
But with self hosting i can spin up a bunch of vm's and isolate them for much cheaper than in the cloud. This way they cant spread. This definitely brings up the point to host one service per instance to mitigate the blast radius if something gets compromised. The difference for a vps is its on someone else's hardware and internet connection. Still need to make sure what your running is secure and patched.
Thanks Christian.
You're welcome
Isn't that a frontend gui thing for nginx?
The problemen is more that people make bug report but do not what pay for the project are help out
The problem is also that many people report bugs without providing enough data to reproduce the bug or just entirely ignore issue templates that repo maintainers set up and just write random things which are not helpful. If people that reported bugs would invest some effort into bug reports, they potentially wouldn't be such a tedious thing to go through and resolve.
I do understand that project maintainers are burned out in general and all the reported low effort crap they have to go through doesn't help at all.
Thanks for sharing this. Is Caddy more secure than NPM?
Ha, the OSS project I'm a part of has almost 500 open issues 😂
People need to get over themselves regarding the issue count. Security-related bugs are fixed immediately, everything else is fixed whenever someone feels like it.
Proxy manager is a time saver but I would never consider it for production as there are many small details that nginx need to be adapted that the nice little gui has.
I totally apreciate the proxy manager even exists and its free!! they should charge for it and fix issues :)
It was ok npm, but i merged to pfsense and haproxy, faster and more reliable
Christian, thank you. I have tried nginx and use it internally in the lab... this makes me reconsider using it altogether. It was an easy tool but i guess that comes with consequences.
You're welcome! :)
To be clear. nginx wasn't impacted here. It is an unrelated project "nginx proxy manager" that had the flaw.
dude has a wild take on all the hate for any project not faang managed
Thanks for the great channel.
I'm just starting out with a Linux server (Ubuntu) running off a laptop for my "home lab". If you were to restart with roughly $2000 - $3000, what would you go for? It would mostly be test environments, but could potentially host initial production databases and applications. Could you do a vlog on this, unless the solution is stupidly straight-forward? Thanks.
Something like nginx proxy manager seems more like an internal tool, and it probably shouldn't be exposed publicly anyway
Thank you so much for this. and i have believed that it is safe so after this, you have started to doubt.
unfortunately, it's the only thing I've found that is user-friendly. traefik also talked a lot about but it is light years for me to understand how to do with it. Do you know any other alternative with ui that you would recommend and maybe even make a video about.
I have no problem with using Nginx Proxy Manager for my playground.
But for something more important, i'm a lot more picky.
Dang, I just started using this, maybe I'll switch back to plain ol' nginx then.
If you want a generic gateway, that's probably a vpn. I know VPNs have come in for a lot of hate, but running strong security at the application layer is so much harder, as you have to secure every application, and applications focus on features, not security.
I am waiting for your video on cloudpanel that can also manage nginx. I found it recently and wondering how it works?
Interresting.
I disagree with the point of view that self hosting is less secure than using a large service because the large service is a MUCH bigger target and will be attacked far more often.
However, the inherent design of what's evolved in the self hosting space (a large number of tools exposing numerous web interfaces) is completely idiotic as all of those interfaces creates a huge attack surface. There is a massive history of exploits in web interfaces.
This really should be consolidated into ONE interface, with one entry point that can be throughly tested.
That’s not what I said. I said when you’re selfhosting it’s better not to throw each and every small project in your stack without researching how well it’s maintained.
Find different project like NPM that has a good GUI - so its veeery easy to manage without digging into config files - potentially exposing more then you want xD NPM is just easy to learn xD in fact WAF is a good way to secure external assets. Layer7 is needed today.
Looks like my decision to switch from Nginx Proxy Manager to a manually set up HAProxy instance running in a jail container has finally paid off.
Did it tho? The CVE for nginx proxy manager was rather minor as it required admin access to the service and then that said admin doing the attack. This is a problem for shared hosting for sure, but not really for self hosting scenario as you already have way bigger rights. Either way, something that should be fixed. But HAProxy actually has had critical level of CVE's. Looks like jailed instance of nginx proxy manager would have been safer option if past CVE's is what we care about.
I am using NGPM, what would you suggest as a better alternative ?
Traefik! :D
To start with, I like most of your videos. They're informative. This one though... Sorry. We should all be security focused, and each piece of software you expose to the internet is an attack vector, we can't deny that.
That being said, you fail to mention a few things:
1) the biggest use case for NPM - was the CVE on the biggest use case? I don't really think so.
2) more than 1000 issues open, yeah true, but the vast majority aren't security related.
3) using traefik in homelab scenarios is like using RHEL these days... you'll be booted, you just don't know when.
Try doing a video on HAProxy for example. Much more useful than that...
dont sleep on left side. sleep on the back instead. or on the right. your moth is geting toward left.
One should be looking at the complexity of the codebase and NOT the size of the teem developing it. Bigger codebases are inherently more difficult to secure as more code = larger attack surface. A small teem developing a simple tool should be perfectly fine.
The point is that small teams mostly lack the skills and experience how to tackle security problems. Bigger projects aren’t protected against security vulnerabilities, but the chances are higher there are people and processes in place for handling security.
Unpopular opinion: This is inducing paranoia in people who don't really care to read about the vulnerabilities and understand them. As if using big companies are usually not patient zero. Just wait long enough until your business dies the next time aws goes down. And BTW big companies rely on small projects en mass. Self hosting is the most important gate for newbies to learn. As for alternatives to NPM - Traefik specifically has been proven for over 4 years to be missing fundamental core features that require plugins or some honestly stupid workaround to maybe resolve the missing features problems. The documentation is atrocious as well. I want to keep the tone positive, so i wont say anything about the way they respond to issues or feature requests. Literally any other proxy/ingress is better than traefik specifically. But trefic feels sponsored man. Lastly - dont ever count on any software to be completely exploit proof - just be smart about how you use it.
Any review in open appsec nginx proxy manager
Kind of misleading title. It should be is: "Nginx Proxy Manager save for self-hosting?"
This is personally, why I use the real deal and not hobbyist's projects that try to make things easier (I'd say that it makes it harder). Less complexity means that CVEs like this are far less likely to happen, it's not very common to have a severe CVE in something like vanilla Nginx.
You're telling me that NGINX is small? Holy hell.
You didn't watch the video, didn't you :D
@@christianlempa I must have missed the part when you said bazinga
I mean yes, I understand that the nginx proxy manager team doesn't have the time to take care of all of the problems of the tool but nobody should let such a critical think as the nginx manager login page exposed to the public internet
One mistake to avoid is that you shouldn't presume that thousands of "reported" bugs or vulnerabilities should be seen as verifiable, reproducable, and mass-exploitable. Many of theses could be one-offs, or just users who have misconfigured something, or are using other poorly engineered software. Also, you can understand how a lot of for-profit companies might want to attack a small FOSS project. One which gives away something for free which the attacker wants to be able to charge people for. So, in an effort to stir up doubt and harm that project's rep, someone might file a bunch of false and very scary sounding reports, all the while deleting all the reports on their own site, and not fixing them. I'm not saying that's the case here, but it's something to take into consideration.
Those are open cases - 1000 of them, and not many have comments on them. Seeing how many cases are open and how few have comments, it looks like the developer(s) are not even looking at the cases. IQ3000 move: 'if you don't look at them then the issues cannot be verified'? Just because the issues have not been looked at doesn't mean all users have made a mistake or did something with a one in a trillion chance of happening again.
I'm not saying all 1000 cases are actual issues because we have no idea. I think it's naive to assume there aren't any issues in the open cases.
@@Shocker99 Did you think that's what I was implying? Because, THAT would be naive.
@@TheChadXperience909 That is exactly what you said. If that is not what you meant then i think you should learn how to better communicate.
Every script, application and service has open and unrealized exploits. Nothing on the internet is ever nor will ever be 100% secure. You should always keep backups. For this very reason. You’re never 100% safe, because where there is a will. There is a way.
Small software projects are best used behind closed networks.
Well to be fair that that nginx proxy manger security issue is a joke. Getting full sys access while being logged in?
I mean comon i can steal all your admin passwords when i'm logged in. They would've probably fixed it quickly if the issue was criticial
I think it's time to switch back to virtualmin nginx proxy
I'm not fully agree with this one.
Because how much of a safety risk is this bug in Nginx?
Second Nginx is not a big program and has a lot lesser code. Than a OS like windows. So te comparison between windows and Nginx and a big company and a small company doesn't say much in my eyes.
And bug what aren't a big big safety risk will be solved on a later time. This happens by big companies as wel.
The best thing in my eyes is popular "Open Source Software" because then everyone can see the source code.
teleport .... 😂
beyond trust PRA is the professionnal level.
Did somebody say Apache?
Cloudflare tunnels anyone
However you define small seems to be the issue here.
Small as in low level of maintainers and resources. NGINX is a large industry standard tool, NPM is not
when i said to you haproxy ... a lot of people laught at me : so continue laughing guys 😂
Dunno y u mentioned Microsoft. Windows is bugless. 😏
Nice shirt. Use a VPN if you have something to hide. 😅
Thx :D
this is the reason why big companies will always have more sales than small ones - everybody is AFRAID the small company will simply vanish. Same for projects
Unless you're Google, then the project probably won't reach it's first birthday.
Don't expose services to the internet, keep it in your home network, easier to manage that and much safer.
1st
In my experience, home-lab enthusiasts have better security than enterprise implementations. I worked for companies that had RSA hard-tokens on web-cams accessible to the Internet. I could go on and on.. nginx, docker, proxmox are not small tools, just small usage footprints in our hobby.
Love the focus on security with this video. With all the threats and constantly emerging vulnerabilities, it's a moving target and easy to forget the threat is real. Hope to see more videos touching on security.
Thank you :)