I'm so glad you prioritize on value of content over time or usual pleasing algorithm BS... This was a phenomenal video. I can't wait to play with it...
14:25 Absolutely never ever do that. There's literally never a reason to do that. RFC1918 is there for a reason. Now if something on the internet happened to be in that range, you wouldn't be able to access it. Even if currently it's random Taiwanese ISP probably handling end-customers, that could easily change, perfect example being 1.1.1.1, which people used in the past for internal traffic. Also, very ironic of you to say "ipv4 is a mess" and then making it even worse of a mess.
Nice video. I've been using this outside of work myself for ad-hoc groups where we want to share each others systems for development / research and it is pretty handy. Great presentation.
That keyboard needs a track pointer. Track pointer are so easy to use, your hands never need to leave the keyboard. I have one on my laptop and I love it. When I get back to my desktop with a keyboard without a trackpointer, I find my index finger wiggling looking for the trackpointer.
Hey apalrd! Noticed a small issue on your blog post for the video. The video link on there is incorrect and leads to the “This video isn’t available anymore” page on TH-cam. Don’t want future blog-exclusive readers going to the wrong place! Anyway, I’ve just recently discovered your channel and it’s a gold mine. You keep doing you! I appreciate you spreading the knowledge.
you should have a look at pritunl zero, just to do a bit of a review on an alternate zero trust solution. Also, while this establishes trust between hosts across private/public boundaries (which is great), it doesn't actually solve the problem of the example you gave in regards to a trusted host getting a virus. Regardless of the network being behind a firewall, or each host being behind a firewall in a zero trust situation, once they are all trusted they are all equally vulnerable. It's actually the ACLs you put in place which begin to reduce the vulnerability of other clients - but that's exactly what should be used in a private trusted network - ACL's either network based (on hardware or host OS) or user based using AD and Radius to define permissions to access files/folders/sites/ports/wifi networks so on and so forth. So in short the second A in the AAA security model is still required regardless of way the first A is handled - just wouldn't want people to be lulled into a false sense of security =)
Very true - The big advantage of this type of zero-trust system is that the ACLs can be written much more narrowly since they are applied on every single device and have (cert signed) knowledge of the node's identity, not just the IP address range. So this removes trust in the network IP address (which requires more physical security) as a source of identity. But you do still have to write good ACLs.
Awesome video. Is it possible to setup nebula to make it work as an exit node where all traffic from iphone is routed and exited through the server in digital ocean for example. ??
I don't have multiple systems with >1G currently setup to test, but I was able to iperf at 923Mbits without and 880Mbits with Nebula, which is expected due to the lower MTU. So basically it's not limited by anything but the 1G wire on my test setup. That of course doesn't mean it will do 10G or higher, but it's certainly performant enough for all of my Proxmox systems.
awesome video. Is there a mobile app for nebula? I see a Chinese iOS app, ‘Nebula Mesh’ along with the Official ‘Mobile Nebula’. Wondering how the handshake would affect battery life/power consumption when the device goes to sleep. Usually this isn’t a problem with the WireGuard app, although I have noticed increased usage with the Tailscale app.
Mobile Nebula by Defined Networking is the one. No idea on battery life, but it shouldn't keep a connection open unless there are packets periodically going across the tunnel.
When I looked at Nebula a long time ago, it was when relaying was not a thing yet. I have since settled with headscale. I actually like headscale a lot more, but nebula looks really good now. I might try it out. The cool thing about headscale is that you sign in on the node one time, and the server takes care of pushing routes, DNS, ACLs, etc. Nebula seems to be something that has to be managed at scale through something like ansible.
Yeah, the lower load on the lighthouse and delegated relaying also scales up better at really big setups. Tailscale has a lower barrier to entry for most people. The Nebula lighthouse also has no private data at all, so a compromised lighthouse just means the network stops working, security isn't broken.
Okay - yes! Great video. As always, extremely thorough and helpful. Now, what is that blue, silver & black rocket on the right side of your desk near your case?
What a great tool. Let’s assume I have multiple hosts in the cloud which are all in one nebula network. How the traffic gets routed when I connect from one host to another on the same cloud, via the internet? Is it possible for these kind of connections to use a private cloud network to not route traffic via the internet?
A given Nebula host learns all of its IP addresses from all interfaces. It then connects to the lighthouse(s) and gives them all of the IP addresses it has learned. The lighthouse then knows all of those IPs plus the IP the lighthouse received (if the node goes through NAT on the way out) and shares this with a node wishing to connect. The sending node will try all of the receiving node's addresses in order of similarity to its own addresses, so if they are on a private cloud network they will use that. But if you use IPv6 it doesn't matter since IP will always route most efficiently without any NAT or overlapping address ranges on different networks.
Nice video, but I have question... When you get a public query for spaceship3 and the lighthouse sends the ip of spaceship3, shouldn't the lighthouse check if the client has a valid nebula certificate?
DNS queried over the public network doesn’t require any authentication, dns doesn’t really have any facilities for that. The lighthouse also doesn’t have a database of all clients, it only has a table of clients that have recently established sessions with it, and that’s what it uses for it’s dns query. So if spaceship3 doesn’t have a valid certificate it won’t end up in that table.
Will you do an update video on how to automate the renewal of the certificates? Also, why is the certificate valid for so long (1 year compared to 1 day as smallstep ca does by default). After digging into setting up small step ca, I am still confused how zero trust could work during the renewal of certificates. Because of how I understand how smallstep CA does the http-01 ACME validation, it basically must trust the DNS servers and the routers/network between the CA, DNS server and the client who want to renew its certificate. This does not really seem like zero trust. Are there better ways of doing it in small step/Nebula? Would the ultimate solution be something like device attestation using a YubiKey where the CA has a database of your fleet of devices YubiKey public attestation keys? This is the solution that sounds the most like zero trust to me.
So in general, the danger is that a certificate and it's private key are compromised. The two ways to avoid this are to use hardware-bound keys (Yubikey, TPM2, ...) or continuously re-key everything so a compromised key doesn't have as long of a useful life. This applies to all keys, TLS, Nebula, and SSH. Since the ACME protocol is entirely automated, we can use a really short interval (24 hours) and there won't be any client interruption. With Nebula, for servers we could automate re-keying at that interval, but since clients also use the crypto system and might not be powered on for 24 hours that's a bit too short. So we need a compromise. The Nebula defaults are 1 year, which is decent if you don't have any automation to re-key. It's just a default though. Reading through the weeds a bit, it seems like Slack has a CA which they use to issue minimal-access provisional certs to machines as they go through the automated first boot setup which has enough access for the re-keying automation to come in and replace the key with the real one when it fully sets up the server with a role. They haven't said how they do attestation to re-key. In general you are right and the CA does need to trust something to issue certs. 'Zero-trust' refers more to the individual connections, in which a node trusts nothing other than the root certificate it's told to trust, and then builds trust in other nodes via keys which have been signed by the root. How you attest to the CA at re-key time is still not perfect. You could trust the existing nebula cert to renew the same one, for example. And finally, Smallstep has a provisioner which uses Nebula certs as a means of trust to issue x.509 and SSH certs.
This piece of software feels like: almost right, no IPv6 for the internal network and no yubikey auth for the node..? Which is something the commercial companies do seem to do.
On which node? In general unless you have open security vulnerabilities or an extremely fast internet connection you will saturate the internet connection before there's any problems with the host (lighthouse or otherwise), although this would have happened anyway with any other network architecture using the same connection.
Tailscale also centralizes all of the trust at the tailscale service or headscale server, since it's responsible for all of the ACLs and pushing config to clients.
@@apalrdsadventures This is a good thing (and sometimes required) depending on your zero trust implementation. Besides easier management due to the centralization, the user *and* device identity are validated before connecting unlike Nebula which only validates the device. The user identity part is an extremely common thing that comes up during zero trust discussions which is probably why Nebula isn't advertised as such.
The Tailscale client device can't check a cert for validity, it can only trust that the auth server was correct. This is very centralized trust. Nebula centralizes management at the certificate authority, but once a cert is signed it remains valid. This means any node can now validate a certificate on its own, without relying on the central server. The PKI part isn't done internal to Nebula, so it's not a complete solution for all use cases. It's advertised for server to server zero-trust that can scale up to tens of thousands of nodes, which is something that's hard to do when you make a centralized node key to every decision.
@apalrdsadventures Yup, all depends on the implementation requirements, this one being access vs datacenter/server networks. I said not advertised because looking at their website and docs I only see "overlay networking" referenced, not zero trust (unless it's hidden on a page I'm not seeing). Tho it very much is a server based zero-trust model
It's the answer to life, the universe, and everything But it's also allocated (to a Taiwanese ISP) but not advertised over BGP so it won't route on the public internet either (at least for now).
you cannot just use an IPv4 randomly, 192.168.0.0/16 and 10.0.0.0/8 are there for a reason. calling the IPv4 a mess is not a good reason to make it more messy.
I'm so glad you prioritize on value of content over time or usual pleasing algorithm BS...
This was a phenomenal video. I can't wait to play with it...
14:25 Absolutely never ever do that. There's literally never a reason to do that.
RFC1918 is there for a reason. Now if something on the internet happened to be in that range, you wouldn't be able to access it. Even if currently it's random Taiwanese ISP probably handling end-customers, that could easily change, perfect example being 1.1.1.1, which people used in the past for internal traffic.
Also, very ironic of you to say "ipv4 is a mess" and then making it even worse of a mess.
+1
Nice video. I've been using this outside of work myself for ad-hoc groups where we want to share each others systems for development / research and it is pretty handy. Great presentation.
That keyboard needs a track pointer. Track pointer are so easy to use, your hands never need to leave the keyboard. I have one on my laptop and I love it. When I get back to my desktop with a keyboard without a trackpointer, I find my index finger wiggling looking for the trackpointer.
Just awesome! Been following for a while now, the subjects just keep improving!
Glad you like them!
Wow! Awesome foldable keyboard in advertising! Thanks for recomendation!
I really appreciated this video. Please continue with this type of content. Thanks you.
thanks, you explain it all so well and in extensive detail
this channel is a goldmine
Thanks!
Hey apalrd! Noticed a small issue on your blog post for the video. The video link on there is incorrect and leads to the “This video isn’t available anymore” page on TH-cam. Don’t want future blog-exclusive readers going to the wrong place!
Anyway, I’ve just recently discovered your channel and it’s a gold mine. You keep doing you! I appreciate you spreading the knowledge.
Thanks! Fixed
Thanks for the demo and info, have a great day
Glad you liked it!
So... instead of a small chance of a RFC1918 address conflict... Let's break a /16 worth of internet routable addresses. Yolo.
Yeah but it's 42.69 so it's fine :)
Disrespect RFC1918, steal a whole /16 for yourself. Be a man.
you should have a look at pritunl zero, just to do a bit of a review on an alternate zero trust solution. Also, while this establishes trust between hosts across private/public boundaries (which is great), it doesn't actually solve the problem of the example you gave in regards to a trusted host getting a virus. Regardless of the network being behind a firewall, or each host being behind a firewall in a zero trust situation, once they are all trusted they are all equally vulnerable. It's actually the ACLs you put in place which begin to reduce the vulnerability of other clients - but that's exactly what should be used in a private trusted network - ACL's either network based (on hardware or host OS) or user based using AD and Radius to define permissions to access files/folders/sites/ports/wifi networks so on and so forth. So in short the second A in the AAA security model is still required regardless of way the first A is handled - just wouldn't want people to be lulled into a false sense of security =)
Very true - The big advantage of this type of zero-trust system is that the ACLs can be written much more narrowly since they are applied on every single device and have (cert signed) knowledge of the node's identity, not just the IP address range. So this removes trust in the network IP address (which requires more physical security) as a source of identity.
But you do still have to write good ACLs.
Awesome video. Is it possible to setup nebula to make it work as an exit node where all traffic from iphone is routed and exited through the server in digital ocean for example. ??
First of all, great video! I really like the way you explain things. How is the throughput performance on Nebula?
I don't have multiple systems with >1G currently setup to test, but I was able to iperf at 923Mbits without and 880Mbits with Nebula, which is expected due to the lower MTU. So basically it's not limited by anything but the 1G wire on my test setup. That of course doesn't mean it will do 10G or higher, but it's certainly performant enough for all of my Proxmox systems.
awesome video. Is there a mobile app for nebula? I see a Chinese iOS app, ‘Nebula Mesh’ along with the Official ‘Mobile Nebula’. Wondering how the handshake would affect battery life/power consumption when the device goes to sleep. Usually this isn’t a problem with the WireGuard app, although I have noticed increased usage with the Tailscale app.
Mobile Nebula by Defined Networking is the one. No idea on battery life, but it shouldn't keep a connection open unless there are packets periodically going across the tunnel.
When I looked at Nebula a long time ago, it was when relaying was not a thing yet. I have since settled with headscale. I actually like headscale a lot more, but nebula looks really good now. I might try it out. The cool thing about headscale is that you sign in on the node one time, and the server takes care of pushing routes, DNS, ACLs, etc. Nebula seems to be something that has to be managed at scale through something like ansible.
I'm using headscale as well and love it.
There are some architectural differences, so the headscale server does a whole lot more work in the setup than Nebula's lighthouse.
@@apalrdsadventures This is true. Sort of a different use case. Nebula is a bit more zero-trust than headscale.
Yeah, the lower load on the lighthouse and delegated relaying also scales up better at really big setups. Tailscale has a lower barrier to entry for most people. The Nebula lighthouse also has no private data at all, so a compromised lighthouse just means the network stops working, security isn't broken.
Okay - yes! Great video. As always, extremely thorough and helpful. Now, what is that blue, silver & black rocket on the right side of your desk near your case?
it's a 4 color pen lol each of the fins push down separately to activate one color
It's a flesh light.
What a great tool. Let’s assume I have multiple hosts in the cloud which are all in one nebula network. How the traffic gets routed when I connect from one host to another on the same cloud, via the internet? Is it possible for these kind of connections to use a private cloud network to not route traffic via the internet?
A given Nebula host learns all of its IP addresses from all interfaces. It then connects to the lighthouse(s) and gives them all of the IP addresses it has learned. The lighthouse then knows all of those IPs plus the IP the lighthouse received (if the node goes through NAT on the way out) and shares this with a node wishing to connect. The sending node will try all of the receiving node's addresses in order of similarity to its own addresses, so if they are on a private cloud network they will use that.
But if you use IPv6 it doesn't matter since IP will always route most efficiently without any NAT or overlapping address ranges on different networks.
Nebula is nice. Your demo was great. I have been using Nebula for a few years and it has been great.
Any risk of conflict with an active directory? How would DNS be setup in that case?
I don't use Windows so I'm not an expert on this.
all of this is cool and stuff but... YOUR CAT!?!? AWWWW :3
How do you add snmp devices to the nebula?
The neckbeard is strong on this one.
In my scenario the only solution to solve the NAT problem resolved by SoftEtherVPN secureNAT feature, how can I achieve it using Nebula?
Nebula can use relays, which are nodes not behind NAT. You can configure a node to advertise a relay for itself.
Nice video, but I have question...
When you get a public query for spaceship3 and the lighthouse sends the ip of spaceship3, shouldn't the lighthouse check if the client has a valid nebula certificate?
DNS queried over the public network doesn’t require any authentication, dns doesn’t really have any facilities for that. The lighthouse also doesn’t have a database of all clients, it only has a table of clients that have recently established sessions with it, and that’s what it uses for it’s dns query. So if spaceship3 doesn’t have a valid certificate it won’t end up in that table.
Will you do an update video on how to automate the renewal of the certificates? Also, why is the certificate valid for so long (1 year compared to 1 day as smallstep ca does by default). After digging into setting up small step ca, I am still confused how zero trust could work during the renewal of certificates. Because of how I understand how smallstep CA does the http-01 ACME validation, it basically must trust the DNS servers and the routers/network between the CA, DNS server and the client who want to renew its certificate. This does not really seem like zero trust. Are there better ways of doing it in small step/Nebula? Would the ultimate solution be something like device attestation using a YubiKey where the CA has a database of your fleet of devices YubiKey public attestation keys? This is the solution that sounds the most like zero trust to me.
So in general, the danger is that a certificate and it's private key are compromised. The two ways to avoid this are to use hardware-bound keys (Yubikey, TPM2, ...) or continuously re-key everything so a compromised key doesn't have as long of a useful life. This applies to all keys, TLS, Nebula, and SSH.
Since the ACME protocol is entirely automated, we can use a really short interval (24 hours) and there won't be any client interruption. With Nebula, for servers we could automate re-keying at that interval, but since clients also use the crypto system and might not be powered on for 24 hours that's a bit too short. So we need a compromise. The Nebula defaults are 1 year, which is decent if you don't have any automation to re-key. It's just a default though.
Reading through the weeds a bit, it seems like Slack has a CA which they use to issue minimal-access provisional certs to machines as they go through the automated first boot setup which has enough access for the re-keying automation to come in and replace the key with the real one when it fully sets up the server with a role. They haven't said how they do attestation to re-key.
In general you are right and the CA does need to trust something to issue certs. 'Zero-trust' refers more to the individual connections, in which a node trusts nothing other than the root certificate it's told to trust, and then builds trust in other nodes via keys which have been signed by the root. How you attest to the CA at re-key time is still not perfect. You could trust the existing nebula cert to renew the same one, for example.
And finally, Smallstep has a provisioner which uses Nebula certs as a means of trust to issue x.509 and SSH certs.
19:25 cat
producer
Do you know openziti ?
Wait 42.69.*.* is used on the internet?
This piece of software feels like: almost right, no IPv6 for the internal network and no yubikey auth for the node..? Which is something the commercial companies do seem to do.
What's about DDoS attach?
On which node?
In general unless you have open security vulnerabilities or an extremely fast internet connection you will saturate the internet connection before there's any problems with the host (lighthouse or otherwise), although this would have happened anyway with any other network architecture using the same connection.
You can set how long you want the CA cert to live. Defaut is 1 year.
That is the god damn cutest cat! The video is alright too /s
Jk, thanks for the great content, never heart of Zero Trust Networks before.
Just as a heads up tailscale works the same way. Each system has tailscale installed on it.
Sort of, however you can use it to "break" the model and use subnet routers, which removes the "trust" part of it.
Tailscale also centralizes all of the trust at the tailscale service or headscale server, since it's responsible for all of the ACLs and pushing config to clients.
@@apalrdsadventures This is a good thing (and sometimes required) depending on your zero trust implementation. Besides easier management due to the centralization, the user *and* device identity are validated before connecting unlike Nebula which only validates the device. The user identity part is an extremely common thing that comes up during zero trust discussions which is probably why Nebula isn't advertised as such.
The Tailscale client device can't check a cert for validity, it can only trust that the auth server was correct. This is very centralized trust.
Nebula centralizes management at the certificate authority, but once a cert is signed it remains valid. This means any node can now validate a certificate on its own, without relying on the central server. The PKI part isn't done internal to Nebula, so it's not a complete solution for all use cases. It's advertised for server to server zero-trust that can scale up to tens of thousands of nodes, which is something that's hard to do when you make a centralized node key to every decision.
@apalrdsadventures Yup, all depends on the implementation requirements, this one being access vs datacenter/server networks.
I said not advertised because looking at their website and docs I only see "overlay networking" referenced, not zero trust (unless it's hidden on a page I'm not seeing). Tho it very much is a server based zero-trust model
I'm envious of your brain
Chinese IP range... I like it.
It's the answer to life, the universe, and everything
But it's also allocated (to a Taiwanese ISP) but not advertised over BGP so it won't route on the public internet either (at least for now).
you cannot just use an IPv4 randomly, 192.168.0.0/16 and 10.0.0.0/8 are there for a reason. calling the IPv4 a mess is not a good reason to make it more messy.
*attack
@apalrdsadventures not able to ping from host to lighthouse, I followed everything.
anon p2p ai would be more interesting while using pki/qkd you should add another monitor