Bro can you please tell me how much time it takes to learn assembly language from which you can write your own script exploits and malwares and ransomwares. And which are the best high level languages which are used to write a malware or ransomwares
@@PrinceKumar-yo9lr It really depends on what you're exploiting. If it's a local vulnerability (one through the operating system) then you'll need to have full control over the arguments you pass to the operating system's APIs. This is best done with a low level language such as C, C++, Zig, or even Rust. If you need *really* fine control, you can use Assembly, but it's not common. However, must vulnerabilities are rooted in networking. These can be made in any language that has networking capabilities. Rust is my pick, just because of how easy it is to throw some bytes through a socket. POSIX's C sockets take a bit more work to set up than a higher-level TCP stream, but it's way better than Winsock from what I've seen. However, this all assumes you have a vulnerability to exploit. Vulnerabilities are often patched as soon as they pop up. Something way cooler than just malicious stuff is "fun" malware, which was away more prevalent back then. It didn't usually do anything super serious, but it would take control of the system and show you something really cool looking. There are some Windows 10 programs that do this with WinGDI (see MEMZ and Chloroform). You could learn to do the same, since it looks really cool. You don't need to exploit anything to make something really cool, just look at Furnace.
There has to be an invisible hand from the intelligence community to plant some of these. I'm not saying that programmers never make mistakes that allow sploits, but those are probably the exception, not the rule.
@@brainites by definition, or they wouldn't be breakers. Sometimes builders can be ahead, as is the case with attempts at quantum computer proof crypto that is being worked on before quantum supremacy is reached.
@@AkivaB Guess it's difficult to maintain the wireguard configuration for all your devices, especially for multiple users - personally I like to use an open source mesh VPN like Tailscale or ZeroTier.
I think at this point we can update the saying to "the three hardest problems in computer science are cache invalidation, naming things, asynchronous programs and 'Off By 1' errors"
Bugs like this are part of why I use a pretty aggressive fail2ban. The attacker doesn't get 10,000 tries... instead they get 3 tries or sometimes even less. The bans eventually expire, but instead of hours to get in, it would take decades. Plenty of time to install a fixed version.
You can get nailed on the first try if you're unlucky, or the timing might never work for an attacker. Even 64 bit systems could get catastrophically unlucky. At least it's an easy fix this time.
@@parad0xheartI'm not sure about fail2ban specifically, but it's standard to block the whole /64 range for IPv6. Each customer / network is supposed to get its own /64, so it makes sense to block the entire range.
very well explained. i love that the vulnerability is put under real word context and report is not just a scary click bait. if one has a cloud server e.g. amazon, they should limit their client IP address for that ssh port.
Is that the recommended method? I also always thought It would be risky to use an ssh server outside my home network. But don't know what to do instead. What if there is a coffee shop with the same provider and open wifi nearby. Wouldn't they also have the same IP? Of course it would still be a lot harder to hack the server than.
@@leok42 , your ISP typically assigns an unique but non-permanent address for each location. so your server would see different ip address at your coffee shop vs your home address unless you tunnel through your home address.
@@leok42 Usually, because of the lack of ipv4 addresses - the ISP can use a DS-lite tunnel to map multiple customers onto a single ipv4. On this case you will have an ipv6 address which is unique tho. In my situation (Germany): The way you get your internet differs immensly. My contract for my home is as described above (DS-lite + ipv6). Some households only have an ipv4 that changes daily. Sometimes they only have an ipv4 that only changes every ~month.
Yeah Ubuntu already patched it up on July 1st openssh 1:9.6p1-3ubuntu13.3 CVE-2024-6387 Edit: From the bug report itself 2024-05-19: We contacted OpenSSH's developers. Successive iterations of patches and patch reviews followed. 2024-06-20: We contacted the distros@openwall. 2024-07-01: Coordinated Release Date.
Your explaination for laypersons is very very good. I'm not a programmer or security expert by any means, but found it was easy to comprehend thanks to your summary
This has all my windows people at work scream LINUX VIRUS and im so exhausted of telling them it would take literal hours and using fail2ban is a dead simple mitigation any public server should have anyway. Ugh... That said, this explanation was really good! Reminds me of the late Tetris level shenanigans where VBlank interrupts cause almost the same situation - albeit of a different nature.
This is also more exploitable as the paper mentioned on 32-bit CPUs... which in 2024, who is seriously even using 32-bit for anything, let alone a server on the Internet for anything productive? So, this is essentially a very minor issue in my eyes and shouldn't affect that many people or servers.
...i have heard whispers & jokes of "Linux" & "packet sniffing": But they're so busy laughing that I cannot understand what they're saying... Can you comment on this at all ?
Great job explaining this vulnerability. But I think you got the LoginGraceTime part wrong. According to sshd_config's man page: "The server disconnects after this time if the user has not successfully logged in. If the value is 0, there is no time limit." - Which could result in a DoS if the maximum unauthorized connections are exhausted.
10:51 It does not close it immediately but rather does not close it at all. That's why as researchers mention it make you vulnerable to dos attacks as attacker does not have time limit for spawning too many waiting logins
I don't personally like your implied criticism of open source software twards the end of these kinds of videos. While I understand being cautious, it makes it kinda feel like its somehow a bad solution to an other wise worse alternative. Personally I think instilling fear in something that has been the better choice in security since the dawn of the internet is not a good idea. I do agree that its not perfect, but until theres an objectively better option, I would prefer that you didn't make it sound as if the world is going to collapse because we rely on the better of our options in software security.
@@gavabundo_0072 This vulnerability is about signal safety - that is a whole other level of safety that Rust does not provide. When the signal handler is invoked in this exploit, the heap is corrupt. If you do anything with the heap at that point, you are bound to have something exploitable and a signal handler is Rust CAN interact with the heap.
I sent a similar video to someone at my office. He's like: updating the libraries now. We then talked about the importance of testing known weak points in code (since it was a regression). Gotta keep an eye on known previous points of failure.
I wanted to touch on something you noted late in the video, regarding recommending not exposing SSH on the internet, which invites the question of what do you suggest instead? You can do a lot to try and isolate management networks/etc, but ultimately you need a legitimate way in. Your argument that 'code can have bugs' applies to pretty much anything, we've seen various firewall vendor and VPN bugs in the past, so they're not different. How would you handle remote access?
@@LowLevelTV I can agree with you on that. Sometimes that presents a practicality problem, but it does significantly improve the posture when possible. And then in the case of this particular bug, something like fail2ban would probably go a long way in mitigation (though not closing off the bug entirely), given the large number of tries required. Thanks as always for the great content!
Run SSH on a non-standard port, use fail2ban, or limit what IP blocks you allow to access (if you're in the US, do you need to allow access from other continents?). For big organizations that have their own IPv4 blocks they got from a RIR it's super easy, you just only allow from your own IP blocks and reject everything else
I think he was referring to don't connect it to the internet while using the vulnerable version, not don't use SSH for its intended purpose ever. If that's what he did mean, then there's a couple of things you can do like whitelisting only specific IPs, or port knocking, but these only reduce the attack surface not make it safe. IMO its worth the risk if you take proper cautions like, IP address whitelisting, but not using a tool just because there's a possibility it could be vulnerable is dumb.
This sounds like an early implementation of a TAS speed run with a wrong warp. It sounds impossible to execute but determined people can make these issues exploitable at a moment’s notice.
I mean yeah you're right, this isn't the kind of exploit to some random individual is going to use to hack into a bunch of servers. But for extremely sophisticated, targeted attacks, stuff like this can be and is exploited.
Great coverage on the subject when everyone else is screaming everything could be on fire. Seriously though big points to reviewing the mitigations and explaining the exploit in a easy to consume video!
@@somebodystealsmyname Establishing SSH connections costs very little bandwidth. Depending on the exact timing, AWS may not be enough. But a small host with good connectivity to your target ranges, which can be established with a BGP looking glass, and many of these have very limited to no KYC -- those are great for these attacks
already covered in the video. OpenSSH throttles new connections to... 100 in a second? which is why it takes 3-4 hours based on how quickly it allows connections to come in.
For critical projects like this (at the very least), there should be a process built into the commit procedure that checks for various types of vulnerabilities, and especially for specific vulnerabilities that were previously found and patched.
Ed, you are an amazing researcher and very smart guy, if I may, do you have any attack vectors that you came up on any of those situations that show us? Just asking,
Thx for ur efforts to do this video, one simple question, wouldn’t be awesome if the developers of OpenSSH project rewrite most of their base code using Rust language! Due to enormous hype about it, in which of its core features eliminating race conditions and other memory faulty stuff?!
On one hand its insane that syslog caused this vulnerability. This is probably an issue with other software as well, given using syslog is rather common. But on the other hand, this only exist if the attacker manages to time a timeout-timer between two lines inside a malloc call. They say mallocing heap is slow but its pretty fast compared to that. Just insane work by the redearchers who made this real.
@somebodystealsmyname actually kinda incorrect the problem is much more deeper and fundamental to how operating systems manage running processes. "Process Interrupt Race" style bugs are old since 2001 at least they were known but are gaining a resurgence again. There just much more rare in single process async code scenarios because other then signal interrupts there's not much code that can overlap or interleave at the time the process can be interrupted. I'm sure there's other bug variations though where a process works on shared data rather then your typical threads and interrupting the process during specific instructions will leave say a DBMS in an inconsistent state. We could see flaws like this on phps zend interpreter process and maybe a reawakening of maybe not SQL injections but new forms of IDORs and other wild bypasses that stem from interrupting execution flow of a process. Even in processes without Reentrancy and even some processes that run mostly Synchronize code I could see just interrupting a process causing issues while another process acts on the same shared state like cart data of an e-commerce site. But as for single process async overlap stuff to mess with heaps like in this openSSH stuff nothing comes to mind other then signal handlers and maybe system calls which causes the kernal to block a process temporarily.
Same! I learnt more about `ssh` and `tmux` JUST YESTERDAY and now I get to watch this! ... Thank you, Ed. At least I know how to keep my `ssh` connections more secure _nauw!..._
I wonder how long this exploit has been known and if it is being actively utilized in the wild. My server has been getting hammered hard since the end of June. I was getting over 2,000 access attempts a day for a while. Up from just a few hundred a week.
I am a little bit perplexed. Firstly, removing openssh from the Internet may be an option for some, but many hosting providers actually allow clients to use ssh to update their files on the host. For them to remove ssh from the Internet would mean that clients have lost the ability to upload files to their host (or at least have lost that path by which they can upload). Secondly, if this problem is in malloc(), the I would think that potentially it has far wider issues than merely OpenSSH, malloc() is a core part of memory management on most systems (even if it is behind the scenes), and it would seem very probable that there are other pieces of software out there that allow signal interrupts while using malloc().
Use sshv2 straight up instead of using unsecure free replacement programs. On top of that, dont allow direct internet access to the ssh port of those servers. You could easily implement a vpn that could provide access to whatever management is needed securely
@@lillones none of these alternatives are guaranteed to be any safer than OpenSSH. If the problem is the way malloc() interacts with signals then there is no reason to be so certain that a VPN would not have the same issue. All that a closed source alternative to OpenSSH would give you is the bliss of ignorance - you same problems might be there but it would take a lot longer to find them.
The vpn would allow you to send requests for ssh internally on the intranet rather than the open internet. It would block any rando from being able to even attempt to initiate a session to take advantage of the vulnerability
This can be mitigated with a trivial change to a config file It takes concerted effort over time on 32-bit It probably isn't possible on 64-bit due to planning to prevent these things It's been fixed and an update has been released It's never been proven to exist in the wild. So, not losing any sleep AT ALL over this one. Academically fun, but not something that rising to any concern whatsoever.
I'm missing something... the attacker tries to log in and waits for the grace period to expire, and they need it to expire right when malloc is being called so that heap memory is corrupted. But how exactly is the attacker injecting code? How do they send the malicious code to the machine running ssh, and then how do they exploit the heap corruption to shove their code into a place where it can be executed? They're not providing a custom sigalarm handler are they? That seems like it would be too easy.
I believe the code is present in their certificate or some other part of the SSH connection request. They time sending some portion of their request so that it is present in the heap with the signal handler is invoked. The signal handler then ends up executing what they had inserted in to the heap. The signal handler is one written by OpenSSH to deal with connection timeouts. The signal handler ends up interacting with the heap though, which is attacker controlled and leads to the exploit.
Basically while the OpenSSH "regreSSHion" vulnerability sounds concerning, it's not a major threat. Exploitation is complex and requires hours of attempts under specific conditions, making widespread attacks unlikely. Many systems already have mitigations like brute-force detection in place, and the scope is limited to certain OpenSSH versions. Patch your systems ...no need to panic.
Have you ever made a video on the Intel Management Engine? My professor in my OS class mentioned that every Intel chip has its own OS that is based on Minix3. I've read people call it "ring -3", and so Intel basically has a root kit on every Intel PC. I would be interested to hear your thoughts on this
I love how a serious RCE gets resurfaced because "oops I [un]commented the wrong #ifdef" AKA how often does stuff like this make it into open code bases because no one notices? Who knows if this oops was intentional or not.
None of my servers ever get direct internet access. They are all firewalled off from the internet and I VPN into my local network and then access any of my servers. Never direct SSH or any other method directly into a server over the internet! It's much easier to "secure" and keep a close eye on a single known external connection than a bunch of smaller random entry points all over the network.
Appreciate you spent ~2min of this video explaining what SSH is, (assuming the viewers could never use it) and then spend nothing on explaining what heap or malloc are 😂
"keep SSH off the internet" significantly dings the usefulness of SSH though. people will continue to use SSH as their primary method for accessing VPS instances for the forseeable I think
3:32 Not really C programming, more "I code on top of an OS" programming. It's just that especially scripting languages don't let you interact with them directly. Java for example uses signals to detect when you dereference a null pointer and instead of dying you get a NullPointerException. But you can (and kinda have to) interact with signals in every at least system level programming language.
Noob here. What does it mean "Step 1: Get your SSH off the internet" How then am I supposed to connect to a remote server? Or was that meant generally to keep the SSH port closed if you are not using it?
@@nicebhaalu961 yes, the port should not be open for random people to attempt authentication to, if you can avoid it. Making it accessible only through a VPN or overlay network is one strategy. Using SPA to selectively open the port when you want to connect is another strategy. Basically, add additional layers to mitigate the risk that any single layer fails.
I would argue that the root of this problem is the complexity that signals introduce due to their ability to interrupt critical code. It’s hard for a developer to anticipate every possible place a signal might interrupt execution. A smart developer really should avoid the use of signals for application functionality.
Every time when i see exploit like this i wonder, maybe it is not feasible for remote access, but could be used for privileges escalation for local users and rooting phones, consoles etc.
You missed one of the interesting bits - why it's a regression. OpenBSD implemented a signal-safe version of syslog() so they cleaned up the signal handler to use that. But ports without safe syslog() became vulnerable again because OpenSSH depended on the safe OpenBSD version. It's a fascinating interaction where making things safer paradoxically opened a vulnerability. I don't know what the answer is - everybody should be watching what OpenBSD does and following suit but that takes time to catch up. maybe OpenBSD needs to drastically rename interfaces when they fix them so it's extremely obvious that using the older interface is dangerous? This isn't the first time porting OpenSSH has caused problems, and probably won't be the last. Debian famously fixed an uninitialized memory read "error" and generated the same 65,000 SSH keys on every system.
I do not know much about this topic, but I always wondered what even the advantage of code and data sharing the same memory is? Wouldn't this kind of exploit be impossible if the PC could never point to a block of data?
I still remember the first time I had a server open to the internet the only reason I didn't get hacked is because the logs from their attack filled my 40gb drive and crashed the system. Something like this would have been very bad.
timing here is interesting. would an attack perhaps be exploitable faster with less network latency deviation (e.g. intra-datacenter exploits) i would presume attack could be performed much faster if you knew additional information about where in the cloud your victim is hosted and network link speeds are much more predictable.
So setting grace time to 0 is a fix. I guess because of that, having password authentication disabled also fixes it since it instantly rejects any incorrect private key. Sweet.
I never allow ssh to be open to the public internet anyway. Lock it down. Bind SSH to a private IP or only allow from a specific address/subnet. Then access over VPN.
10:20 So what is the alternative? Of course only having physical access to a server would be the most secure option but it is not practicable. What else can be done apart from using software such as fail2ban and putting the SSH server on a non standard port?
@LowLevelLearning Have they completed the 64bit PoC yet? Last I saw they still only had only successfully exploit in 32-bit. However, they were working on a 64-bit version
10:42 "Don't expose ssh to the internet!" can you elaborate? Do you mean to use it only in a local network? Or not to make the path straight, like an IP or domain pointing directly to your ssh server? Or maybe to gatekeep the connection before reaching the ssh server? Or just not make it public knowledge? Learning security bugs me because security professionals give 4 different meanings to the same exact phrase, yet the only way of knowing the right answer is to already have enough knowledge to figure it out on your own.
"Don't expose x to the Internet" is pretty straightforward. It means don't let any rando with a public address talk to that service. So restrict it to an IP whitelist, or to only your private network.
Ideally you use an IP whitelist or restrict it to a small closed network to which you vpn in to connect, or use a non standard port, any combination of these
Nice and interesting video. However I am not that convinced by the conclusion. How am I supposed not to expose openssh ? Setup a vpn ? Well vpn can also have security vulnerabilities. It’s like saying close your http and https port : well how am I supposed to expose my website to the world. Not a fan of this conclusion. Instead of course update ssh to get the fixes anytime a vulnerability is discovered. And do not expose non necessary services
But if this really boils down to signal + malloc, isn't a lot of software besides OpenSSH affected? And does this mean that signals are useless for everything except maybe doing some cleanup and logging before shutting the process down? I really hope I misunderstood something.
So if i understand correctly, the exploiter injects the required function pointers for shell root onto to the compromised heap via the certificates being sent?
It seems like you forgot to mention this exploit is only for glibc system. OpenBSD is not vulnerable to this exploit. This also seem a good lead in to doing a video about ASLR and KARL.
haha wouldn't it be cool if you learned C and assembly haha lowlevel.academy
no it wouldnt
Bro can you please tell me how much time it takes to learn assembly language from which you can write your own script exploits and malwares and ransomwares.
And which are the best high level languages which are used to write a malware or ransomwares
no it wouldnt
@@PrinceKumar-yo9lr It really depends on what you're exploiting. If it's a local vulnerability (one through the operating system) then you'll need to have full control over the arguments you pass to the operating system's APIs. This is best done with a low level language such as C, C++, Zig, or even Rust. If you need *really* fine control, you can use Assembly, but it's not common.
However, must vulnerabilities are rooted in networking. These can be made in any language that has networking capabilities. Rust is my pick, just because of how easy it is to throw some bytes through a socket. POSIX's C sockets take a bit more work to set up than a higher-level TCP stream, but it's way better than Winsock from what I've seen.
However, this all assumes you have a vulnerability to exploit. Vulnerabilities are often patched as soon as they pop up.
Something way cooler than just malicious stuff is "fun" malware, which was away more prevalent back then. It didn't usually do anything super serious, but it would take control of the system and show you something really cool looking. There are some Windows 10 programs that do this with WinGDI (see MEMZ and Chloroform). You could learn to do the same, since it looks really cool. You don't need to exploit anything to make something really cool, just look at Furnace.
@@PrinceKumar-yo9lr 2 minutes and 34 seconds 👍
Temple OS is once again not affected? Coincidence?
Too holy to get hacked 🙏
Not a coincidence.
kowinkydink?
it's GOD'S work my dude
Thats why god uses it
The creativity of threat hunters will NEVER cease to amaze me
Breakers are one or several steps ahead of builders.
Agreed. People doing this kind of work are fascinating and awesome
There has to be an invisible hand from the intelligence community to plant some of these. I'm not saying that programmers never make mistakes that allow sploits, but those are probably the exception, not the rule.
@@brainites by definition, or they wouldn't be breakers. Sometimes builders can be ahead, as is the case with attempts at quantum computer proof crypto that is being worked on before quantum supremacy is reached.
The lack of creativity of developers will NEVER cease to amaze me
That's why we call it "OpenSSH".
no cap
I still don't get why people open the SSH port when they can use wireguard since if the device is compromised all bets are off anyway
@@AkivaB Guess it's difficult to maintain the wireguard configuration for all your devices, especially for multiple users - personally I like to use an open source mesh VPN like Tailscale or ZeroTier.
the door is wide open
OpenSeshame
LLL: "It's from 20 years ago, 2006."
Me: "It's not THAT long -- Oh shit..."
Yeah, my mind jumped to the 90s as well
@@mephistovonfaust20 years ago is, and forever shall be, the 80s.
you're old
@@prototypeinheritance515 respect your elders, boyo. ;)
LOL exactly! :-) For me is 10-15 years ago in the 1980s and last year is about 2001. :-)
"Everyone can do it" - Yeah for now nobody was able to do it on a 64 bit system only on 32 bit systems lol.
just was about to comment that "everyone"-line. I doubt my mom could time the attack right.. she always forgets to compensate for latency...
Nor if there's a connection limit via firewall. Even with the biggest botnet it would take forever.
I think at this point we can update the saying to "the three hardest problems in computer science are cache invalidation, naming things, asynchronous programs and 'Off By 1' errors"
Throw in interrupts like SigAlarm and you got a nightmare.
@99temporal I see what you did there
2B OR ≠2B
I endorse this comment.
Can I just say this? Thank you Low Level Learning for dark mode. So many yt chanels flash bang me.
Nothing worse than when a TH-camr bangs you, that’s for sure
Agreed 👏
I wish a TH-camr bangs me too someday
@@_Salaar_khan 😂😂😂
Bugs like this are part of why I use a pretty aggressive fail2ban. The attacker doesn't get 10,000 tries... instead they get 3 tries or sometimes even less. The bans eventually expire, but instead of hours to get in, it would take decades. Plenty of time to install a fixed version.
You can get nailed on the first try if you're unlucky, or the timing might never work for an attacker. Even 64 bit systems could get catastrophically unlucky. At least it's an easy fix this time.
@@parad0xheart There are ways to make it detect and block IP ranges, in both ipv4 and ipv6. It just depends on whether the admin actually bothered.
@@parad0xheartI'm not sure about fail2ban specifically, but it's standard to block the whole /64 range for IPv6. Each customer / network is supposed to get its own /64, so it makes sense to block the entire range.
@@parad0xheart or you just disable IPv6 for SSH, by setting the protocol to "inet" in ssh config.
I fail2ban myself so often because SSH automatically tries multiple different keys😅
very well explained. i love that the vulnerability is put under real word context and report is not just a scary click bait. if one has a cloud server e.g. amazon, they should limit their client IP address for that ssh port.
Is that the recommended method? I also always thought It would be risky to use an ssh server outside my home network. But don't know what to do instead. What if there is a coffee shop with the same provider and open wifi nearby. Wouldn't they also have the same IP? Of course it would still be a lot harder to hack the server than.
@@leok42 , your ISP typically assigns an unique but non-permanent address for each location. so your server would see different ip address at your coffee shop vs your home address unless you tunnel through your home address.
@@leok42 Usually, because of the lack of ipv4 addresses - the ISP can use a DS-lite tunnel to map multiple customers onto a single ipv4. On this case you will have an ipv6 address which is unique tho.
In my situation (Germany): The way you get your internet differs immensly. My contract for my home is as described above (DS-lite + ipv6). Some households only have an ipv4 that changes daily. Sometimes they only have an ipv4 that only changes every ~month.
oh that is why an openssh update was avaliable.
They patched it already?
@@johndank2209 it was revealed as the patch is out
Yeah Ubuntu already patched it up on July 1st
openssh 1:9.6p1-3ubuntu13.3
CVE-2024-6387
Edit:
From the bug report itself
2024-05-19: We contacted OpenSSH's developers. Successive iterations of
patches and patch reviews followed.
2024-06-20: We contacted the distros@openwall.
2024-07-01: Coordinated Release Date.
@@johndank2209 it was an accidental regression, should be super easy to patch. Just revert the code that was never supposed to be there anyway
Just wanna say I love your vids man , high prod quality and clear description of the issue.
I use sssh. Safer ssh
there is no such thing as "safe"
ssssssshhhhhh
@@sineflake sshhhh, its "safer", not "safe"
Hahaha I see what you did here!
Don't google sssh 🤣Straight to PH
Your explaination for laypersons is very very good. I'm not a programmer or security expert by any means, but found it was easy to comprehend thanks to your summary
This has all my windows people at work scream LINUX VIRUS and im so exhausted of telling them it would take literal hours and using fail2ban is a dead simple mitigation any public server should have anyway. Ugh... That said, this explanation was really good! Reminds me of the late Tetris level shenanigans where VBlank interrupts cause almost the same situation - albeit of a different nature.
the regression has been fixed anyway already even my old ubuntu lts jammy pi home server already got a patch for it
Like OpenSSH is not present on Windows also...
Be sure to update your fail2ban sshd filter after installing openssh 9.8 ;)
This is also more exploitable as the paper mentioned on 32-bit CPUs... which in 2024, who is seriously even using 32-bit for anything, let alone a server on the Internet for anything productive? So, this is essentially a very minor issue in my eyes and shouldn't affect that many people or servers.
...i have heard whispers & jokes of "Linux" & "packet sniffing": But they're so busy laughing that I cannot understand what they're saying... Can you comment on this at all ?
Great job explaining this vulnerability. But I think you got the LoginGraceTime part wrong. According to sshd_config's man page: "The server disconnects after this time if the user has not successfully logged in. If the value is 0, there is no time limit." - Which could result in a DoS if the maximum unauthorized connections are exhausted.
That's exactly what the paper points out. (10:58)
10:51 It does not close it immediately but rather does not close it at all. That's why as researchers mention it make you vulnerable to dos attacks as attacker does not have time limit for spawning too many waiting logins
Which could easily be a bigger problem than a vulnerability with no known exploit.
What an excellent explanation, you are a great teacher.
Subscribed!
I don't personally like your implied criticism of open source software twards the end of these kinds of videos. While I understand being cautious, it makes it kinda feel like its somehow a bad solution to an other wise worse alternative. Personally I think instilling fear in something that has been the better choice in security since the dawn of the internet is not a good idea. I do agree that its not perfect, but until theres an objectively better option, I would prefer that you didn't make it sound as if the world is going to collapse because we rely on the better of our options in software security.
This is a really high quality and useful video for me. It makes me look smart to my bosses. Thank you :)
That's it, you're going into the Rust rewriter
Oh boy, the rewrite in rust gang is coming!
OH LAWD! HE COMIN!
Is Rust also Thread Safe?!
@@gavabundo_0072that’s like half the whole point
@@gavabundo_0072 This vulnerability is about signal safety - that is a whole other level of safety that Rust does not provide. When the signal handler is invoked in this exploit, the heap is corrupt. If you do anything with the heap at that point, you are bound to have something exploitable and a signal handler is Rust CAN interact with the heap.
we gonna be ruSHing
Please add sections to your video! 🙂
Especially for experts, it is nice to skip stuff like explanations what SSH is.
@@ForcefighterX2 +10000000
I sent a similar video to someone at my office. He's like: updating the libraries now. We then talked about the importance of testing known weak points in code (since it was a regression). Gotta keep an eye on known previous points of failure.
I wanted to touch on something you noted late in the video, regarding recommending not exposing SSH on the internet, which invites the question of what do you suggest instead? You can do a lot to try and isolate management networks/etc, but ultimately you need a legitimate way in. Your argument that 'code can have bugs' applies to pretty much anything, we've seen various firewall vendor and VPN bugs in the past, so they're not different. How would you handle remote access?
unfortunately imo the only other way is IP address whitelisting. it's not pretty but it significantly reduces the attack surface
@@LowLevelTV I can agree with you on that. Sometimes that presents a practicality problem, but it does significantly improve the posture when possible. And then in the case of this particular bug, something like fail2ban would probably go a long way in mitigation (though not closing off the bug entirely), given the large number of tries required. Thanks as always for the great content!
@@NigelVH One low-tech way to reduce risk is to require a port knock or similar. It's primitive, but still sufficient to stop most attacks.
Run SSH on a non-standard port, use fail2ban, or limit what IP blocks you allow to access (if you're in the US, do you need to allow access from other continents?).
For big organizations that have their own IPv4 blocks they got from a RIR it's super easy, you just only allow from your own IP blocks and reject everything else
I think he was referring to don't connect it to the internet while using the vulnerable version, not don't use SSH for its intended purpose ever. If that's what he did mean, then there's a couple of things you can do like whitelisting only specific IPs, or port knocking, but these only reduce the attack surface not make it safe. IMO its worth the risk if you take proper cautions like, IP address whitelisting, but not using a tool just because there's a possibility it could be vulnerable is dumb.
This sounds like an early implementation of a TAS speed run with a wrong warp. It sounds impossible to execute but determined people can make these issues exploitable at a moment’s notice.
Yuffie mentioned.
Interesting video & well explained. I'll be coming back to this channel for more content like this, good stuff! 👍
Bro said "code is code." genius.
yo, i'm no code guy but enjoy stuff like this from u, primeagen, dave's garage n the likes, i appreciate the logic n informative value u guys bring
I mean yeah you're right, this isn't the kind of exploit to some random individual is going to use to hack into a bunch of servers. But for extremely sophisticated, targeted attacks, stuff like this can be and is exploited.
Great stuff. Thanks ever so much LLL!
Great coverage on the subject when everyone else is screaming everything could be on fire. Seriously though big points to reviewing the mitigations and explaining the exploit in a easy to consume video!
@0:27 "...not that scary"
Title: ABSOLUTELY WILD !!!!
😂😂
great point! 🤣
Think of a giraffe. Wild? Wild. Scary? Not so much.
This is how we all live now
@@szelest88 Yeah, what they pulled off is insane and I now have much more respect for that company. I may actually attend their next webcast.
@@szelest88 *a wild exploit has appeared*
😆
I swear every time I get a notification from low level learning it's some scary vulnerability that may affect one of my systems
How well can that 4-6 hours be parallelized? If an attacker can work on thousands+ of targets simultaneously then it still seems pretty bad
You need a pretty stable connection for race conditions. So, working on thousands of targets would be extremely expensive.
@@somebodystealsmyname Establishing SSH connections costs very little bandwidth. Depending on the exact timing, AWS may not be enough. But a small host with good connectivity to your target ranges, which can be established with a BGP looking glass, and many of these have very limited to no KYC -- those are great for these attacks
already covered in the video. OpenSSH throttles new connections to... 100 in a second? which is why it takes 3-4 hours based on how quickly it allows connections to come in.
Finally! I don't have to worry about forgetting my password anymore.
For critical projects like this (at the very least), there should be a process built into the commit procedure that checks for various types of vulnerabilities, and especially for specific vulnerabilities that were previously found and patched.
Can you imagine any legacy devices common on local networks that use the vulnerable ssh? Perhaps even those not owned by the user
Would you be able to do a video explaining ASLR? I understand the basic concept, but don't understand how it doesn't cause code to break.
Ed, you are an amazing researcher and very smart guy, if I may, do you have any attack vectors that you came up on any of those situations that show us? Just asking,
It's amazing, the blogger is really creative and worth watching
Thx for ur efforts to do this video, one simple question, wouldn’t be awesome if the developers of OpenSSH project rewrite most of their base code using Rust language! Due to enormous hype about it, in which of its core features eliminating race conditions and other memory faulty stuff?!
This is literally the meaning of "grace" and since it was implement it has always been known to be a potential vulnerability.
Crazy exploit! Thanks for making me aware to this
On one hand its insane that syslog caused this vulnerability. This is probably an issue with other software as well, given using syslog is rather common.
But on the other hand, this only exist if the attacker manages to time a timeout-timer between two lines inside a malloc call. They say mallocing heap is slow but its pretty fast compared to that. Just insane work by the redearchers who made this real.
This is not a problem with syslog though, but with using non async safe code in an async context.
@somebodystealsmyname actually kinda incorrect the problem is much more deeper and fundamental to how operating systems manage running processes. "Process Interrupt Race" style bugs are old since 2001 at least they were known but are gaining a resurgence again.
There just much more rare in single process async code scenarios because other then signal interrupts there's not much code that can overlap or interleave at the time the process can be interrupted.
I'm sure there's other bug variations though where a process works on shared data rather then your typical threads and interrupting the process during specific instructions will leave say a DBMS in an inconsistent state.
We could see flaws like this on phps zend interpreter process and maybe a reawakening of maybe not SQL injections but new forms of IDORs and other wild bypasses that stem from interrupting execution flow of a process.
Even in processes without Reentrancy and even some processes that run mostly Synchronize code I could see just interrupting a process causing issues while another process acts on the same shared state like cart data of an e-commerce site.
But as for single process async overlap stuff to mess with heaps like in this openSSH stuff nothing comes to mind other then signal handlers and maybe system calls which causes the kernal to block a process temporarily.
kind of video you wanna see right after starting openSSH
frfr just setup my vps yesterday for a minecraft ds-lite nat proxy tunnel and well haha sudo apt update sudo reboot
Same! I learnt more about `ssh` and `tmux` JUST YESTERDAY and now I get to watch this!
...
Thank you, Ed. At least I know how to keep my `ssh` connections more secure _nauw!..._
A phrase to parallel JerryRigEverything: “Code is code, and code breaks”
@@bork_games I have written unbreakable code
observe
int main() {
return 0;
}
I think you’re the only youtuber who covers this with proper manner instead of reading some news outlet who wrote no background of programming
That seems like an art piece or concept work... A meandering of what's possible, might not be practical but possible and clever none the less.
I wonder how long this exploit has been known and if it is being actively utilized in the wild. My server has been getting hammered hard since the end of June. I was getting over 2,000 access attempts a day for a while. Up from just a few hundred a week.
I am a little bit perplexed.
Firstly, removing openssh from the Internet may be an option for some, but many hosting providers actually allow clients to use ssh to update their files on the host. For them to remove ssh from the Internet would mean that clients have lost the ability to upload files to their host (or at least have lost that path by which they can upload).
Secondly, if this problem is in malloc(), the I would think that potentially it has far wider issues than merely OpenSSH, malloc() is a core part of memory management on most systems (even if it is behind the scenes), and it would seem very probable that there are other pieces of software out there that allow signal interrupts while using malloc().
Use sshv2 straight up instead of using unsecure free replacement programs.
On top of that, dont allow direct internet access to the ssh port of those servers. You could easily implement a vpn that could provide access to whatever management is needed securely
@@lillones none of these alternatives are guaranteed to be any safer than OpenSSH.
If the problem is the way malloc() interacts with signals then there is no reason to be so certain that a VPN would not have the same issue. All that a closed source alternative to OpenSSH would give you is the bliss of ignorance - you same problems might be there but it would take a lot longer to find them.
The vpn would allow you to send requests for ssh internally on the intranet rather than the open internet. It would block any rando from being able to even attempt to initiate a session to take advantage of the vulnerability
@@lillones yes, but the VPN itself is using an encrypted tunnel just as SSH does.
@@Hfil66 yes... and water is also wet, but that has nothing to do with my point
the last time I was this early the queen was still alive
The Queen is still alive though? 🤔
I'll be embarrassed if I check the news and see that Camilla has died...
@@MenaceInc The Queen that actually mattered, not the current one
@@mochafennec Victoria? Zenobia? Cleopatra? Freddie Mercury?
This can be mitigated with a trivial change to a config file
It takes concerted effort over time on 32-bit
It probably isn't possible on 64-bit due to planning to prevent these things
It's been fixed and an update has been released
It's never been proven to exist in the wild.
So, not losing any sleep AT ALL over this one. Academically fun, but not something that rising to any concern whatsoever.
I'm missing something... the attacker tries to log in and waits for the grace period to expire, and they need it to expire right when malloc is being called so that heap memory is corrupted. But how exactly is the attacker injecting code? How do they send the malicious code to the machine running ssh, and then how do they exploit the heap corruption to shove their code into a place where it can be executed? They're not providing a custom sigalarm handler are they? That seems like it would be too easy.
I believe the code is present in their certificate or some other part of the SSH connection request. They time sending some portion of their request so that it is present in the heap with the signal handler is invoked. The signal handler then ends up executing what they had inserted in to the heap.
The signal handler is one written by OpenSSH to deal with connection timeouts. The signal handler ends up interacting with the heap though, which is attacker controlled and leads to the exploit.
Basically while the OpenSSH "regreSSHion" vulnerability sounds concerning, it's not a major threat. Exploitation is complex and requires hours of attempts under specific conditions, making widespread attacks unlikely. Many systems already have mitigations like brute-force detection in place, and the scope is limited to certain OpenSSH versions. Patch your systems
...no need to panic.
good to see that ASLR is almost doing the job it's designed to do. #speedbump
Great content! Thank you!
Have you ever made a video on the Intel Management Engine? My professor in my OS class mentioned that every Intel chip has its own OS that is based on Minix3. I've read people call it "ring -3", and so Intel basically has a root kit on every Intel PC. I would be interested to hear your thoughts on this
This exploit is pure art!
I love how a serious RCE gets resurfaced because "oops I [un]commented the wrong #ifdef" AKA how often does stuff like this make it into open code bases because no one notices? Who knows if this oops was intentional or not.
and this is exactly why i never expose ssh to the internet, but rather behind a preconfigured wireguard intranet.
None of my servers ever get direct internet access. They are all firewalled off from the internet and I VPN into my local network and then access any of my servers. Never direct SSH or any other method directly into a server over the internet! It's much easier to "secure" and keep a close eye on a single known external connection than a bunch of smaller random entry points all over the network.
Great video and breakdown!
Reminds me of one of the exploits in the chain for Eternal Blue.
Appreciate you spent ~2min of this video explaining what SSH is, (assuming the viewers could never use it) and then spend nothing on explaining what heap or malloc are 😂
"keep SSH off the internet" significantly dings the usefulness of SSH though. people will continue to use SSH as their primary method for accessing VPS instances for the forseeable I think
You just need to use a IP whitelist or a VPN
What a champ and good explainer.
Great video and explanation
3:32 Not really C programming, more "I code on top of an OS" programming.
It's just that especially scripting languages don't let you interact with them directly.
Java for example uses signals to detect when you dereference a null pointer and instead of dying you get a NullPointerException.
But you can (and kinda have to) interact with signals in every at least system level programming language.
Everything else aside, I'm really happy the paper starts by quoting The Interrupters.
"Don't expose your ssh to the internet" Prime's startup in shambles
thanks for the great explanation.
Noob here. What does it mean "Step 1: Get your SSH off the internet"
How then am I supposed to connect to a remote server? Or was that meant generally to keep the SSH port closed if you are not using it?
@@nicebhaalu961 yes, the port should not be open for random people to attempt authentication to, if you can avoid it. Making it accessible only through a VPN or overlay network is one strategy. Using SPA to selectively open the port when you want to connect is another strategy.
Basically, add additional layers to mitigate the risk that any single layer fails.
Finally! I found someone that also pronounces it as 'daymon' instead of 'deamon'! A.k.a. the correct way!!!!!
I would argue that the root of this problem is the complexity that signals introduce due to their ability to interrupt critical code. It’s hard for a developer to anticipate every possible place a signal might interrupt execution. A smart developer really should avoid the use of signals for application functionality.
This is yet another reminder that the original release mentioned this is excruciatingly hard to exploit, and trivial to mitigate.
Remember to breathe.
Every time when i see exploit like this i wonder, maybe it is not feasible for remote access, but could be used for privileges escalation for local users and rooting phones, consoles etc.
Portable version only, that is, not using BSD kernel. Plus unlikely if you are not a big target (like a public ip address)
people incorrectly assume that updating your software to the latest version is safer than running older code
well since they patched out the regression right now it is
Does this mean that a bunch of SSH enabled devices that manufacturers have locked down can now be cracked open for legitimate reasons?
Not sure about x86 / ARM thing but if 32bit just means 32bit then - yes.
You missed one of the interesting bits - why it's a regression. OpenBSD implemented a signal-safe version of syslog() so they cleaned up the signal handler to use that. But ports without safe syslog() became vulnerable again because OpenSSH depended on the safe OpenBSD version. It's a fascinating interaction where making things safer paradoxically opened a vulnerability. I don't know what the answer is - everybody should be watching what OpenBSD does and following suit but that takes time to catch up. maybe OpenBSD needs to drastically rename interfaces when they fix them so it's extremely obvious that using the older interface is dangerous? This isn't the first time porting OpenSSH has caused problems, and probably won't be the last. Debian famously fixed an uninitialized memory read "error" and generated the same 65,000 SSH keys on every system.
I do not know much about this topic, but I always wondered what even the advantage of code and data sharing the same memory is? Wouldn't this kind of exploit be impossible if the PC could never point to a block of data?
I rarely had to do signal handlers but the first thing I do is making sure no mallocs are reachable.
I still remember the first time I had a server open to the internet the only reason I didn't get hacked is because the logs from their attack filled my 40gb drive and crashed the system. Something like this would have been very bad.
timing here is interesting. would an attack perhaps be exploitable faster with less network latency deviation (e.g. intra-datacenter exploits) i would presume attack could be performed much faster if you knew additional information about where in the cloud your victim is hosted and network link speeds are much more predictable.
The 10000 tries the researchers got were under lab conditions. So it will mostlikely be longer in real world conditions.
Think the research group the lowest latency they were attempting at was 10ms or something crazy low like that.
So setting grace time to 0 is a fix.
I guess because of that, having password authentication disabled also fixes it since it instantly rejects any incorrect private key.
Sweet.
I never allow ssh to be open to the public internet anyway. Lock it down. Bind SSH to a private IP or only allow from a specific address/subnet. Then access over VPN.
10:20 So what is the alternative? Of course only having physical access to a server would be the most secure option but it is not practicable. What else can be done apart from using software such as fail2ban and putting the SSH server on a non standard port?
@LowLevelLearning Have they completed the 64bit PoC yet? Last I saw they still only had only successfully exploit in 32-bit. However, they were working on a 64-bit version
Was waiting for this vid
10:42 "Don't expose ssh to the internet!" can you elaborate?
Do you mean to use it only in a local network? Or not to make the path straight, like an IP or domain pointing directly to your ssh server? Or maybe to gatekeep the connection before reaching the ssh server? Or just not make it public knowledge?
Learning security bugs me because security professionals give 4 different meanings to the same exact phrase, yet the only way of knowing the right answer is to already have enough knowledge to figure it out on your own.
Use it it only in local network. Nothing good can come out from that kind of remote access.
"Don't expose x to the Internet" is pretty straightforward. It means don't let any rando with a public address talk to that service.
So restrict it to an IP whitelist, or to only your private network.
Ideally you use an IP whitelist or restrict it to a small closed network to which you vpn in to connect, or use a non standard port, any combination of these
Damn he really exposed it. This was a closely guarded secret for years 😣
and apt upgrade is not going for the latest version automatically.. great job
Nice and interesting video. However I am not that convinced by the conclusion.
How am I supposed not to expose openssh ?
Setup a vpn ? Well vpn can also have security vulnerabilities.
It’s like saying close your http and https port : well how am I supposed to expose my website to the world.
Not a fan of this conclusion.
Instead of course update ssh to get the fixes anytime a vulnerability is discovered.
And do not expose non necessary services
C has been here.
How can you tell?
Memory management based vulnerability.
"re evaluate your life" did this guy just tell me to unalive lmao
_nu._
But if this really boils down to signal + malloc, isn't a lot of software besides OpenSSH affected? And does this mean that signals are useless for everything except maybe doing some cleanup and logging before shutting the process down?
I really hope I misunderstood something.
So if i understand correctly, the exploiter injects the required function pointers for shell root onto to the compromised heap via the certificates being sent?
It seems like you forgot to mention this exploit is only for glibc system. OpenBSD is not vulnerable to this exploit. This also seem a good lead in to doing a video about ASLR and KARL.
I thought it was well known a signal handler should do nothing but set a flag variable. Wu-ftpd had a famous signal remote root bug in the 1990s.