0:14 AFAIK OpenSSH does not directly depend on lzma. Some distros (e.g. Debian) patch libsystemd into OpenSSH for readyness notification, which in turn depends on liblzma. Upstream ssh should be safe. More on this in the HN thread.
@@gingeral253hi definitely is an accomplished developer who stumbled upon the Valgrind issue and was motivated enough to investigate and then find it. This guy probably wanted to help fix a memory issue and had the perseverence to go to the bottom of it.
Kudos to this „just a guy“ who probably just wanted to help out an open source project in fixing a memory leak and performance issue and then probably spent weeks of his time getting to the bottom of this. This is what makes all of us safer. He deserves a medal or something.
@@cigmorfil4101 well, Microsoft has a bug bounty program, as most big companies have nowadays. The ‚just a guy‘ is Andres Freund, some blogs call him ‚microsoft security researcher‘.
X.x What interesting events, a tinfoil hat might start to think supply chain issues, and notice the same kind of ticks in other things like 'heartbleed bug' and just how 'slightly off', but not enough it was to be spotted by all but the most ocd or diligent person for their own thing. Just sayin.... Maybe Open source is more at danger of being infected by forign bodies than closed source, and taken for safer .. when most don't have a single clue about wtf is going on, or just why an issue or even performance hick-up exists. Who would ever be absurd enough to question 1-2 second single hick-up in a sign in process for something meant to exist on the internet where that delay would be unreasonable to impossible to spot in the live environment. Speaking of cool things, Isn't it great that there are so many easy to access cloud-based hosting solutions meant for personal & small business users? That convenance is great, just have to deal with a tiny bit of internet lag because tech stuff. I mean it would suck if they all used a common supplier that managed access in some way or something, or data security... *cough cough*, because some big big target would be pained on the backs of such a supplier and be closely watched. Man, the amount of qq from governmental bodies about data encryption and traffic routing has sure gone down, I bet they only realized how foolish it was for them to keep on that. I guess all is well, and the days of the governmental bodies pulling overly complicated & deep state affairs on the people they care about is over for our own people right?
Important Clarification (since I feel this isn't clarified): upstream OpenSSH doesn't use liblzma, however many distros like Debian patch OpenSSH to use SystemD Notifications through libsystemd, which in turn uses liblzma. Distros like Arch (which don't patch OpenSSH) or distros without SystemD like Void should be fine with regards to SSH (however most distros are already downgrading xz anyway for obvious security reasons) Source: the latest Arch News post regarding this backdoor EDIT: to quote directly from the Arch news post: "Arch does not directly link openssh to liblzma, and thus this attack vector is not possible"
@@mx338but if your systems are sufficiently firewalled... What is the exploit going to do? This is not a good situation, but it won't break the world either. Vulnerabilities are found almost every day it seems in so much software.
@bzuidgeest you're right, ideally you do not expose SSH to the internet without restrictions, I run everything on prem for work and don't allow SSH from the internet. However in cloud computing that's often different, if you run and AWS EC2 or any other cloud VPS, SSH is going to be exposed to the internet by default.
Using a single makefile line that looks weird but not totally out of the ordinary is so creative. On the other hand how was this mf able to commit compressed binaries without getting nuked by a maintainer, i do that on accident occasionally on my branches and it draws the aggro of every devops guy in a 10 mile radius when i submit the PR
@@twentylush they passed it off as a test case. It makes sense at a quick glance, having a premade file that _should_ decompile properly, then using that in testing to see if everything works as expected it's just that the file wasn't only used for testing
I can already read online people saying that this is proof that open source code is less secure than proprietary On the contrary, the fact that it was caught, and caught relatively quickly, shows that open source is more secure against these kinds of backdoor attacks
"this" was caught but there could be many more backdoors which will never be discovered. Imagine if a war breaks out between countries. Of course no government agency would trust proprietary software during an active war. In that scenario, the country that managed to put backdoors in open source software will have immense advantage.
Indeed.If someone can do this in code that's opening visible, imagine the shit that goes on behind the curtain in non-opensource crap. (Solarwinds comes to mind.)
Also, we have a clear method to identify and track the malicious actor's influence. We know what account they're using, and can thus see every open source project they have touched.
After this, he better be. This is really well designed malware. The ways they are cleverly hiding it, all the involved packages, the linker, removed the symbols from the malware in xz, using the testing branch, those mentioned in the video, or the fact that it's not even in libssh, and it only works if you run through systemd which systemd would be compiled to use xz. That's really well orchestrated, that's gov malware I think.
@@brians7100 Some of his suspicious push requests date back over 2 years, so its possible this isn't the first time he has tried to do something like this, just the first time he got caught.
@@brians7100 Even if he isn't active anymore, the damage is done and needs to be detected and reverted in each repository this guy has committed to. I bet it'll last some time as an active issue until things are resolved for most packages affected, and many machines will be compromised until then
Kudos to the person who found this. His modesty does him credit, but the realization that something was amiss, the desire to delve into it, the analytical process, depth of research, and the willingness to share what he discovered with the wider community - totally aligned with the OSS ethos - shows he definitely has the right mindset to be a security researcher and has hit the ground running. Hats off, sir; bravo!
as a sidenote from what I gathered: The person who "contributed" this backdoor was not just some person who randomly came out of nowhere with a Merge Request. It was someone who contributed to the project for an extended period of time to a point where they themselves became a maintainer (not the main one, but projects like this often have multiple).
@@JordanPlayz158they were obviously being paid already for this. Plenty of maintainers are on the payroll of Google, Microsoft, Redhat etc. This is just the first time a malicious group decided to go that route as well
@@Pharoah2 just the first time? Lol do you think that between the thousands of FAANG+ employess none of them is on the payroll of some 3 letters agency?
Idk if I would call Andres Freund “some guy” haha he’s a Postgres contributor/developer and Principle SWE at Microsoft. I get your point though. Not technically a security researcher.
Yeah. Just got the news about it 9h ago. My Manjaro installation has the aforementioned versions installed and now I don't know what to do. Rollback, or directly wipe the system and return to some more newbies OS like Mint.
Loving these security vulnerability breakdowns man! I can only imagine the effort and background knowledge it takes to put content like this together, thank you.
That is definitely one of the most crazy, complex, and sophisticated backdoor injection attempts I ever seen in my life. The engineering behind it is very impressive. The guy who discovered that deserves a reward, he just literally saved the world
After this report there were some extra findings that people might feel valuable. 1. Some time ago the only mantainer got a lot of pressure from different users to accept another mantainer to the project. 2. The binary files in question were compressed files that are usually made to test that the decompressing tool was working, in this case the malicious tests were two, one for a large file, and another for a corrupted file. As you can see both made for making it difficult to find that there was hidden code there. 3. The malicious mantainer got enough trust to be able to sign the distributed tars with malicious code and to contact linux distro mantainers to presure them to update to the backdoored version, the attacker even sent a patch to the google repo that was harmless by itself but a requirement for the exploit. 4. After the backdoor was created random accounts submitted prs to different projects to update to the vulnerable version. This was a large well orquestated attack that was most likely planned by more than one person and only discovered due to it having performance problems and certain bugs, otherwise we might have never noticed.
@@Dratchev241 Eh, if you're going to do an intelligence operation involving names but no face-to-face contact, you'd probably use a foreign name. I'd definitely say a state actor, but frankly, it could be anyone. I was leaning towards someone Western but smaller, but I really have nothing to base that on.
Yeah this was definitely orchestrated and planned for a long time and took a lot of planning. They just got unlucky that someone noticed the performance decrease and investigated and found it by pure accident.
@@Dratchev241 Hah, because obviously Chinese intelligence agencies would use a fucking Chinese name to do this, right? If anything this was probably done by american intelligence agencies using an asian name to generate propaganda headlines if it didnt work.
@@Dratchev241 You don't need a state actor to create and coordinate multiple accounts. There is no proof that those multiple accounts requesting the update on client programs belonged to different people. It only requires coordination. But one person can easily coordinate with themselves. Your point 1(pressure on the single maintainer of a core library to accept other contributors) is actually what you should expect. You don't want a single unsupervised individual to be the alpha and omega of such a central library with zero backup. That's what actually scares me about OSS and Linux. There are so many core technical libraries that are maintained by a very small group of persons. Any rogue in one of those core groups can cause wide damage. There way too little people interested in working in those dark places where you get no recognition when everything works and all the blame when something goes bad in another software just because someone else used your library in a way that it was not supposed to be used. Point 2 is the actual issue: porous build process. Test files stored in the test directory should never find a way into the compiled binary. There should be a stronghold protecting the source path and compilation tools so that nothing outside of that scope can get into the binary files that are signed and published.
Bullet dodged. For real. This could have been the worst money grab backdoor by far. It's literally in every system. It's especially scary that the project owners approved the compiled binaries. Hopefully it's not a maintainer behind this.
It's literally the project owners or a representative of the project owners org. Both of the main devs that are the contact points for package maintainers have had their GitHub accounts suspended reportedly
It sounds like the original, single maintainer was overloaded and started passing off duties to another contributor who "helpfully" stepped in recently to reduce the load. Fast forward a few months and it looks like this helpful contributor was (likely) a state agent...
They aren't compiled binaries. They're compressed files. Which is totally normal for testing a compression tool. What was missed was potentially looking into the contents of the compressed files, which I know very few people would bother to do, or looking at the quality of the new tests. If you're just making existing code paths faster, why do you need new compressed files? Edit: auto-incorrect
You realize that this backdoor was caught but so many other could be successfully installed in open source code without our knowledge and without extremely deep code reviews. Though I would expect that other project would start scrutinizing things more deeply in the next couple of weeks and we'll be hearing more surprises
It is slightly more subtle, from what I understand. It is not that openssh uses liblzma, but liblzma is used in systemd. On systems where openssh is patched to use systemd as well, you end up with a security issue. This appears to be limited to the combination of x86_64 and linux and systemd. That is still a significant fraction of all linux systems.
@ok-tr1nwthis is a very specific backdoor that only affects software that uses lzma and does RSA with this specific library. How many programs fit that description...
so if I'm understanding this, the intent is to be able to use a specific key that doesn't have to be installed [legitimately] on a system (no direct attack necessary) to effectively gain "authorized" access to a system. to simplify it, this behaves like a master key or a master lock (iykwim).
Not exactly technically speaking, but the exact same effect in practice. The difference, fortunately, being that this way the attack can only happen under specific conditions. For example, some environment variables can make this not run, since the attack depends on specific code being injected. Of course, even with this stroke of "luck" it still is one of the most serious attacks/vulnerabilities found in the last couple of years.
The malicious commit was made Feb 23 and entered xz-utils stable on Feb 24. Detected in Debian-unstable. 5 points for early detection, before it should have been distributed to production anywhere?
I agree. Even the build systems I maintain for PHP code run tests and then (if the tests passed) checks out the Git repo again in order to do final deployment actions on a clean slate.
it doesn't, really... the test fixture just exists in the repo doing nothing, but with the modified build script, that test data is used to reconstruct the malicious code. in fact the test fixture in there isn't even used at all in a test (which should have been sus)
the actual git tree of the project doesn't have the M4 line which triggers this. The M4 line was added to the public source tarball post release by someone who had keys to do that.
Apologies if I missed it, but it should be clarified that this did not make it into any production releases. The fact that it was caught before release is a demonstration of the strength of the open source model.
I'd say we got very, very lucky that someone noticed some odd behavior and went 10 extra miles tracking it down. In fact I'd say this was a pretty big failure in terms of the model. As it looks like there is there was no verification that the source being used was even the same as the source in the repository(You know, the stuff people usually check for things like this). If it had been complied with the repo then it wouldn't have been an issue at all(vestigial, non executing code aside). Kind of clever really. Leave out the easiest to see bits from the source. Slap them in the release tarball and as long as the behavior isn't too different from the original nobody notices(and if it is you can probably just patch and blame it on something else like the did in 5.6.1) So all the normal checks and balances that would normally catch this in OSS(as far as I know) just don't work here. Only weeks after release and by blind luck did this get caught and that's not something to celebrate. Say what you will about closed source, but at least the back doors in their software are intended by the company ;)
OpenSUSE was vulnerable. MicroOS was vulnerable as a result. It's an os built for containers targeted at scalable services and bigger entities. Curious how many kube clusters got taken over.
@@haifutter4166 As someone stated here Arch doesn't use patched OpenSSH and since manjaro upstreams directly from Arch, I would say it didn't affect it. For Fedora on the other hand, it affected Fedora 40 beta in a few flavours, rawhide and silverblue (iirc?) being two of those. Older versions did not get affected
Wow - this is absolutely crazy. Never would have thought that this passes a review. Obfuscated code… I have thought several times - e.g. when people talk about hardware wallets’ software being open source - you dont know what ends up on the device, and even if the source is fine, you dont know what happens during the build process. A case like this shows me this suspicion isnt that unrealistic. Thank you for the video - I subscribed.
These videos are great. But please add more context or explanation about how these exploits could be used. I forwarded your previous video on kernel exploit to the management and they came back saying "we dont understand the implications from this"
It’s pretty simple, they could ssh into your server. What you have running on that server and whether or not they could they could perform privilege escalation would be required knowledge to determine what the bad actor “could do”, and from that would you determine the implications.
both of these are only doable if the attacker already has access to the network and most companies have an encrypted secure vpn and the ssh would be in the vpn right.. ?, so both are relatively harmless from external attacks, they do have to worry about sabotage from inside. Even then records would be kept and the culprits found.
How does management feel about everyone in the company who uses those servers having access to all of their files, including customer data? From an insider threat perspective, that's what we're talking about. A larger concern would be if one site gets hacked, then everything on that server would be hacked.
@@italianbasegard that's doesn't provide enough actionable context. It should include "in bold" the identifiers of the compromised releases. It wouldn't take too much effort to highlight that this only about "RSA's implementation" + "liblzma [...] tarballs for 5.6.0 and 5.6.1" and NOT just "SSH libraries".
@@italianbasegard I don't think they would require privilege escalation. From my understanding of the video, they could hypothetically log in as root, using their key, and get root access.
The backdoor payload was hidden among dozens of innocent compressed test files, and the hook to execute the script that embedded the payload was slipped into a makefile in the release tarball but not committed to Git. We absolutely should be in the habit of diffing our release tarballs vs. the Git tags from now on, but we're going to need to come up with a smart way of analyzing any high-entropy files checked into Git for potential hidden payloads.
@@CFSworks I feel like its a failure of keeping up with the times though! this exploit is a complete dead-end if you policy-out artifacts in code repos and instead only pulling them in containerized environments when the "tests" are actually run. Not to say its all the maintainers fault, I mean the OG seemed pretty burnt out and overloaded, and there's not a lot of up-to-date devops guys with a golden-heart and clear schedule to bring these OS projects to the modern ci/cd era
@@twentylush I suspect that an attacker trying to pull off an attack like this in the future would just switch to hiding the payload in a non-artifact binary file instead. Perhaps a base64 chunk hidden in the metadata of the project's logo, or a loose object under .git. There are too many opportunities for outright steganography here. :/
@@absurdengineering Definitely. But here we had a malicious maintainer, so even with auto-upload of the git-archive output, they would have just secretly replaced the tarball 3 seconds afterward with the backdoored version.
Microsoft employees have names and adresses, they'd be found on an audit, fired, blacklisted and possibly jailed for something like this. Open source maintainers? Good luck going after a dude you're not even sure exists on the other side of the world.
That is actually a huge problem of OSS. Many "popular OSS libraries" (aka core building blocks used by hundreds of projects) are actually not popular at all. They are maintained by 1 or 2 people at best and everyone else just blindly integrates those libraries into their project trusting that some magic being has actually independently reviewed and validated what those guys did. And for that exact reason those libraries are very easy to infiltrate and alter for rapid damage on a lot of projects at once. There is no actual incentive outside of pure benevolence for someone to actively participate in securing some of the most technical and sensitive core blocks of the OSS architecture. Being open or closed software is irrelevant here. It's an organisational issue of the dev teams. Everybody wants reviewed binary that they can trust but nobody wants to be the one that spends time doing the actual review.
No. The problem is patching security sensitive programs to use 3rd party libraries and, of course, not letting the most important daemon on the system (systemd) that handles everything using 3rd party libraries. There's a reason people are monitoring openssh a lot more then some compression library and this problem wouldn't have been found for who knows how long if a persons OCD hadn't kicked in to thinking a program takes too long to open (half a second). It shouldn't have happened and on systems that didn't try and screw around just to make a config file look prettier, it didn't.
@@MrFujinko Actually... I wonder, if someone could Chainfuck EFI. You just need a lib, which is most likely run as root. So the same crap, but write code, which drops a EFI binary into /boot and hooks that into NVRAM and the loader, which in turn reloads or mimics GRUB. Part of this EFI binary goes straight to Ring -3 on AMD or Ring -1 on Intel, acting as a hypervisor. The PC boots normally, but the Chainfucker is in control of the system by opening ports outside linux control. It's just a theory though, I am no expert.
@@Cobalt985That's not the whole truth. Arch was affected but OpenSSH on Arch was not affected. Quote from Arch news: "Arch does not directly link openssh to liblzma, and thus this attack vector is not possible". But in xz package history you can clearly see that affected upstream versions 5.6.0 and 5.6.1 has been released by Arch. Still and forever user of Arch 😍
unpopular opinion: workflow 1 pulls the repository, builds everything and then tests everything, workflow 2 pulls the repo without using workflow 1's output, builds and publishes, that way malicious code must be in the build commands or toolchain where it will be noticed, tests just test and cannot affect the release
The malicious code is in the build commands, added to a random .m4 file in the release tarball (not Git) only. The payload was just hidden among the test files since that's a non-suspicious location to put a large binary blob.
@@CFSworksYou can still implement segregation of testing and building. Which to me as a simple physicist seems like the natural thing to do if you care about security. Same reason the manhattan project was broken up, as well as any other modern project with high security
@@1495978707 For sure, but I think that's done already. The tests are primarily run by the XZ project's developers and the CI system when each change is made. The backdoor is only targeting RPM+DEB package builders, and usually those buildsystems don't run the tests.
@@Rozenmorteso it lasted barely a month. Is it tasteless to make an infant mortality joke about backdoor malware? Eh, eitherway just remember that next month someone tries to say open source is less secure when even the pretty bloody advanced backdoors like this don't even last long enough to be picked up by a single release cycle.
I found a keylogger in some custom keyboards back around 1990. Some of the extended function keys were producing the wrong scan-code combinations. While working out what was going on, I found it. It was a bit of a rabbit-hole.
That sounds implausible. When keyboards weren't even USB yet, and the Internet didn't have a WWW yet, I don't see how a keylogger would do much good on a peripheral unless it was combined with a sneakernet.
@@AllAmericanGuyExpert The methods of data retrieval for keyloggers in pre-USB and pre-WWW era would differ from those today, but the fundamental utility of capturing keystrokes covertly does not. If such keyboards existed, they would be used much differently, focusing more on targeted data collection from specific individuals or systems, while relying on physical access for data retrieval. I understand the skepticism but the absence of the WWW doesn't necessarily render the concept of hardware keylogging implausible!
It appears that this backdoor fails to run properly on NixOS due to the non-standard FHS and the dynamic linker by default does not work as expected compared to Vanilla Linux. But steps are being taken in nixpkgs to downgrade xz anyway. Malware targeting vanilla linux has a hard time working on NixOS without careful forethought. I know that I am talking about security through obscurity, but sometimes obscurity just works. Also, in NixOS liblzma in not used as a dependency for sshd or openssh.
Thanks for putting this all into language that normal IT guys can understand without needing to have much experience with Linux library management and deep security knowledge.
Some more additions: As far as investigation of this problem went to this point, this backdoor was introduced around the 5.6.0 release. You can run "xz --version" to check, what version liblzma you run. If it is pre-5.6.0 you _may_ be safe (pending final analysis). But if you are on >= 5.6.0, you got the backdoor. At this point, it is recommended to downgrade to a version prior to 5.6.0. Regard this post as an immediate stop-gap measure for your system, not a proper security fix of the issue.
Manjaro received a fix. Also, do you use it on the server? Regular desktops aren't effected as tehre's a little reason to use ssh server there and with rsa key.
im gonna be honest ... i am subscribed to like 200 youtube channels. i am starting to get the most excited when you have an upload. super interesting stuff, and some of it above my head. but i have a feeling little things like this are going to be of outsized importance in the years to come.
Found a similar issue (in our Librarys) from 2017, in the City, Which is owned by the State. Doesn't help when the Admins are using out of Date Software to *Secure* those Computers.
Thanks for breaking down this issue so fast! I was made aware of it but wanted to understand how it was done and had a hard time doing that just reading the sources.
If I understood it right, it was a random compressed file which was used in a unit test to "test" some stuff. But instead it executes some code, which changes the build process (unit tests are usually ran during the build process), which then changes the linker, which hooks into a call to a third party library, which then can grant access to the malicious user. It wasn't even clear that it was a non-readable binary file. At first glance it just looked like a compressed file for a specific unit test which tests uncompressing.
Question : when was this backdoor added originally? It'd be quite interesting to compare how long advanced backdoors like this exist in Open Source compared to known backdoors added to closed source software Edit : couldn't find a date skimming through the video, but XZ 5.6 was released in February from a quick search, so at least about a month
Dont worry, the backdoors you should worry about wont even be found, only honeypots like this to keep the open source community smug, overconfident and blind to the real threats.
So this dude is listed as a developer officially for the project: "The current project members are Lasse Collin and Jia Tan. Jia became a co-maintainer for the XZ projects in 2022." Unless he's got some info on how his account got compromised, everything in the history of mankind he's ever touched needs to be reverted and his name plastered on security blogs to the top of google results. In either case, an obvious laugh to OSS maintainers in general.
It is hard to believe that his account got compromised, the oldest commit related to this incident is the one that introduced the compiled binaries, and it comes from January 23, it is a bit hard to believe that him, a co-maintainer did not notice strange commits coming from his account during the last 2 months.
I saw a few videos about something called xz pop up on my feed and I thought, I gotta get a proper rundown from Low Level Learning. Thanks for being a go-to
After this, you can be sure, people will verify all tests now, to see if that lib was "infected" the same way. Expect some funny updates the next week.
Saw the article and wasn't sure if I should click on a video about an article I read, and security issues that I've already taken care of, but this is a really great informative video. Thanks for making it!
Yes, it is that damn SystemD crap breaking subsystem isolation and hoping for the best again. But the core problem of patch reviews in open software projects and maintainer responsibility remains valid regardless.
Doesn't affect most distros (Debian, etc) unless on testing branches or rolling release. Arch Linux has already released an updated patch to repos. Sad to see this happened, but glad it was found and fixed before damage was done.
Awesome report! The best way I was able to explain this to a non technical person (and I'm barely able to follow this myself) was as follows; Imagine the index at the back of a book - the entry you want is actually on page 5 but the index has been changed to say page 11. That would be obvious, because the index page isn't in the same font, etc - so you'd be instantly suspicious. Imagine if someone then hacked the printer to print "11" instead of "5" - you'd never know and would go there instead of the "real" destination. I realise this is a massive simplification and makes it a lot more trivial (whilst also not being exactly correct) but it's the best anaolgy I could draw.
There is going to be a very long conversation about a lot of issues raised by this incident. Retirement and transition of FOSS project leadership, trust of upstream build systems and lots more. I don't have answers to any of that but I do have a simple request for distro maintainers. Can we not take a well engineered, mission critical bit of code from the security focussed OpenBSD folks and then add a bunch of extra library dependencies that most people don't need, want or expect. sshd is the last bit of software I expect maintainers to be adding random patches and additional library dependencies. ldd on arch lists 12 less dynamic libraries than on my debian stable systems but for my purposes the arch package lacks nothing. I think distro users need to push back on this stuff and not demand their pet feature be added to core libs.
@@fwfy_ Sigh... Stuff like this really needs a /s on the internet, but thanks for making me look/hyperventilate. There are 15 CVE's mentioning Wireguard, three from the year 2024, all of which are tagged as vulns in Cilium specifically.
I’ve been writing some ssh code in C using their header files and stuff. I have always said that OpenSSh is wayy too big. It is over 130,000 lines of c code. For reference Amazon was able to make a ssh library that’s as good as openssh and it was only 9000 lines. Let this be a lesson that bloated software is hard to manage and that complexity allows hackers to take advantage of it.
It wasn't an SSH library though, so your point isn't really relevant here. Complexity does create security challenges, but this wasn't complexity that contributed to this issue.
The title is misleading. The back door was not in anything relating to OpenSSH upstream. Distros patch the OpenSSH server code to include support for notifying systemd of logins. This is not built into OpenSSH, as the OpenSSH devs would never include such a silly feature. This is Linux distros shoehorning insecure systemd nonsense into everything, and yet again, it has burned users.
@@nou712 openssh should be built without PAM support if you intend to use it on a remote accessible machine. Linux and FreeBSD PAM are both hot messes.
Yeah, it's a great video but that title makes me sad, it's misleading to get more clicks, I mean I know he wants to grow the channel and all, but throwing SSH under a bus to get them?... The OpenSSH project did nothing wrong here.
And that friends was why in our shop, build files had to be as transparent as possible. They did get checked over to make sure they did ONLY what was needed and NO tricks, etc. Now the fun part was FOS libraries.
Where I work, all oss dependencies are built with cmake. If they only use autotools, we port the build to cmake. It’s crazy how much cruft is there for irrelevant platforms of the past. Truth is, most gnu code that’s “autotools only” does just fine with rather simple cmake scripts that anyone can understand.
The weird thing is that the malicious stuff is only found in the dev's hosted archive file of the source code, it is not found on github or other locations. Very odd.
Valgrind isn't just about leak detection. The main tool, memcheck, also validates that memory is initialized when it is read and that reads and writes are to valid memory. Both of these are usually far more serious than leaks. In this case it detected an invalid write below the stack pointer.
If I read it right this ‚just a guy‘ actually was a maintainer of the fedora linux distribution and the malicious actor managed to become the maintainer of the XZ library by posing as an active developer for years. The ‚just a guy‘ did valgrind memory tests to verify if the library is safe to include and it failed. The malicious actor tried to convince everyone the valgrind errors are GCC bugs to hide this, and ‚just a guy‘ even helped fixing the memory errors before finding out the true nature of the patch. Edit: I can no longer find the mailing list post that confirms the version above. The newer posts by Andres Freund (the ‚just a guy‘) tell the story a little bit differently. See my response below for sources
Part of the exploit was rolled into released source tarballs on the Github website (but not the Git repo IIRC). If upstream package maintainers used the provided tarballs to build the binary packages and then made the signatures available, they would be 'trusted' regardless. In which case, you're screwed. There are multiple issues the perpetrator exploited to plant this backdoor; there's going to be a ton of lessons learned.
Do I recall correctly that early versions of the C compiler had a cooperating backdoor with the login program that would install a hard to spot backdoor into the login program? This was in the early, early UNIX systems. The hack you mentioned in the video is incredible. Dynamic linked libraries can have all sorts of interceptors & interposers placed on the, by simply having suitable LD_LIBRARY_PATH setting that refer to your version of an API. I used this trick onetime to create an interposition on malloc & free that would add additional header and footer memory to malloced storage, so the memory had sentinels at both ends to detect overwrites, and I think I also had it link all of the malloc'd memory together in a list so an analyzer could scan through the malloc list and find any areas that had been overrun, and compute some statistics about how much memory was allocated, maybe how long it was allocated for, etc. Probably around 1992 or so.
At least theoretically the self-propagating backdoor you're describing was described way back when in an academic paper I remember being told about, "Reflections on Trusting Trust", or something like that, and it's been a sort of... matter of conspiratorial debate whether or not it actually happened, at least that's how I remember it.
@@05MatzYes, it was from long ago -- maybe the 80s when the description came out. The source code for Version 6 and Version 7 UNIX is available (including for "cc'"), so if the hack was still there , it should be observable. It was one of the original UNIX guys, I think Brian, but maybe Dennis who talked about it and provided some details about how it worked. [as I recall]
Andres Freund isn’t a security researcher, but he’s not “some guy” either 😅 He’s a principal engineer at Microsoft and a PostgreSQL developer. (Not a developer who uses Postgres, but a developer who builds and maintains Postgres itself.)
This feels like it'll be such a shock to open source projects... I would expect that other project would start scrutinizing their code and contributions more deeply in the next couple of weeks and we'll be hearing more surprises
Seems almost as if someone or a group of very knowledgeable engineers is deliberately targeting open source software. And it's not just technical, this also targets the way changes to open source software are usually review, validated and merged into the code base as well. The whole change process for open source software seems to be in need of an upgrade.
Oh, certainly. I'd bet that this was the work of some kind of three-letter-agency* from some country somewhere. Probably better than even money that it was a nominally 'friendly' country to you and/or me (in my case Canada, so the whole usual suspects of Five Eyes/etc.)! Not so friendly when they show their hand at their persistent efforts to make everyone in the world unsafe in the name of their stupid squabbling power games. *Name may not be exactly three characters, but is probably an acronym.
@@bobmorgan1575 I don't think they'd do something with this much risk of getting caught, just to besmirch the security of a competing platform. I think whoever did this planned to USE the backdoor, not for it to only exist as FUD.
is it only when you enable compression or just in general. if it compromises the binary in the build process does that mean it only effects ssh thats recompiled from source code or does it effect every binary that comes with every system? (sorry im unfamiliar with how that kinda stuff works)
every binary that was compiled after the change was introduced is compromised, which may or may not include that one in your system depending on how old this is
If the source code was compromised then binaries where also comprised. But the issues was only created 3 weeks ago so a lot of systems will never have received the update, also you would have to be a ssh user to be affected, because ssh is not something that is enabled by default. This is my take anyways from what little info was in the video.
Seriously guys thank you for watching, the channel has been exploding these last few days. ( come learn C at lowlevel.academy 🥺)
Keep bringing this type of videos😽😽
Seems like security issues have been exploding these last few days!
Keep up these videos, they are amazing!
I think Freund rhymes with Freud. Any Germans here quickly correct me if I'm wrong.
0:14 AFAIK OpenSSH does not directly depend on lzma. Some distros (e.g. Debian) patch libsystemd into OpenSSH for readyness notification, which in turn depends on liblzma. Upstream ssh should be safe. More on this in the HN thread.
The guy who found this has just been elevated from "just a guy" to "security researcher" by action alone
100%
I wonder where he got the skills to understand this stiff
@@gingeral253internet and motivation bro
It’s very very very very disrespectful to say he is just a a guy. Foolish commentary in that regards
@@gingeral253hi definitely is an accomplished developer who stumbled upon the Valgrind issue and was motivated enough to investigate and then find it. This guy probably wanted to help fix a memory issue and had the perseverence to go to the bottom of it.
Kudos to this „just a guy“ who probably just wanted to help out an open source project in fixing a memory leak and performance issue and then probably spent weeks of his time getting to the bottom of this.
This is what makes all of us safer. He deserves a medal or something.
The last commit related to this was three weeks ago, but I can't check when the first commit was because GitHub has quarantined the repository
Nobel Prize honestly
Just think if he had found such a bug in Windwos he'd have been arrested and charged as opposed to being thanked.
@@cigmorfil4101 well, Microsoft has a bug bounty program, as most big companies have nowadays. The ‚just a guy‘ is Andres Freund, some blogs call him ‚microsoft security researcher‘.
@@cigmorfil4101 What is one single shred of evidence that leads you to believe that? This has gotta be one of the most absurd things I've heard.
BLAZINGLY FAST NEWS REPORTING
BREAKING NEWS
Could definitely add that to title lol
@_JohnHammond Reverse engineer it and make a video about it!
every time i read your name i read it in the moonbase alpha tts "john madden" voice
X.x What interesting events, a tinfoil hat might start to think supply chain issues, and notice the same kind of ticks in other things like 'heartbleed bug' and just how 'slightly off', but not enough it was to be spotted by all but the most ocd or diligent person for their own thing.
Just sayin.... Maybe Open source is more at danger of being infected by forign bodies than closed source, and taken for safer .. when most don't have a single clue about wtf is going on, or just why an issue or even performance hick-up exists. Who would ever be absurd enough to question 1-2 second single hick-up in a sign in process for something meant to exist on the internet where that delay would be unreasonable to impossible to spot in the live environment.
Speaking of cool things, Isn't it great that there are so many easy to access cloud-based hosting solutions meant for personal & small business users? That convenance is great, just have to deal with a tiny bit of internet lag because tech stuff. I mean it would suck if they all used a common supplier that managed access in some way or something, or data security... *cough cough*, because some big big target would be pained on the backs of such a supplier and be closely watched.
Man, the amount of qq from governmental bodies about data encryption and traffic routing has sure gone down, I bet they only realized how foolish it was for them to keep on that. I guess all is well, and the days of the governmental bodies pulling overly complicated & deep state affairs on the people they care about is over for our own people right?
Important Clarification (since I feel this isn't clarified): upstream OpenSSH doesn't use liblzma, however many distros like Debian patch OpenSSH to use SystemD Notifications through libsystemd, which in turn uses liblzma. Distros like Arch (which don't patch OpenSSH) or distros without SystemD like Void should be fine with regards to SSH (however most distros are already downgrading xz anyway for obvious security reasons)
Source: the latest Arch News post regarding this backdoor
EDIT: to quote directly from the Arch news post:
"Arch does not directly link openssh to liblzma, and thus this attack vector is not possible"
@@mx338but if your systems are sufficiently firewalled... What is the exploit going to do?
This is not a good situation, but it won't break the world either. Vulnerabilities are found almost every day it seems in so much software.
@@Finkelfunk Install Gentoo.
@@mx338 You did not understand a word from this right. This relates only to very latest versions, not stable ones so NO servers are not compromised...
@bzuidgeest you're right, ideally you do not expose SSH to the internet without restrictions, I run everything on prem for work and don't allow SSH from the internet. However in cloud computing that's often different, if you run and AWS EC2 or any other cloud VPS, SSH is going to be exposed to the internet by default.
@@Finkelfunk Is this what the children call a "skill issue"?
As horrifying as such backdoors in widely used software is, the engineering behind such exploits is insanely impressive and creative
Using a single makefile line that looks weird but not totally out of the ordinary is so creative. On the other hand how was this mf able to commit compressed binaries without getting nuked by a maintainer, i do that on accident occasionally on my branches and it draws the aggro of every devops guy in a 10 mile radius when i submit the PR
@@twentylush LMAO draws the agro bro im stealing that one
@@twentylush Youre just simply a devops guy magnet
@@SuperBotcreator MrsBest, Penguins1 & 555NipperDog's farts are better than you 😋
@@twentylush they passed it off as a test case. It makes sense at a quick glance, having a premade file that _should_ decompile properly, then using that in testing to see if everything works as expected
it's just that the file wasn't only used for testing
I can already read online people saying that this is proof that open source code is less secure than proprietary
On the contrary, the fact that it was caught, and caught relatively quickly, shows that open source is more secure against these kinds of backdoor attacks
Out of sight out of mind
"this" was caught but there could be many more backdoors which will never be discovered. Imagine if a war breaks out between countries. Of course no government agency would trust proprietary software during an active war. In that scenario, the country that managed to put backdoors in open source software will have immense advantage.
Indeed.If someone can do this in code that's opening visible, imagine the shit that goes on behind the curtain in non-opensource crap. (Solarwinds comes to mind.)
Also, we have a clear method to identify and track the malicious actor's influence. We know what account they're using, and can thus see every open source project they have touched.
@@aldproductions2301 with that particular account, maybe. People can make as many accounts as they like
How many more security issues are going to be found this week?!
Life is a security issue
They asked ChatGPT
security issues are just skill issues in disguise
all of them.
@@bigboi1004 and unsafe language issues passed off as skill issues
It's not just SSH. The dev seems to be suspect in way more packages.
After this, he better be. This is really well designed malware. The ways they are cleverly hiding it, all the involved packages, the linker, removed the symbols from the malware in xz, using the testing branch, those mentioned in the video, or the fact that it's not even in libssh, and it only works if you run through systemd which systemd would be compiled to use xz. That's really well orchestrated, that's gov malware I think.
is his github account still active?
@@brians7100 Some of his suspicious push requests date back over 2 years, so its possible this isn't the first time he has tried to do something like this, just the first time he got caught.
@@brians7100 Even if he isn't active anymore, the damage is done and needs to be detected and reverted in each repository this guy has committed to.
I bet it'll last some time as an active issue until things are resolved for most packages affected, and many machines will be compromised until then
@@brians7100recently suspended
Kudos to the person who found this. His modesty does him credit, but the realization that something was amiss, the desire to delve into it, the analytical process, depth of research, and the willingness to share what he discovered with the wider community - totally aligned with the OSS ethos - shows he definitely has the right mindset to be a security researcher and has hit the ground running. Hats off, sir; bravo!
as a sidenote from what I gathered: The person who "contributed" this backdoor was not just some person who randomly came out of nowhere with a Merge Request. It was someone who contributed to the project for an extended period of time to a point where they themselves became a maintainer (not the main one, but projects like this often have multiple).
What a poor misuse of trust and time to achieve this
@@JordanPlayz158 The contributor probably thought someone would pay them a lot of money to achieve this.
@@wasd____ It was his mission.
@@JordanPlayz158they were obviously being paid already for this. Plenty of maintainers are on the payroll of Google, Microsoft, Redhat etc. This is just the first time a malicious group decided to go that route as well
@@Pharoah2 just the first time? Lol do you think that between the thousands of FAANG+ employess none of them is on the payroll of some 3 letters agency?
damn, NSA taking a bunch of loses recently
They'll always have windows!
Bro not every vulnerability ever is the NSA?
@@kreuner11 lol you’re taking his comment too literally
@@kreuner11
______
Joke ---/ you \--->
NSA is a reasonable suspect, in all honesty.
Idk if I would call Andres Freund “some guy” haha he’s a Postgres contributor/developer and Principle SWE at Microsoft. I get your point though. Not technically a security researcher.
This channel is great for my daily dose of anxiety
Yeah. Just got the news about it 9h ago. My Manjaro installation has the aforementioned versions installed and now I don't know what to do. Rollback, or directly wipe the system and return to some more newbies OS like Mint.
lol
You are not nearly anxious enough.
Loving these security vulnerability breakdowns man! I can only imagine the effort and background knowledge it takes to put content like this together, thank you.
Thanks! Please make a video on the topics of object file/linker crc and most of the topics you talked about, A deep dive will be amazing!
1:12 "He's not a security researcher, he's not a malware reverse engineer, he is just a Freund."
He is definitely an experienced developer
That is definitely one of the most crazy, complex, and sophisticated backdoor injection attempts I ever seen in my life. The engineering behind it is very impressive. The guy who discovered that deserves a reward, he just literally saved the world
After this report there were some extra findings that people might feel valuable.
1. Some time ago the only mantainer got a lot of pressure from different users to accept another mantainer to the project.
2. The binary files in question were compressed files that are usually made to test that the decompressing tool was working, in this case the malicious tests were two, one for a large file, and another for a corrupted file. As you can see both made for making it difficult to find that there was hidden code there.
3. The malicious mantainer got enough trust to be able to sign the distributed tars with malicious code and to contact linux distro mantainers to presure them to update to the backdoored version, the attacker even sent a patch to the google repo that was harmless by itself but a requirement for the exploit.
4. After the backdoor was created random accounts submitted prs to
different projects to update to the vulnerable version.
This was a large well orquestated attack that was most likely planned by more than one person and only discovered due to it having performance problems and certain bugs, otherwise we might have never noticed.
honestly sounds like a state actor to me.. and with the names being used I suspect CCP.
@@Dratchev241 Eh, if you're going to do an intelligence operation involving names but no face-to-face contact, you'd probably use a foreign name. I'd definitely say a state actor, but frankly, it could be anyone. I was leaning towards someone Western but smaller, but I really have nothing to base that on.
Yeah this was definitely orchestrated and planned for a long time and took a lot of planning. They just got unlucky that someone noticed the performance decrease and investigated and found it by pure accident.
@@Dratchev241 Hah, because obviously Chinese intelligence agencies would use a fucking Chinese name to do this, right?
If anything this was probably done by american intelligence agencies using an asian name to generate propaganda headlines if it didnt work.
@@Dratchev241 You don't need a state actor to create and coordinate multiple accounts. There is no proof that those multiple accounts requesting the update on client programs belonged to different people. It only requires coordination. But one person can easily coordinate with themselves.
Your point 1(pressure on the single maintainer of a core library to accept other contributors) is actually what you should expect. You don't want a single unsupervised individual to be the alpha and omega of such a central library with zero backup. That's what actually scares me about OSS and Linux. There are so many core technical libraries that are maintained by a very small group of persons. Any rogue in one of those core groups can cause wide damage. There way too little people interested in working in those dark places where you get no recognition when everything works and all the blame when something goes bad in another software just because someone else used your library in a way that it was not supposed to be used.
Point 2 is the actual issue: porous build process.
Test files stored in the test directory should never find a way into the compiled binary. There should be a stronghold protecting the source path and compilation tools so that nothing outside of that scope can get into the binary files that are signed and published.
Bullet dodged. For real. This could have been the worst money grab backdoor by far. It's literally in every system. It's especially scary that the project owners approved the compiled binaries. Hopefully it's not a maintainer behind this.
It's literally the project owners or a representative of the project owners org.
Both of the main devs that are the contact points for package maintainers have had their GitHub accounts suspended reportedly
It sounds like the original, single maintainer was overloaded and started passing off duties to another contributor who "helpfully" stepped in recently to reduce the load. Fast forward a few months and it looks like this helpful contributor was (likely) a state agent...
@@troflquestion is which state agent... I would say in likely by order Mossad, then the NSA/CIA/MI6 then FSB and or MSS-CCP
They aren't compiled binaries. They're compressed files. Which is totally normal for testing a compression tool.
What was missed was potentially looking into the contents of the compressed files, which I know very few people would bother to do, or looking at the quality of the new tests. If you're just making existing code paths faster, why do you need new compressed files?
Edit: auto-incorrect
You realize that this backdoor was caught but so many other could be successfully installed in open source code without our knowledge and without extremely deep code reviews. Though I would expect that other project would start scrutinizing things more deeply in the next couple of weeks and we'll be hearing more surprises
It is slightly more subtle, from what I understand. It is not that openssh uses liblzma, but liblzma is used in systemd. On systems where openssh is patched to use systemd as well, you end up with a security issue. This appears to be limited to the combination of x86_64 and linux and systemd. That is still a significant fraction of all linux systems.
openrc saved the day
That does not surprise me at all.
Systemd is one giant security issue. The biggest security issue about it being the megalomanic ego of its creator.
@@thelvadam5269 Ever heard of black humour? 👹
@ok-tr1nwthis is a very specific backdoor that only affects software that uses lzma and does RSA with this specific library. How many programs fit that description...
From the Openwall discussion:
I think that distros should be very careful when they start patching openssh in general.
so if I'm understanding this, the intent is to be able to use a specific key that doesn't have to be installed [legitimately] on a system (no direct attack necessary) to effectively gain "authorized" access to a system. to simplify it, this behaves like a master key or a master lock (iykwim).
Exactly
Not exactly technically speaking, but the exact same effect in practice. The difference, fortunately, being that this way the attack can only happen under specific conditions. For example, some environment variables can make this not run, since the attack depends on specific code being injected. Of course, even with this stroke of "luck" it still is one of the most serious attacks/vulnerabilities found in the last couple of years.
ikwym wrt lpl reference :)
Apparently, this leads to RCE rather than just auth bypass
The malicious commit was made Feb 23 and entered xz-utils stable on Feb 24. Detected in Debian-unstable.
5 points for early detection, before it should have been distributed to production anywhere?
There was no "commit". The hack was only placed in a non-rev controlled tarball that other projects then imported.
Andres Freund, who found and reported this, should be called "The XZorcist".
Crazy to think the test can change the build. This should be clearly separated.
I agree. Even the build systems I maintain for PHP code run tests and then (if the tests passed) checks out the Git repo again in order to do final deployment actions on a clean slate.
it doesn't, really... the test fixture just exists in the repo doing nothing, but with the modified build script, that test data is used to reconstruct the malicious code. in fact the test fixture in there isn't even used at all in a test (which should have been sus)
the actual git tree of the project doesn't have the M4 line which triggers this. The M4 line was added to the public source tarball post release by someone who had keys to do that.
Yeah, for sure a no-no, but perhaps this was by design. Harder to notice things when you assume everyone follows the rules.
Right??? Build processes shouldn't allow random extra .o's not generated from sources to appear :O
Apologies if I missed it, but it should be clarified that this did not make it into any production releases. The fact that it was caught before release is a demonstration of the strength of the open source model.
What about Manjaro?
I'd say we got very, very lucky that someone noticed some odd behavior and went 10 extra miles tracking it down.
In fact I'd say this was a pretty big failure in terms of the model. As it looks like there is there was no verification that the source being used was even the same as the source in the repository(You know, the stuff people usually check for things like this). If it had been complied with the repo then it wouldn't have been an issue at all(vestigial, non executing code aside).
Kind of clever really. Leave out the easiest to see bits from the source. Slap them in the release tarball and as long as the behavior isn't too different from the original nobody notices(and if it is you can probably just patch and blame it on something else like the did in 5.6.1)
So all the normal checks and balances that would normally catch this in OSS(as far as I know) just don't work here. Only weeks after release and by blind luck did this get caught and that's not something to celebrate.
Say what you will about closed source, but at least the back doors in their software are intended by the company ;)
OpenSUSE was vulnerable. MicroOS was vulnerable as a result. It's an os built for containers targeted at scalable services and bigger entities. Curious how many kube clusters got taken over.
@@haifutter4166 As someone stated here Arch doesn't use patched OpenSSH and since manjaro upstreams directly from Arch, I would say it didn't affect it. For Fedora on the other hand, it affected Fedora 40 beta in a few flavours, rawhide and silverblue (iirc?) being two of those. Older versions did not get affected
@@gregoryreimer869 A rogue developer could do the same in closed source software, and we'd have even less of a chance to catch them.
OpenSSH doesn't use liblzma directly. Some GNU/Linux distros patch it to use libsystemd and that uses liblzma.
And most stable distros like Debian and Ubuntu LTS, do not use the latest version of programs, so not affected.
@@rj7250a And the OpenSSH upstream is unaffected since systemd is not present there. :) (OpenBSD)
@@rj7250a it was officially in debian-unstable, so it was gonna be in the next main release
Well noted
Dude I'm loving this exploits news and guide. You're really helping me get a better understanding of these things!
Wow - this is absolutely crazy. Never would have thought that this passes a review. Obfuscated code… I have thought several times - e.g. when people talk about hardware wallets’ software being open source - you dont know what ends up on the device, and even if the source is fine, you dont know what happens during the build process. A case like this shows me this suspicion isnt that unrealistic. Thank you for the video - I subscribed.
These videos are great. But please add more context or explanation about how these exploits could be used. I forwarded your previous video on kernel exploit to the management and they came back saying "we dont understand the implications from this"
It’s pretty simple, they could ssh into your server. What you have running on that server and whether or not they could they could perform privilege escalation would be required knowledge to determine what the bad actor “could do”, and from that would you determine the implications.
both of these are only doable if the attacker already has access to the network and most companies have an encrypted secure vpn and the ssh would be in the vpn right.. ?, so both are relatively harmless from external attacks, they do have to worry about sabotage from inside. Even then records would be kept and the culprits found.
How does management feel about everyone in the company who uses those servers having access to all of their files, including customer data? From an insider threat perspective, that's what we're talking about. A larger concern would be if one site gets hacked, then everything on that server would be hacked.
@@italianbasegard that's doesn't provide enough actionable context. It should include "in bold" the identifiers of the compromised releases. It wouldn't take too much effort to highlight that this only about "RSA's implementation" + "liblzma [...] tarballs for 5.6.0 and 5.6.1" and NOT just "SSH libraries".
@@italianbasegard I don't think they would require privilege escalation.
From my understanding of the video, they could hypothetically log in as root, using their key, and get root access.
That’s why I say code (and commits) must be as dumb, clean and understandable as possible. No overcomplication.
The backdoor payload was hidden among dozens of innocent compressed test files, and the hook to execute the script that embedded the payload was slipped into a makefile in the release tarball but not committed to Git. We absolutely should be in the habit of diffing our release tarballs vs. the Git tags from now on, but we're going to need to come up with a smart way of analyzing any high-entropy files checked into Git for potential hidden payloads.
@@CFSworks I feel like its a failure of keeping up with the times though! this exploit is a complete dead-end if you policy-out artifacts in code repos and instead only pulling them in containerized environments when the "tests" are actually run. Not to say its all the maintainers fault, I mean the OG seemed pretty burnt out and overloaded, and there's not a lot of up-to-date devops guys with a golden-heart and clear schedule to bring these OS projects to the modern ci/cd era
@@twentylush I suspect that an attacker trying to pull off an attack like this in the future would just switch to hiding the payload in a non-artifact binary file instead. Perhaps a base64 chunk hidden in the metadata of the project's logo, or a loose object under .git. There are too many opportunities for outright steganography here. :/
@@CFSworksContinuous integration should be the source of tarballs FFS. Nobody should be uploading those manually.
@@absurdengineering Definitely. But here we had a malicious maintainer, so even with auto-upload of the git-archive output, they would have just secretly replaced the tarball 3 seconds afterward with the backdoored version.
If it was missed in popular OSS, imagine how easy it is for skilled bad actors to sneak these things into closed source products!
Especially when the bad actors are the companies who make the products!
Windows has left the chat.
I'm not sure if xz could be considered popular OSS. Popular package, maybe, yes. But not project itself.
Microsoft employees have names and adresses, they'd be found on an audit, fired, blacklisted and possibly jailed for something like this. Open source maintainers? Good luck going after a dude you're not even sure exists on the other side of the world.
That is actually a huge problem of OSS. Many "popular OSS libraries" (aka core building blocks used by hundreds of projects) are actually not popular at all. They are maintained by 1 or 2 people at best and everyone else just blindly integrates those libraries into their project trusting that some magic being has actually independently reviewed and validated what those guys did. And for that exact reason those libraries are very easy to infiltrate and alter for rapid damage on a lot of projects at once. There is no actual incentive outside of pure benevolence for someone to actively participate in securing some of the most technical and sensitive core blocks of the OSS architecture.
Being open or closed software is irrelevant here. It's an organisational issue of the dev teams. Everybody wants reviewed binary that they can trust but nobody wants to be the one that spends time doing the actual review.
The problem here is accepting commits that run binary executables. That should have been suspicious from the get go.
No. The problem is patching security sensitive programs to use 3rd party libraries and, of course, not letting the most important daemon on the system (systemd) that handles everything using 3rd party libraries. There's a reason people are monitoring openssh a lot more then some compression library and this problem wouldn't have been found for who knows how long if a persons OCD hadn't kicked in to thinking a program takes too long to open (half a second).
It shouldn't have happened and on systems that didn't try and screw around just to make a config file look prettier, it didn't.
Yeah. Why not commit a BIOS firmware update at this point? lmao
@@MrFujinko Actually... I wonder, if someone could Chainfuck EFI. You just need a lib, which is most likely run as root. So the same crap, but write code, which drops a EFI binary into /boot and hooks that into NVRAM and the loader, which in turn reloads or mimics GRUB. Part of this EFI binary goes straight to Ring -3 on AMD or Ring -1 on Intel, acting as a hypervisor. The PC boots normally, but the Chainfucker is in control of the system by opening ports outside linux control.
It's just a theory though, I am no expert.
Yea. I even look ten times if someone adds a png or font to some kind of front end environment. Never not be paranoid.
@@bl8de3 TBH with how easy it is to Steghide data inside an image file, that's not unfounded to suspect malicious info could be hiding anywhere.
With every passing day, templeOS is starting to look more and more reasonable
Boundless switch
You’ll pass up the BSDs?
yes there is a lot to learn about templeOS, for example auditing the code that runs is easy and fast to do. no compiling.
they said that terry was mad, but who is going crazy is society...
@@CEOofGameDev society was always crazy you're just accelerating at a slower rate than the people around you that's why you can actually see it
I think I just got a good reason why I stopped using a rolling release! Thanks for this info!
Arch was not affected.
@@Cobalt985That's not the whole truth. Arch was affected but OpenSSH on Arch was not affected. Quote from Arch news: "Arch does not directly link openssh to liblzma, and thus this attack vector is not possible". But in xz package history you can clearly see that affected upstream versions 5.6.0 and 5.6.1 has been released by Arch.
Still and forever user of Arch 😍
unpopular opinion: workflow 1 pulls the repository, builds everything and then tests everything, workflow 2 pulls the repo without using workflow 1's output, builds and publishes, that way malicious code must be in the build commands or toolchain where it will be noticed, tests just test and cannot affect the release
The malicious code is in the build commands, added to a random .m4 file in the release tarball (not Git) only. The payload was just hidden among the test files since that's a non-suspicious location to put a large binary blob.
Just read this as I squeezed a gripping fart. Contemplating this binary
@@CFSworksYou can still implement segregation of testing and building. Which to me as a simple physicist seems like the natural thing to do if you care about security. Same reason the manhattan project was broken up, as well as any other modern project with high security
@@1495978707 For sure, but I think that's done already. The tests are primarily run by the XZ project's developers and the CI system when each change is made. The backdoor is only targeting RPM+DEB package builders, and usually those buildsystems don't run the tests.
@@1495978707 Absolutely, but I think that's been done already. The RPM/Debian buildsystems don't typically run the test suites.
When was the backdoor mechanism committed?
Looks like it was added in Feb 24, 2024 for version 5.6.0 and 5.6.1 of xz
@@Rozenmorteso it lasted barely a month. Is it tasteless to make an infant mortality joke about backdoor malware?
Eh, eitherway just remember that next month someone tries to say open source is less secure when even the pretty bloody advanced backdoors like this don't even last long enough to be picked up by a single release cycle.
@@felixjohnson3874 nah he found it by luck
@@felixjohnson3874 moving forward I feel like we need more eyes looking at the skeletons, we kinda got lucky "just a guy" stumbled upon this one
I found a keylogger in some custom keyboards back around 1990. Some of the extended function keys were producing the wrong scan-code combinations. While working out what was going on, I found it.
It was a bit of a rabbit-hole.
Wow that sounds very interesting, could you give us more details?
That sounds implausible. When keyboards weren't even USB yet, and the Internet didn't have a WWW yet, I don't see how a keylogger would do much good on a peripheral unless it was combined with a sneakernet.
@@AllAmericanGuyExpert The methods of data retrieval for keyloggers in pre-USB and pre-WWW era would differ from those today, but the fundamental utility of capturing keystrokes covertly does not. If such keyboards existed, they would be used much differently, focusing more on targeted data collection from specific individuals or systems, while relying on physical access for data retrieval.
I understand the skepticism but the absence of the WWW doesn't necessarily render the concept of hardware keylogging implausible!
@@AllAmericanGuyExpert There was Novell etc.
It appears that this backdoor fails to run properly on NixOS due to the non-standard FHS and the dynamic linker by default does not work as expected compared to Vanilla Linux. But steps are being taken in nixpkgs to downgrade xz anyway.
Malware targeting vanilla linux has a hard time working on NixOS without careful forethought.
I know that I am talking about security through obscurity, but sometimes obscurity just works.
Also, in NixOS liblzma in not used as a dependency for sshd or openssh.
Thanks for putting this all into language that normal IT guys can understand without needing to have much experience with Linux library management and deep security knowledge.
Some more additions: As far as investigation of this problem went to this point, this backdoor was introduced around the 5.6.0 release. You can run "xz --version" to check, what version liblzma you run. If it is pre-5.6.0 you _may_ be safe (pending final analysis). But if you are on >= 5.6.0, you got the backdoor. At this point, it is recommended to downgrade to a version prior to 5.6.0. Regard this post as an immediate stop-gap measure for your system, not a proper security fix of the issue.
A relief if true. Ubuntu 22.04 is 5.2.5
Allready checked. Fck
I use Manjaro. Maybe should return to something like Mint.
@@haifutter4166 Manjaro already has an update for the affected packages.
Download of lib32-xz (5.6.1-2) finished
Download of xz (5.6.1-2) finished
Manjaro received a fix. Also, do you use it on the server? Regular desktops aren't effected as tehre's a little reason to use ssh server there and with rsa key.
I read that you shouldn't run `xz --version` or `xz -V`, use brew command instead
im gonna be honest ... i am subscribed to like 200 youtube channels. i am starting to get the most excited when you have an upload. super interesting stuff, and some of it above my head. but i have a feeling little things like this are going to be of outsized importance in the years to come.
This doesn't seem like coincidence anymore
Never was.
What? Why would it be? A backdoor is intentional.
There is no coincidence, just malicious people out here. Nothing new. Thankfully, there are plenty that aren't and will counter it, every time.
@@terminalvelocity4858not just people, countries too.
I've fellt like the heart beat / bleed was delibret in opensll
Holy shit, get ghidra we are going for a wild ride
This is the equivalent of bystanders watching comic book heroes and villains fight it out in the sky
Found a similar issue (in our Librarys) from 2017, in the City, Which is owned by the State.
Doesn't help when the Admins are using out of Date Software to *Secure* those Computers.
Great! Now it's back to smoke signals!
Ehh but smoke signals aren't exactly encrypted
They can be @@9a3eedi
@@9a3eedi blackmailed by little timmy next door
Damn what is happening this week.
shit's going down this week
Like...?
@@terminalvelocity4858 And Subscribe.
bug in apple computers, linux exploit
Thanks for breaking down this issue so fast! I was made aware of it but wanted to understand how it was done and had a hard time doing that just reading the sources.
How can a public library get a commit of a non-readable binary file approved???
If I understood it right, it was a random compressed file which was used in a unit test to "test" some stuff. But instead it executes some code, which changes the build process (unit tests are usually ran during the build process), which then changes the linker, which hooks into a call to a third party library, which then can grant access to the malicious user.
It wasn't even clear that it was a non-readable binary file. At first glance it just looked like a compressed file for a specific unit test which tests uncompressing.
Sneaky.@@joost00719
Do you know that Linux is full of binary blobs and there was a project called Linux libre that tries to exclude all the binary blobs from linux
Question : when was this backdoor added originally?
It'd be quite interesting to compare how long advanced backdoors like this exist in Open Source compared to known backdoors added to closed source software
Edit : couldn't find a date skimming through the video, but XZ 5.6 was released in February from a quick search, so at least about a month
Presumably the lzma project is in git so you could clone it and find the precise commits. Then you would know the dudes handle too
Correct, and the "binary tests" that were added weren't even used
It's quite ingenious
About 3 weeks ago the relevant xz file was updated.
Dont worry, the backdoors you should worry about wont even be found, only honeypots like this to keep the open source community smug, overconfident and blind to the real threats.
@@Bramble20322 Grandpa, take your meds. The nurses said they'd press charges if you have another episode.
So this dude is listed as a developer officially for the project: "The current project members are Lasse Collin and Jia Tan. Jia became a co-maintainer for the XZ projects in 2022." Unless he's got some info on how his account got compromised, everything in the history of mankind he's ever touched needs to be reverted and his name plastered on security blogs to the top of google results.
In either case, an obvious laugh to OSS maintainers in general.
It is hard to believe that his account got compromised, the oldest commit related to this incident is the one that introduced the compiled binaries, and it comes from January 23, it is a bit hard to believe that him, a co-maintainer did not notice strange commits coming from his account during the last 2 months.
Good thing i haven't updated any of my VMs in years
I saw a few videos about something called xz pop up on my feed and I thought, I gotta get a proper rundown from Low Level Learning. Thanks for being a go-to
Marvelous exploits, I should learn to exploit testing.
Exceptional videos and shorts, man. Keep up the great work!
Disguising the backdoor as a files in the tests directory is...pretty new and kinda scarily big brain wtf
Obfuscation by Obscurity
After this, you can be sure, people will verify all tests now, to see if that lib was "infected" the same way. Expect some funny updates the next week.
Saw the article and wasn't sure if I should click on a video about an article I read, and security issues that I've already taken care of, but this is a really great informative video. Thanks for making it!
OpenSSH doesnt use directly this library, this video is a bit misleading... but other than that, nice vid
Yes, it is that damn SystemD crap breaking subsystem isolation and hoping for the best again. But the core problem of patch reviews in open software projects and maintainer responsibility remains valid regardless.
andres freund is tbf not "just a guy", he's a microsoft principal software engineer and one of the main contributors to postgresql
Great stuff , i want to see your full break down when you upload it .
That's a crazy backdoor. Wouldn't surprise me to see some other popular packages compromised in a similar way.
Doesn't affect most distros (Debian, etc) unless on testing branches or rolling release. Arch Linux has already released an updated patch to repos. Sad to see this happened, but glad it was found and fixed before damage was done.
Awesome report! The best way I was able to explain this to a non technical person (and I'm barely able to follow this myself) was as follows;
Imagine the index at the back of a book - the entry you want is actually on page 5 but the index has been changed to say page 11. That would be obvious, because the index page isn't in the same font, etc - so you'd be instantly suspicious. Imagine if someone then hacked the printer to print "11" instead of "5" - you'd never know and would go there instead of the "real" destination.
I realise this is a massive simplification and makes it a lot more trivial (whilst also not being exactly correct) but it's the best anaolgy I could draw.
There is going to be a very long conversation about a lot of issues raised by this incident. Retirement and transition of FOSS project leadership, trust of upstream build systems and lots more. I don't have answers to any of that but I do have a simple request for distro maintainers. Can we not take a well engineered, mission critical bit of code from the security focussed OpenBSD folks and then add a bunch of extra library dependencies that most people don't need, want or expect. sshd is the last bit of software I expect maintainers to be adding random patches and additional library dependencies. ldd on arch lists 12 less dynamic libraries than on my debian stable systems but for my purposes the arch package lacks nothing. I think distro users need to push back on this stuff and not demand their pet feature be added to core libs.
My man, you’re the best. Love your videos
Shit like this is why I stopped exposing any ports aside from Wireguard to the internet.
BREAKING NEWS: new wireguard RCE exploit found
@@fwfy_ Sigh... Stuff like this really needs a /s on the internet, but thanks for making me look/hyperventilate. There are 15 CVE's mentioning Wireguard, three from the year 2024, all of which are tagged as vulns in Cilium specifically.
@@fwfy_ 🤣
you made sure to disable ipv6 right?
What about malware that's already in your network? There could be a lot more that we just haven't found yet.
I'm kind of impressed how they managed to sneak that in.
I’ve been writing some ssh code in C using their header files and stuff. I have always said that OpenSSh is wayy too big. It is over 130,000 lines of c code. For reference Amazon was able to make a ssh library that’s as good as openssh and it was only 9000 lines. Let this be a lesson that bloated software is hard to manage and that complexity allows hackers to take advantage of it.
It wasn't an SSH library though, so your point isn't really relevant here. Complexity does create security challenges, but this wasn't complexity that contributed to this issue.
I am against the use of shared libraries, I think nowadays they cause more problems than benefits.
Crystal clear explanation thank you!
The title is misleading. The back door was not in anything relating to OpenSSH upstream. Distros patch the OpenSSH server code to include support for notifying systemd of logins. This is not built into OpenSSH, as the OpenSSH devs would never include such a silly feature. This is Linux distros shoehorning insecure systemd nonsense into everything, and yet again, it has burned users.
how to learn/understand all these stuff?
It's not just a systemd issue, it's an issue for PAM as well which doesn't just affect linux either.
@@nou712 openssh should be built without PAM support if you intend to use it on a remote accessible machine. Linux and FreeBSD PAM are both hot messes.
Yeah, it's a great video but that title makes me sad, it's misleading to get more clicks, I mean I know he wants to grow the channel and all, but throwing SSH under a bus to get them?... The OpenSSH project did nothing wrong here.
Just found your channel thanks YT algo. Your presentation is excellent.
*Claims to not be a security researcher or a reverse engineer*
*Proceeds to research security and discover an open source backdoor exploit*
And therein lies the magic of open source; the opposite of security by obscurity.
And that friends was why in our shop, build files had to be as transparent as possible. They did get checked over to make sure they did ONLY what was needed and NO tricks, etc.
Now the fun part was FOS libraries.
Where I work, all oss dependencies are built with cmake. If they only use autotools, we port the build to cmake. It’s crazy how much cruft is there for irrelevant platforms of the past. Truth is, most gnu code that’s “autotools only” does just fine with rather simple cmake scripts that anyone can understand.
The weird thing is that the malicious stuff is only found in the dev's hosted archive file of the source code, it is not found on github or other locations. Very odd.
Huh, who is allowed to change the make file? Is that traceable?
What a clever way to inject a backdoor! Everyone is focused on reviewing the source code, nobody expects the build process itself to be altered
Who TF commited that?! Let's start there >_>
Committed by Mr. N$4
one of the main maintainers lol
This kind of stuff is more popular than you think. Thank you for covering this.
The build process is too open if the code being built can affect it that way.
Valgrind isn't just about leak detection. The main tool, memcheck, also validates that memory is initialized when it is read and that reads and writes are to valid memory. Both of these are usually far more serious than leaks. In this case it detected an invalid write below the stack pointer.
Some nation state is pissed
Ohh "nation state". This guy is sophisticated.
kremlin or ccp?
The fact this was caught by sheer chance makes me wonder how many more backdoors using similar injection techniques are out there in the wild.
If I read it right this ‚just a guy‘ actually was a maintainer of the fedora linux distribution and the malicious actor managed to become the maintainer of the XZ library by posing as an active developer for years.
The ‚just a guy‘ did valgrind memory tests to verify if the library is safe to include and it failed. The malicious actor tried to convince everyone the valgrind errors are GCC bugs to hide this, and ‚just a guy‘ even helped fixing the memory errors before finding out the true nature of the patch.
Edit: I can no longer find the mailing list post that confirms the version above. The newer posts by Andres Freund (the ‚just a guy‘) tell the story a little bit differently. See my response below for sources
How do you know this
@@luigicorciulo8190 He writes "see my responses below for sources" but I see nothing so consider this bs.
Amazing find. Not to mention the genius of the back door itself!
Source and software distribution platforms should run hash checks on releases to avoid those situations, it's insane that this is not the case yet.
If it's too expensive to implement globally, it could instead be done once a project becomes big enough, it could earn some sort of verified status.
what is hash checks?
this is happening at compile time tho most people are going feel completely safe after compiling the lib themselves.
Part of the exploit was rolled into released source tarballs on the Github website (but not the Git repo IIRC). If upstream package maintainers used the provided tarballs to build the binary packages and then made the signatures available, they would be 'trusted' regardless. In which case, you're screwed. There are multiple issues the perpetrator exploited to plant this backdoor; there's going to be a ton of lessons learned.
Do I recall correctly that early versions of the C compiler had a cooperating backdoor with the login program that would install a hard to spot backdoor into the login program? This was in the early, early UNIX systems. The hack you mentioned in the video is incredible. Dynamic linked libraries can have all sorts of interceptors & interposers placed on the, by simply having suitable LD_LIBRARY_PATH setting that refer to your version of an API. I used this trick onetime to create an interposition on malloc & free that would add additional header and footer memory to malloced storage, so the memory had sentinels at both ends to detect overwrites, and I think I also had it link all of the malloc'd memory together in a list so an analyzer could scan through the malloc list and find any areas that had been overrun, and compute some statistics about how much memory was allocated, maybe how long it was allocated for, etc. Probably around 1992 or so.
At least theoretically the self-propagating backdoor you're describing was described way back when in an academic paper I remember being told about, "Reflections on Trusting Trust", or something like that, and it's been a sort of... matter of conspiratorial debate whether or not it actually happened, at least that's how I remember it.
@@05MatzYes, it was from long ago -- maybe the 80s when the description came out. The source code for Version 6 and Version 7 UNIX is available (including for "cc'"), so if the hack was still there , it should be observable. It was one of the original UNIX guys, I think Brian, but maybe Dennis who talked about it and provided some details about how it worked. [as I recall]
The complexity of projects raises, but this backside is peace of art… 🖼️
nice video with good incentives to further dive into
I thought "open source" meant that obfuscated code was never accepted or allowed?
it was a compressed test archive. it was, in fact, very well hidden.
I love your past-participle of "to dig"
Description is from previous vid still. Also I hear Valgrind isn't pronounced -grind (like coffee) but -grinned (like the Cheshire cat).
fixed thank you
Yeah, it means gate
Thanks for this great video, 0% fat and straight to the point. Cheers.
Andres Freund isn’t a security researcher, but he’s not “some guy” either 😅 He’s a principal engineer at Microsoft and a PostgreSQL developer.
(Not a developer who uses Postgres, but a developer who builds and maintains Postgres itself.)
He is definitely an experienced developer
This feels like it'll be such a shock to open source projects... I would expect that other project would start scrutinizing their code and contributions more deeply in the next couple of weeks and we'll be hearing more surprises
At least it was caught and is open source.
Thanks! Installed 5.6.1-2...
Seems almost as if someone or a group of very knowledgeable engineers is deliberately targeting open source software. And it's not just technical, this also targets the way changes to open source software are usually review, validated and merged into the code base as well. The whole change process for open source software seems to be in need of an upgrade.
But it's open that's the problem and benefit
Oh, certainly. I'd bet that this was the work of some kind of three-letter-agency* from some country somewhere. Probably better than even money that it was a nominally 'friendly' country to you and/or me (in my case Canada, so the whole usual suspects of Five Eyes/etc.)! Not so friendly when they show their hand at their persistent efforts to make everyone in the world unsafe in the name of their stupid squabbling power games.
*Name may not be exactly three characters, but is probably an acronym.
@@05Matz More like 6 letter agency tbh. They do be trying to spy on everyone all the time.
@@05Matz This is something that I wouldn't put past MS as a way to eliminate platform competition.
@@bobmorgan1575 I don't think they'd do something with this much risk of getting caught, just to besmirch the security of a competing platform. I think whoever did this planned to USE the backdoor, not for it to only exist as FUD.
Dude I love you for these videos. Thank you
is it only when you enable compression or just in general. if it compromises the binary in the build process does that mean it only effects ssh thats recompiled from source code or does it effect every binary that comes with every system? (sorry im unfamiliar with how that kinda stuff works)
every binary that was compiled after the change was introduced is compromised, which may or may not include that one in your system depending on how old this is
If the source code was compromised then binaries where also comprised. But the issues was only created 3 weeks ago so a lot of systems will never have received the update, also you would have to be a ssh user to be affected, because ssh is not something that is enabled by default. This is my take anyways from what little info was in the video.