Understanding the Linux Backdoor: Implications for Open Source [When Penguins Cry]
ฝัง
- เผยแพร่เมื่อ 3 เม.ย. 2024
- Dave explains the new backdoor in SSH. For my book on life on the Spectrum: amzn.to/49sCbbJ
Any requests to contact me on Telegram, etc, are scams...
Follow me on Facebook at davepl for daily shenanigans!
Follow me on Twitter at @davepl1968
Image Credit: Medium.com - วิทยาศาสตร์และเทคโนโลยี
I should note that the individual/group who injected the attack had spent years on a social engineering campaign to gain trust and get authority as a maintainer
*the agency
Spook. E.
So we can safely assume, fixes tin hat, that he had his target all prepared, knew that it would fixed super fast, so he prepared to execute his attack and is now fulfilled. I wonder who the target was… removes tin hat.
Lines up well with Russian timing tho?
Step 1: Ukraine
Step 2: The world?
@ It's pretty clear that they were hoping it would get into the stable release given that they submitted their updated version to Debian when it was close to releasing a new stable version. Remember Debian was supposed to have a new stable release out in 2 days from now and the attacker was trying hard to get it into that release. They definitely did not fulfill their goals, whoever the target was they got very lucky.
The human engineering part of this backdoor is equally as interesting.
It is, and it’s shown that OSS’s biggest attack surface is the humans.
That’s also true of closed source software of course, but one assumes that a company paying salaries to developers has some idea who those developers really are. With OSS, we don’t and nor does anyone else.
My worry is that as more package maintainers retire this is going to become easier to pull off. To guard against that there’d be pressure for maintainers to hand ownership over to some trustworthy organisation. Who is that?
The danger is that it becomes RedHat. And then they own more and more of Linux. They already effectively own systems Wayland and gnome, they could be in line to own even more.
Exactly they preyed on the developer who was alone, stressed and feeling the pinch. It went on for a while with several people or puppets involved.
I still wonder if "Jia Tan" was always a hacker or was originally a college kid who eventually was contacted by his national secret police to collaborate with them and inject the backdoor into XZ after they had gained the trust of the community
@@abarratt8869 It is also the largest attack vector
Where is he from?
Closed source could have backdoors already and we’ll never know as we can’t inspect the code.
Yes. The correct analogy is not internal Microsoft testing (they obviously wouldn't have caught something this subtle). It's imagining a developer running benchmarks on Windows, finding something taking 0.5 seconds longer than it should, and then not being able to explore what's causing it. So they, what, email Microsoft tech support and explain something is taking 0.5 seconds too long and should look into it?
This shows the danger of closed source. That email will just get deleted due to how silly it sounds and the exploit will exist in production code forever, never being caught.
Solarwinds
Meanwhile Windows takes like forever to update, can any one check if it is backdoored?
We don't know it but we I'm sure. Is that your logic?
@@wardm4 Some functions *are* benchmarked. But it depends more on if it's a function expecting many uses. So if syndicating account login to a separate server, then the login process should be part of a benchmark to verify how many login/hour such a server can handle.
But it isn't likely a single-user login would be benchmarked, since it isn't part of some critical capacity pipeline.
So I would be very surprised if Microsoft would have caught this. But if an AD server can handle less requests/second, then that is very likely to be quickly caught.
Some will see this as a failing of open source. I invite them to read about the Sony rootkit scandal of 2005, in which a "real" company installed rootkits on millions of PCs. If closed-source software is no defense against threats, I for one would rather live in the open-source world where there's at least a chance to discover them.
Ohhhh yeah, I remember that.
IIRC, the Sony CEO(?) when confronted about it basically went "it's fine that we did this because most people don't know what a rootkit is, and what you don't know can't hurt you".
Even the most tech-illiterate people can probably tell *something* is wrong with that attitude, even if they don't know exactly what.
@@LendriMujina assuming they actually used the word "rootkit" it should sound dubious enough to the average person that the flippant attitude towards it being discovered by the CEO should warrant suspicion
@@LendriMujina It was even worse than that. The Sony rootkit used cloaking functionality which could be reused and several viruses actually used the built in cloaking functionaliy of the rootkit to hide their viruses from anti-virus programs, basically making themselves invisible to operating system and apps.
Sony seemed unable to see what the problem was. They were just doing what they wanted. You must remember that Sony has authority, granted by Sony, to do anything that Sony wants to enable Sony to fulfil its right to do anything that Sony wants.
Truth being said here. Open source is ideal to spot weaknesses, backdoors, etc and correct them. A private company would probably have hidden this
Open source may have allowed in the compromised code, but it also allowed for the compromised code to be caught. So whether this is 'good' or 'bad' for the reputation of open source, to me, it cancels itself out. But it will surely also make open source better for it, as everyone is now made more aware of the potential vector of compromise.
This is the strength of open source.
Don't even mention the time period between discovery and fix. Microsoft isn't even dreaming about those response times. Btw. Which OS are they using for their Azure Cloud. Windows server right? 😅
The problem is that the amount of people that can do meaningful code reviews is relatively limited. While in theory open source code can be reviewed by all and will be reviewed, the practice is different. More often then not important libs like this are maintained by just one or two persons.
One of the issues with Open Source is weak identity verification of the contributors. At this point, the alias of the contributor is known but their real identity is unknown. It could be a state actor or an individual, no one knows.
The other thing is that a large company like MS does take things seriously and check their stuff. Does a smaller company? Or what about one that makes software for their bespoke hardware?
We do have a somewhat recent supply chain attack on the proprietary side of things with Solar Winds hack.
something super important here; xz-utils isn't part of the kernel whatsoever. This wouldn't end up on Linus' desk like a change to the kernel would.
I agree, but for precision, I never said it was part of the kernel, of course.
But when people in the community say "in Linux", they generally mean the kernel. It's like saying "in NT" when discussing Windows.
"In systemd" would be more accurate than "in Linux", but still completely untrue. It would, however, win you brownie points with the systemd haters, who tend to think that the architectural philosophy behind systemd is just asking for something like this.
@@JonBrase It's even more more like saying "in Windows", when IIS has a bug.
When common people say "Linux", they mean the whole software system, not the operating system that Linus maintains even today.
Even the word "distribution" goes past most people not understanding what it means, why they mistake it as well to "different OS".
So common people make wild claims that if any software that isn't part of the OS gets bad things happen, then it is the OS.
The problem is that not even most software engineers understand what OS is, what it does, what is its purpose. It is simple really, but not as simple as walking someone at their car and opening engine hood and show them where their car engine and all other different parts inside the engine compartment, as not everything in there is part of the engine. But at least most people understand that car isn't engine, and engine isn't a car like people think about software systems and operating systems.
Why word "Linux" is always causing misinformation when it is used to talk about anything else than the operating system itself, aka "Linux Kernel".
@@JonBrase there is, however, no denying that systemd is a steaming pile of kaka - regardless of it's philosophies.
actually, it was more hidden than just hidden in the makefiles where no one checked, it was a script only in the release tarballs that added some lines to the makefile if it met some conditions, which would then add in the backdoor. (this is mentioned in the original openwall post)
GitHub allowing people to manipulate releases separately like that is a pretty exploitable feature, imo.
95% of the chain was commit to git, the last, the last 5% that triggered the integration in the build process was added only as part of the release tarball.
@@nurmrSure, but that 5% that triggers it is the key trigger that would look suspicious for anyone who takes a closer look. The remaining 95% are the binary files in test folder, but you need a way to incorporate that in to the build system, and randomly grabbing test files during a build, and doing lots of decoding operations is going to raise eyebrows to whoever is reading it. Maybe it would have slipped in anyway since xz isn't super active, but someone like Lasse Collin (the other maintainer) might have noticed it. Hiding it in the release tarball and relying on the fact that no one checks the consistency of release artifacts is IMO the a key technique here, and it relies on maintainer access because this is something a contributor can never do.
@@BrotherCheng "rabdomly grabbing test files" was hidden too, there were some very weird obfuscated regexes that would only match those files.
@@The_BoctorGitHub's own tarballs are automatically generated, but you can attach any file you want to a release (otherwise it'd be a pretty shoddy distribution mechanism), and I guess for whatever reason it's a common practice in some projects for the devs to upload their own tars instead of using the automatic one? (I assume there's a reason for this since otherwise why bother with that effort, but I don't know enough to say what that reason is.)
What is scary is that the maintainer that created this backdoor, worked on getting trust over the course of several years. This was most likely some coordinated attack of a group that wanted a zero day into most servers in the world. I wonder if there are other projects that have been compromised in such a way.
it also means everything done by that maintainer is suspect, going over two years of history to check if there isn't anything else nefarious probably isn't a small task
If they put that much effort into getting trust for an open-source project, how much more likely are sponsored people to be working at closed-source companies e.g. Microsoft, AWS, Oracle, Okta, Azure, Google, LastPass, Antivirus providers with nefarious goals?
@@pietersmit621 Probably even more, but by compromising the linux kernel, they will compromise all those big companies.
At the very least we should start looking at how we could harden open source projects from such attacks. This was pretty much a wake up call.
@@pietersmit621 As Dave said, those big companies have a process and more people looking at everything so you'd have to get many more people involved. Afaik this was a "small" project with a single overworked maintainer that was pushed to give the bad actor access
Recently someone involved with F-Droid mentioned that a few years ago they had a similar situation happen where someone was really pushing for code to be merged that would allow SQL injections to happen in the app's search function and would result in a crash. They put off reviewing it, but when they did review it, the original submitter deleted their account to hide evidence. This could be related to the group behind the XZ backdoor, I think I heard.
The Linux Backdoor was found, only God knows how many backdoors are in Windows.
Microsoft has publicly known of one major backdoor being utilized by hackers, the US government, and even foreign governments for over four years now, and still hasn't patched it. Pegasus.
/facepalm
I'm utterly disappointed my comment has been seemingly deleted. We DO know of multiple backdoors in Windows that still have not been patched. There, maybe this one won't get deleted.
The more interesting comparison is how the back door was able to get there in the first place. In this case the attacker had to spend years building trust and with the maintainer, obfuscate the code and hope the malicious changes went unnoticed.
In a closed system if you want a back door all you have to do is ask.
Or tell as is the case with publicly traded US companies
And also rely on the fact that the library was only used in a roundabout way when a specially patched version of sshd was loaded via systemd. This attack is brilliant on multiple levels!
@@stephanweinberger I'm not sure it was brilliant. Certainly novel, but if you give any systems level dev who has a couple years of experience the same task they can likely all come up with something similarly hard to detect. The most clever thing about it is it uses a xz compressed file in the xz library to test xz for the backdoor. Everything else is just typical obfuscation which doesn't take much skill to create
@@__Brandon__ It's not the obfuscation alone, but also how the backdoor is activated in the field: it's planted into sshd indirectly via systemd - both of which don't need to be touched in any way. That's thinking around _two_ corners...
Sounds like the NSA, DOJ, CIA, or any other alphabet organization of the Government.
Makes you think about how many backdoors we will never catch because of closed source.
Ding ding ding!
And many are probably INTENTIONAL.
Just think about how the UK government wanted to pass a law to basically force backdoors into messaging apps, for example.
Or in a way you could consider telemetry as a sort of backdoor as it basically grants unlimited access to your data to the companies.
Open source should be the standard, if we actually want safe code
If someone told me the CIA had backdoors to all ios and android versions, I'd only half doubt them.
@@no_name4796 "Just think about how the UK government wanted to pass a law to basically force backdoors into messaging apps, for example." - Actually they DID pass that law...
@@CoolKoon yeah... And what about the apps that won't comply? They probalby install CA cert in your machine to doesn't have JUST your messages, but ALL your traffic yeah?
Someone went to the trouble of plotting all the "utils" baked into the core of Linux; made a scatter-plot of ubiquity on the y-axis and unsexy/sexyness on the X-axis, and number of active contributors on the Z-axis, and diligently chose the one in the top-left corner, with the fewest active contributors. That's pretty thorough, and the long-con hasn't even started... Scary stuff.
Remember when the University of Minnesota got banned from kernel submissions?
Yup, very naughty.
Why were they banned if i may ask
@@MohammedShuaybNo, you may not ask. Sorry.
@@MohammedShuayb They were purposfully adding bad code into the project to see if people would catch them. They claimed it was because of some research they were trying to do but thats a load of horseshit and they were banned and still are from what I know for a very good reason.
Yes.
Microsoft watches the code very carefully, to make sure that only their malware goes into their software.
Win10 ltsc is actually pretty great though, can't really understand the hate.
@@emptylog933 If you are used to the speed and user friendly Windows 7 you will hate Windows 10 and Windows 11 a lot.
@@emptylog933 It made A LOT of stupid anti-user friendliness changes from Windows 7 to accommodate dumb users (and/or tablet users) that it's infuriating. Also the forced updates, forced maintenance, forced bloatware, forced fucking everything makes it a chore to use. Sure there are improvements in some regards but there's so many annoying details and major issues that go unfixed (like the 100% HDD usage even on clean installs) that it makes using Win10 a shitty experience for many.
@@emptylog933For one thing, Microsoft's aggressive campaign for everyone to upgrade to Windows 11. The fact you have to refer to "ltsc" should make that evident.
ROTFL just like Google which does ultra deep antivirus scanning without user consent and keeps stats for Google as a bonus. Google Play Services basically runs as root and actually owns your life.
Fun fact, the guy who found it ("Andreas Freund")
... his last name ("Freund") means friend in German.
also, their name is Andres
It was a Postgres developer who found this.
PSYOP
I'm honestly quite disappointed you didn't do your research into this. The story of how it got into the source code is readily available and actually pretty interesting from a process perspective.
Me myself don't want to google for xz utils right now. Might as well keep the tin hat on for some time. The more information you recieve, the more painful it gets. Anyway the guy behind Tukaani Project must be a Finn, since his name and the Project are in Finnish. This feels sad, because in Finland we just had our suicide wave after there was a hacker exploiting psychiatric data.
On the other hand, Microsoft is a huge consumer of open source for their products now. Unless they give the same level of scrutiny and attention to all the open source software they consume before they deploy a product, they are just as vulnerable to a supply chain attack as SolarWinds or any other proprietary software company that has integrated a lot of open source projects into their software.
That was one of the clever aspect - ssh is security sensitive so changes get checked thoroughly - but xzutils (or rather liblzma) isn’t normally used by ssh so doesn’t get checked the same way. Some configurations do happen to inject it, though, enabling this backdoor to trigger.
Very very sneaky stuff, someone put a lot of time/money/work into injecting this code, to the extent of becoming a project maintainer just to do this.
It sure would be bad if a state actor had a backdoor to systems or servers. Imagine Microsoft giving a federal agency access to journalists emails.
To be fair, I don’t think journalists were the targets here. How many journalists use Linux??? My guess is this was supposed to be a cyber weapon to be used against a foreign adversary. The only thing that would truly be safe would be old systems that didn’t have the patch and air gapped systems that couldn’t directly connect to the outside world.
@@benjaminlynch9958 yeah not many journalists, just most of the servers across the whole internet, nothin' big.
Journalists work for the alphabet agencies anyway. They’d be spying on themselves.
@@benjaminlynch9958most people use Linux or UNIX but don't realise they do. Android is Linux underneath. MacOS is a flavour of BSD with a custom desktop. If this hadn't been discovered it could have found its way into all kinds of systems. If you run Windows, many of the services you use run on Linux, that email website, your VPN, that website you use for managing your source code, they probably run Linux. If someone used a backdoor like this to attack a website like GitHub or JFrog, they could inject malicious into hundreds of products, both open and closed source. The potential implications are truly jaw dropping and could have let someone tamper with release builds of code running on pretty much anything where the developer used any of the common development management tools. This could even have had safety of life implications if the tampered code found its way into things like medical equipment, airliners or equipment in nuclear power plants. If someone intended to use this code to compromise a major cloud provider, the implications could be very serious including unauthorized access to even governmental or military systems.
We can't assume this is the end of the matter. The goal might not have been to compromise every Linux server in the world. It might have been to compromise a particular developer or maintainer's machine to insert some other compromise that hasn't been detected yet. Such a person could already have run the code as they're likely to be on the bleeding edge. Hopefully, this was just someone's plan to steal a boat load of cash from somewhere like a bank or cryptocurrency exchange
@@benjaminlynch9958 or *BSD or Solaris or... But yeah, Linux is rather popular, but for say government systems, one would still have to get past filters, firewalls and more to even try to log onto the server.
So, they'd be looking at beyond difficult to access servers, with pinhole firewall entries for only the necessary services on the unclassified networks, all of the important traffic being on a number of classified networks that cannot be seen or see the internet or the unclassified network and are actually tunneled under it and one another, yeah, this would miss central government networks and basically nail provincial and municipal networks, as well as commercial and utilities networks. And utilities are a target of interest, as some high profile attacks on water supply networks have recently shown.
You asked for it, so... I very appreciate the effects of open source on the community, for example:
- in my spare time, when learning a new programming language, having the actual source code as reference how other dudes did the stuff, instead of relying on documentation (which normally is behind). New developers are faster up to the game and in quality mode, which benefits all of us, not only the company with its proprietary coding guide lines.
- having the ability to see how such hacks work because there are no interests involved by the owners of the code, lets say financial and pr.
- being able to replace malicious code against the will of the owner if reluctant.
- having closed source fan boys raising the quality of open source by reviewing code for the most dedicated attacks and zealously reporting them to the whole world to see, not understanding they make the whole point for open source :-) thank you very much
*The major thing that everyone misses when attacking open source for the failings of the "many eyes" argument is the success of the "many eyes."* This was caught because the backdoor caused a behavioral change that made someone notice it. One eye that didn't even know the code base which was infected blew the backdoor with years of presumably state-sponsored effort behind it completely out of the water.
if you are inserting a backdoor you would want to make it look like an innocent mistake
Dave. I believe the exploit doesn't allow you to actually log in using SSH. I believe it allows them to put linux shell commands in the SSH authentication process itself and have sshd execute those commands with root access. I believe XD infected one of the encryption libraries in the sshd service. Very clever stuff. I saw a video of the live working demo of this yesterday. I'll try to link to it here.
Yeah, it's complex, like VERY complex, but if you have the private key for it, and only 1 person presumably does, it allows you to run commands as root simply by sending a specially crafted handshake message that will trigger the backdoor and run the command embedded in the handshake message. Low level learning has an excellent video discussing how it works, and you can find the video here: vV_WdTBbww4
@@WarrenGarabrandt so many suspects even freund himself because of his connection with microsoft.
Dave doesn't understand it. It was done in this manner because they knew they could never get to SSH directly. It is very clever stuff, and it should make us all wonder about whether there are backdoors in closed source software like this that would never otherwise be detected. The very same techniques can be used.
like a third party attack@@elta6241
Which essentially does the same thing: root privilege escalation. For a over the top explanation from a Microsoft guy to his usual audience this is good enough.
I actually share the opinion of one guy in a blog post: I AM NOT YOUR SUPPLIER.
I posted the code on the internet because i want and not because of you. You want to use my code? Fine, close the source if necessary for your market, i posted there and if anyone wants to use whatever, but don't come @ me saying that "you need to fix this NOW" or "change that NOW", i will do it whenever i want to and if you want me to SUPPORT YOU, well... we can make a deal for commercial licensing and/or support. Seems fair to me.
Absolutely. It’s amazing how often that gets forgotten.
And in that sense, it’s difficult to say that the person who put this backdoor in did anything particularly criminal. They modified their source code and build system (ie they legitimately had access to all this). Other people copied it and used it.
Ok, they were doing so out of malign intent; probably. Or it could have been a security flaw demo that simply got revealed too early (less likely). If one actually took it to court, it could be difficult to make a case stick. Probably one would have to go back to the campaign they launched to cajole the original maintainer into making them an admin, which might be construed as obtain goods and services by fraudulent means (uk ish legal language). But computer misuse? There’s no evidence that I’ve heard of that the backdoor has actually been used.
What we don’t want is every flaw and mistake in OSS repos to become a reason to prosecute the maintainers. After all, what is the real difference between a blatant backdoor and a careless bit of programming? Nil.
Since almost all closed source projects rely on some open source libraries open source vs closed doesn't matter. Fully understanding your supply chain is basically impossible nowadays. It's not only the software. Even the tiniest bits of hardware are manufactured from non-domestic companies accross the globe. There is an interesting recording of Bruce Schneier on this on youtube
This is why copyleft licenses (GPL for example) are so vital! It forces companies like MS to, if they want to rely on such libraries, to make the _entire_ code base opensource as well. This is why projects like AOSP exist. We're kidding ourselves if we think Google makes Android opensource out of the kindness of their heart. They have no choice. 💁
This is why copyleft licenses (GPL for example) are so vital! It forces companies like MS, if they want to rely on such libraries, to make the _entire_ code base opensource as well. This is why projects like AOSP exist. We're kidding ourselves if we think Google makes Android opensource out of the kindness of their heart. They have no choice. 💁
@@soulstenance Once again, "rms" is right. XZ utils isn't GPL or LGPL, it is "public domain" which means ANY company can make changes to use their own code "enhancements". The same goes for any Apache licensed code which Android is licensed except the Linux kernel. I believe that is the part annoying Google so they are developing Fuschia.
The really shocking thing is the realization that such things would never be discovered in closed source because no one would be able to detect such irregular CPU spikes without the code.
and that, at least most Microsoft products, aren't as optimized, so that half a second can't be noticed
Good lord, large companies doing huge loads of time sensitive work open tickets routinely on closed sourced code everyday.
I watched one get opened last week for millisecond spikes on storage after a new patch.
That would drive me nuts. Are those typical anomalies, or ominous indicators?
You obviously didn't even watch half of the video.
@@RonJohn63 Why do you think so? Becaus he says ms has a better process?
Since it never really went mainstream, it's easy to assume there are many other points where this could have been caught in downstream processes as well.
As you say, it got caught early.
Just because within an enterprise the process itself is hidden/internal instead of public, doesn't mean that the public process is inferior.
Although yes, it sure brings more visibility than a private internal process. So you'd never hear about it in a private one.
Probably not, at least not early evetually I thinkl so but as Dave said it had a really small number of devs working on it and the bad guy was already trying to get it included in the stable releases of Fedora and the like.
No the public process isn't inferior. But it is by no means guaranteed (it isn't on closed software either). But a lot of people believe that open source software is inherently safe (because everyone can see the code). In practise it much more nuanced. As you say, its caught early this time. But what if that guy didn't dig in deep, or thought you know what I'll look at it after the weekend (and forgets about it). A lot of open source software is dependant on a small number of people and on the whole there are not many people that can do these kind of analysis, let alone make time for it.
But it went mainstream. Arch Linux shipped it, as well as some other beta versions of other distros. Open source isn't as well scrutinized as one would think, the stage where it would most likely be cough was on the commit review, especially for a project that wasn't that popular.
The issue is, project maintainers do it voluntarily, often a thankless and stressful job. Many develop mental health issues from those problems. And that is the main problem with open source.
@@EmilioBPedrollo Arch Linux is a big offender with rolling releases, honestly if you don't live on the edge there are plenty of distro that stay behind one or two update cycles which is safe. keep in mind the other distro that use the latest updates are the reason such things get caught early, if you're not tech savy or you have sensitive info avoid such distros
@@EmilioBPedrollo I disagree with your definition of mainstream.
To me, mainstream would imply an LTS release of a commonly used distro.
If you want to compare with betas and rolling releases, apples to apples, compare those with Windows betas and builds sent to public testers.
I understand your skepticism, but you'd also be surprised how much more scrutiny is placed between what you describe and until it reaches Ubuntu LTS for example.
This is a victory for open source. If this were closed source, it would never be found.
I can't agree, because (a) it was found through testing without the source (but diagnosed with source), and (b) it was found by a guy at Microsoft who was doing the type of testing that presumably Microsoft does on its own stuff. One hopes, anyway.
Just not true. The benefit of open source is that it can easily adapted to fit your needs and those modifications can easily be shared. You can read and write assembled binaries in much the same way and people do all the time to find exploitable bugs or patch closed source software.
@@DavesGarageThe majority of maintainers getting a bug report about a 500ms delay in a separate process from a single user would've probably considered it very low priority. Luck was definitely involved, but the fact that Microsoft engineer was able to check the source to diagnose the issue themselves is the reason this got fixed so quickly.
@@DavesGarage Just because you replaced word "found" with "diagnosed" doesn't change anything. It was found (not "diagnosed", Dave, "found") because developer had the access to the source code. To the complete source code, not the stuff that hosts on github (vs code says "hi").
Freund didn't "diagnose" that binaries were stored in tests folder, he "found" it. In the open source code. Don't replace word "found" because it breaks your ignorant narrative
@@DavesGarage The guy found it during his easter holidays, when apparently he prefers running Linux for some reason.
This "near miss" was, indeed, scary. _But_ Microsoft now wants to add *_Recall_* into the OS. Roll the dice. Take your chance.
In fact, further investigation shows that this wasn't login without password but the actual backdoor (RCE) was even more complex. The backdoor allows blind command execution as root (that is, no return channel) and the command is encrypted as CA certificate in the handshake. If the CA certificate has some magic bytes set correctly, it will be decrypted with the public key in the backdoor and executed via system() API call. Nobody but the original attacker can create the encrypted content to be executed and nobody can check from the network traffic alone that the attacker has tried to pass a command because it would look like a regular ssh login attempt with non-authorized key and no extra TCP/IP packets sent in either way!
Of course, ability to blindly run commands as root is typically enough to build a reverse connection from the attacked host to some other host controlled by the attacker (maybe another backdoored system) and get a return channel that way. However, your blind command could be something like "sleep 6000 && $build_reverse_connection" to disconnect the reverse connection attempt from the SSH connection attempt.
Or worse, execute one's desired goals non-interactively, not forming the outbound connection at all until one's goal, say data exfiltration, is achieved. Now, one's only potential warning is the outbound - oh crap, data's already gone.
Often enough, many are caught and halted by catching the bidirectional connection, regardless of which end initiated said connection. I've actually done it, captured the entire session in full packet captures, including the malicious software of the week, which was immediately submitted for inclusion in the IPS and antivirus and novel aspects of the attack fully documented and submitted to FBI intelligence, as it was a foreign nation-state actor and known APT.
One such attack involved lateral spread that I'd gotten a sniff of, began packet capturing and captured an RDP session in progress, attacker opened notepad, did a buffer dump of binary data into notepad, saved it and used it to assemble their tools on the target system. Got not only their binary tool signature, but the latest PXE padding for their known tool. Outlined the remainder of their attack, then their session experienced a mysterious termination before the data that remained safe on the victim system could be exfiltrated...
The system was then immediately isolated pending full file pulls and forensics.
Two more attacks and we found their point of entry into the corporate, global network. One forgotten test server on one DMZ, prompting a full review on every DMZ in the corporate network. Yeah, as well received as the plague, but necessary after two golden ticket attacks, one of which was successful and boy, you don't want that kind of aggravation!
Don't forget that windows also runs openSSH
I once asked Alex St. John, when he was writing for b00t, about Windows back doors and his answer was so vague I was convinced he was saying yes.
He probably hedged for the same reason I would... none are known, but you can't prove a negative.
@@DavesGarage You can't prove a negative *by hiding the source code.
If the NSA has a National Security Letter forcing Microsoft to allow insertion of backdoors, it is also a felony for Microsoft to disclose this fact. This is why some projects have a "warrant canary" statement in their periodic disclosures. They say, "We have no state-mandated backdoors." Their position is that if they are ever forced to introduce a backdoor, they can remove the warrant canary statements from future disclosures without explicitly breaking the law, because they are not *telling* users that they have been forced to introduce backdoors; they are simply no longer telling users that they have not been forced to introduce backdoors.
Microsoft doesn't need a back door, just use an Alternate Data Streams.
@@inpito Microsoft IS the backdoor, no wait its just the door, no secrets here.
The xz utils project is independent from Linux. The team behind xz was tiny and thus was easy to exploit. The root of the problem in my opinion is the blind trust that was put into such a project.
The issue is related to the distributions (Fedora Rawhide, Debian Sid, Archlinux, ...) which didn't spot the glaring security nightmare in packaging a library with only two active maintainers. (To be fair, Rawhide & Sid are unstable releases and the backdoor didn't work on Arch)
"But it's not in Linux! Just in the repos!"
How do you get Linux installed?
"Install one of the repos!"
So you can see why this argument is not that compelling...
The attacker started earning the other maintainer's trust over the years, until he finally took over the project. It is believed he created other Github users to put pressure on the maintainers of those distributions to include the newer versions of xz-utils into the repos.
Also, most open source projects (like this one) are done as a hobby or just for the love of contributing to open source!
@@DavesGarage…or, you know, just compile from stable/LTS. Sure, you live and die by the sword, but I don’t know of anyone or any company much less that would willingly run their production on bleeding edge software save for devs on VMs to claim a paycheck. It’s sad to see people conflating bugs and backdoors here in the comments. 🫤
@@DavesGarageYou mean distros, right?
Close source applications use open source libraries. In fact they use the same libraries that macos, android, and linux use. So by this logic window is android. Widows is macos. And windows is linux. Is very bad logic indeed
Dave missed a lot of the context of this backdoor (and of open-source in general). He seems to imply open-source software does not have tests or reviews, which... I mean, it does? In the liblzma case, the bad actor was making positive contributions to the project for YEARS. The previous liblzma maintainer was burned out, and yielded ownership of the project to the bad actor due to their contributions. They added the core backdoor code only after this. The liblzma backdoor is insane not just for because of how well it's hidden, but because it required a years-long campaign of espionage to install.
Did not imply that. Specifically said I am not privy to the process. But if the process was as through as it should be, they would have caught this as a perf regression. But all that is in the video!
…or what if was NOT a ‘years-long campaign of espionage’…
What if the ‘bad actor’ was actually honest and good, but was suborned AFTER receiving ‘the keys to the kingdom’?
Maybe he was blackmailed? Or bribed? Or ‘got an offer he couldn’t refuse’???
🤔
@@ernestgalvan9037 Hard to rule out. But the bad actor was laying groundwork for the backdoor since June/July 2023, which is a very long time for an account to be compromised without a "true" owner noticing or notifying someone.
@@DavesGarage they wouldn't have because xz on it's own wouldn't have had a perf regression during testing. It doesn't get noticeable until sshd gets involved - but not the "official" openssh, only the patched versions that end up being done by distro maintainers to pull in systemd-notify support; so there's at least 2 levels of indirection before you even get to the point where it becomes a potentially detectable issue. And it was detected, and rectified.
@@DavesGarage Liblzma has tests and a specific test file is even part of the way the backdoor is injected into the binary during the build process. One of the test files is an corrupted xz archive which contains obfuscated code for the backdoor. During build process the build script extracts this obfuscated code and adds it to the source to inject the backdoor into the binary. The developer even requested a change to Google's oss-fuzz to prevent the detection of the malicious code by this testing tools. Additionally he helped fixing a "bug" in Valgrind, which was caused by the backdoor. All of this was prepared bit by bit over several month.
The real bug here was that a single maintainer does run a project, that core elements of our infrastructure depends on, entirely on his own, on the wedge to a burnout, completely unpaid. That is similar to the Java bug (Log4J). The bug here is not open source, the bug is in the system that essential open source projects are not paid for. Economy depends on it, national security depends on it. And even more so if the attack with its chain of heavy social engineering was performed by an intelligence service of a foreign state. The real lesson from this incident should be, that states should take tax money to support open source projects more.
Log4j was not a bug, it was a "feature". Java strings by themselves do not go ringing the internet. Log4j intentionally made JDNI support.
I don't see how paying for the work would have prevented either case. "Jia Tan" definitely wouldn't be like "I am being paid for xz, I should quit stop being a state actor on a fat payroll"
Log4j developers(who made the feature) definitely wouldn't be "maybe we need less features?". If anything it'll be opposite: more bloat.
@@AM-yk5yd It's not about Jia Tan, it's about the xz maintainer. Jia Tan got through because he was able to exploit the situation of a single maintainer being on the wedge of a burnout. Jia Tan probably already was on the fat payroll of an intelligence service. He's the one getting paid for infiltrating, the maintainer is not getting paid.
@@dreamyrhodes anyone with an actual working experience can tell you that burnout does not magically disappear because there are money, so this point is moot.
The moment project would be looking for other developers would be a point bad actor intervene. And most likely would look better than other candidates - even because why i say "he" there is no reason to believe even now he was a lone wolf without team. If you not even for a minute think on how bad actors would behave if it situation was different, you are not thinking enough and propose solutions no more effective than asking "are you over 18"on lewdy sites.
Except I am pretty sure Windows HAS been caught with backdoors. Apple didn't even password protect root too. Also, not sure I can believe the process checks for efficiency loss when Vista and 100 exist.
Many Linux distributions don't password protect root. Without a password, root is then denied an interactive logon.
@@spvillano Locking the root account =/= no password.
The changes to the makefile (or rather ./configure script which creates the makefile) wasn't even checked into source control, so most people wouldn't even see it, it was just included in the release file.
And if the makefile isn't under version control is it really open source
So the main lesson to learn would be to always check out the repo and not just copy the tarball.
@@stephanweinberger save that a trusted individual could as easily then generate a new hash for the Makefile, save it and it appears as valid as everything else in the repository.
So, timestamp auditing would also be necessary, as its hash would be saved at a different time and date than the rest of the files. Well, unless that also was altered.
My first rule of information assurance, trust no one, not even myself. So, stumbling blocks, such as checks and balances, auditing before publishing, etc always get inserted, regardless of my gonad pain. Never had a persistent compromise on networks I was in charge of, so I obviously did something right, given we were repeatedly targeted by multiple APT's.
@@spvillano we could all go sign the commits and gpg sign all build artifacts. That would be enough to ensure that atleast we are all using the same source code and that we know which key committed the code. Without gpg you can put anyone's name or email as a commit author to try to muddy the waters
It made the FPGA geek in me smile that Dave's Makefile example isn't even a C code Makefile. It's a Makefile for driving an FPGA design (on a Digilent Spartan-3 board) on the old Xilinx Project Navigator tools (from about 2011 or so).
Actually it's not "in ssh", it's in libxzma, a library used by xz but also by systemd. In systems using systemd ONLY that patched ssh to tie it into sshd and THEN only of you have cutting edge distros that had this particular versions of the libxzma 5.6.x which are only used on very bleeding edge distros, Fedora Rawhide was alledged to be one of these but I have a Rawhide box and it did not have this version so that is bogus. Ubuntu, debian, mint, centos, most other mainstream distros are NOT affected.
Your Rawhide box may just not have pulled the package in between when the update was put out and when it was pulled. It was RedHat themselves saying that all Rawhide boxes should be nuked to the ground.
Thank you, came to say this. Claiming that it affects all of Linux is either misinformed or disingenuous.
Umm. Ubuntu and Debian ARE affected!! But only their cutting-edge beta versions.
@@raidensama1511 exactly, the stable versions of Ubuntu, Fedora and Debian are not affected.
We can't say stable versions are completely unaffected given the bad actor contributed over 700 commits to the project that go back years.
MS17-010 was used by the NSA for countless years before it was found and patched. We still find instances of it today in unpatched old Windows systems. Old servers. ATM machines. Industrial systems. And on and on. That’s basically instant access to a SYSTEM level command prompt while only needing to have a systems IP address and being able to communicate with it over the network. Code review and process checks didn’t find that one.
It sure helped me pass my ethical hacking course 😉
Bugs aren't backdoors, though. I did not and do not claim closed source produced bug free code, so I think this is a strawman fallacy!
Ms17-010/EternalBlue is not a backdoor; rather, it's an exploit that specifically targeted SMBv1 and weaponized it. The NSA did not inject the code responsible for the integer overflow and subsequent actions.
Ms17-010/EternalBlue is not a backdoor; rather, it's an exploit that specifically targeted SMBv1 and weaponized it. The NSA did not inject the code responsible for the overflow and subsequent actions.
@@DavesGarage the difference between a backdoor and a bug is intent.
Hard to determine intent… who knows maybe that was malicious claimed ignorance.
In regards to why was the changes not reviewed beforehand, the attacker had a cohort that became one of the maintainers of the project. This was needed for any of the binary code to be permitted to be merged with the main project. This was not a quick and dirty comit and took years to do because of them trying to wrestle away control of the project from the maintainer. This type of change in another project would have never succeeded because no other maintainer in there right mind would have comitted binary untested or reviewed code into their project. This is a very complicated story on how this even got past reviewers but its by no means typical or even possible in most cases...
Some notes for correction here. The main exploit was not in source code format and was actually binary data.
The attack code was actually in the testing stuff not in the main source code.
Yes a binary was added but it is common in things like compression and encryption code to include TEST DATA that is encoded so that part of the test is to decode it. The bad code wasn't even obvious in the decoded version. The hack literally snipped bits and pieces of it to assemble the bad code from innocent parts.. I have seen test data include things like tar files and further encrypted files (to test effectiveness of compressing compressed data etc.) so it is not as hard to understand how this binary test data got in.
@@julianelischer6961 Dave highlighted exactly why raw Makefiles are terrible. They're a mess and a nightmare to audit. Which is what let the attacker kick the whole thing off. Unfortunately, there's a large number of developers who haven't moved past the 90s, and think C with Makefiles is the ultimate end all be all.
hmm the script (m4?) that injected the malicous object code into the build process wasnt checked into git, but was only added to the sourcecode tarball that the maintainers provided?
That’s the way. Keep it out of the repo but inserted into the release.
Do you think that GitHub should remove the ability to do "hot" changes to existing releases like that? Not asking a rhetorical, I personally dislike that feature because of things like this. A program called MilkyTracker was also broken on a few distros last year, because someone working on it changed a release without incrementing the version or anything. As a result, packagers didn't know there was an update!
I remember reading about a very early backdoor that was done by embedding the backdoor insertion insertion into the compiler, so when it compiled a compiler it inserted itself, and when it compiled a bit of password code it inserted a backdoor.
To find that paper, search for Ken Thompson’s paper entitled Reflections on Trusting Trust
There's an article called "Reflections on Trusting Trust" by Thompson, 1984, that touches on this idea of can you trust the compiler that compiles your compiler. Google for that title and it will come up dozens of times online. Anyway, it's an interesting thought about that.
"Reflections On Trusting Trust" by Ken Thompson. Published in 1984. If anyone hasn't read it yet, now is the time...
So the question is... does it still lives inside the compilers undetected because compiler needs compilation as well :p
Yes, I remember this. This was early days when Linux came on scene, it was evolving and more eyes were getting involved in open source.
Microsoft is actually a major contributer to Linux as well so it doesn't surprise me that a Microsoft employee noticed something unexpected with it.
Hi Dave from Australia. As an old applied mathematician with a 40 year old Down Syndrome son with autistic tendencies (the "daily double" in betting terms! ) I have met hundreds of autistic people in my life ( one of my body surfing cronies is a 40 year old autistic man and we are great mates) and you are trucking quite well. I love the insights you give because I am not an programmer. A man has to know his limitations. When I was doing Fourier theory in the early 1970s I did an assembly language FFT (Fast Fourier Transform) which actually worked after huge effort. That was me done with programming! Keep up the good work.
Presumably that “sudden ending” thing means that more people watch up to the end of your video, so their “watched until” stats become better than if they recognise your outro and bail early.
Open source: the backdoor is there until it is found and closed.
Closed source: the backdoor(s) are there as long as the developer or the authority wants it to.
Or somebody having expensive decompilers(IDA costs a lot, there are also free tools ghidra and radare, but they are not so good). Here it was telling that Freund is "not a security researcher, nor a reverse engineer."
I have worked in computer support since the early CTOS days. I have seen a lot. I do remember at time when we were installing Windows OS from 3.5 inch discs. Discs that came sealed in plastic wrappers. It was standard procedure to check the discs for a virus before installation. In this batch of 50 computers, every pack of Windows OS discs we opened were infected. After that we went to disk imaging from a safe source. So, somethings do get through.
I just found your profile because of this event. I just want to say that thanks for the work you've done. Don't let the MS haters' provocation get to your nerves. I have nothing against Linux though...
The issue with closed source is that we can never know if someone in upper management got a visit from someone in a nice suit wearing sunglasses wanted a specific undocumented "feature" put into the software. At least with open source, it can be investigated.
It would require a conspiracy amongst multiple people at a couple levels, and if you're willing to set those as a criteria, than Linux could be subverted in the same way, I'd bet. But again, since I don't know what the checks and balances are, I don't speculate.
@@DavesGarage This tells us otherwise. The techniques used here can easily be used internally in an organisation. I immediately get suspicious when people use the word ‘conspiracy’.
@@DavesGarageThe only "conspiracy" needed is for the source code to the component in question to be under NDA, with anyone who signs the NDA being told "this backdoor was required by the feds, if you talk about it you can expect a visit from the FBI".
Given the likely actor in this case, however (mainland China) an attack they made on western versions of Windows would likely be conducted in a similar manner (social engineering / corporate espionage leading to a supply chain attack on/through middleware), since they don't have direct jurisdiction over companies headquartered in Redmond.
@@elta6241 people with a corporate mindset (no offense, Dave!) are almost as a rule predisposed to not wanting to think about under-the-table dealings, even though every government has a body of people whose day jobs consist of precisely that. I'm not sure what the Canadian equivalent of the NSA would be, but I don't think I have to explain in a comment section under a tech channel what the problems they pose for software development would be.
@@DavesGarage If it's open source then it's going to need to be subtle or obfuscated or it will risk being spotted. If it's closed source it can be a lot less subtle. Even if you assume most governments wouldn't do this. There's a reason Germany and the US are advising against using Kaspersky products due to what's going on with Russia. And a similar thing with Huawei and China.
“Now a bit of my own backdoor stories.” Do tell Dave. Great stuff!
Be careful! lol
My thought on this is that this backdoor was intended to be found. With current political situation where every country tears apart technology with the intent of grabbing pieces to themselves it's no wonder there's an actor that wants to put doubts in collective effort. The main idea was, probably, to make people say "it's not a good idea to let those people contribute" and thus make open source not so open in the future.
Note that Microsoft may be a direct beneficiary of that. Also note that state actors may be beneficiaries of that.
Increidible. I am start to learn cibersecurity recently, and first thing beign told, if something takes longer than normal, we should start to check, if the system beign hacked.
It seems very likely to have been a state-based long term exploit, given the social engineering and span of effort. No company on the planet is immune from state-based attacks either... so it's not a specific open source issue really.
how many state sponsored backdoors have been baked into CPU chips and closed source software? operation rubicon went on for years. China (im guessing) probably planned this back door for years and got caught.
@@rs.matr1x Bvp47 was in Linux for 10 years (even after being submitted to "Virustotal" in 2013)...
You forget the angle about state actors being involved. Even someone working at Microsoft could be an agent.
Probably is, the trick is sufficient cross checking and auditing to try to trip that up.
The biggest plus in open source is, tens of thousands of eyes on all of the code. Maybe George didn't spot the makefile change, but Bob had an odd command he needed plugged in to compile the thing for his specific needs and ran into the unauthorized change. Bob asks about it, none of the devs know what's going on and start prying, then the lid pops off of that jar. All for free, whereas in closed source, all have to be paid, so are lesser in numbers or odd one-off cases that would cause them to go into a makefile.
Welcome to engineering 101, where everything is a tradeoff. You've taken your first baby step into adulthood. Beware, there be dragons!
No, that wasn't a dragon, I'm just not used to eating beans that much anymore... Sorry!
@@spvillano Bvp47 was in Linux for 10 years (even after being submitted to "Virustotal" in 2013)...
did you already do a video about why no new version of Windows can have a folder named CON
if not .... pretty please? :)
At least in open source a backdoor has a chance to be detected because people can see the make files and the build process and investigate. In closed source all the make files and build process are hidden so the user has no chance of knowing if a slowdown is because of the backdoor or something else.
Denying any connection outside of a specific subnet within sshd_config or host.allow/host.deny doesn't mitigate the backdoor? Edit: it does
Yes, it would mitigate this vulnerability to only allow listed hosts.
@@Sa1985Mr so essentially this isnt a problem, never knew an organization that would just leave ssh open. But then again I still see them use telnet ffs.
I would argue that the makefile deserves as much scrutiny as the source code because it *is* source code!
Nobody reads that shit and you know it. If it compiles, move on with your life.
absolutely. I have always treated makefiles and build scripts as source code - using the same rigour used for the code itself. The same applies to everything used in the build process - build systems should be rebuilt from the asset/version control repositories regularly. Of course, many devops teams have a view that they don't have to eat their own dogfood, but they are very very wrong. I gues thats the main difference between devops and SCM (remember that?)
@@mithrandirthegrey7644 then they shouldn’t be in charge of vetting code changes.
Well I'll bet they'll start now.
This video missed the fact that the malicious build scripts that injected the malicious code only exists in the release tarball, not the Git source. This is why you would not see it if you just browse the source code itself.
What we need to do is to have a reproducible way to generate the release tarballs and have those be checked routinely and automatically to verify that they match the source code. This is harder to do for binary releases (since you need reproducible builds) but for release tarballs for source code it should definitely be done.
I'm not going to watch this, but I came to give it a like for the title 'When Penguins Cry' alone... brilliant, sir.
If you ask me, I would pick open source over close source for safety. Any time of the day.
These things can happen in both Close and open source. At least in open source, it is transparent. The could be few back doors for in windows serving deferent government agencies as we speak. we might never know about.
Very considerate of that hacker to make sure only they can backdoor in my computer. This would have been a disaster if anyone could have used that backdoor😂
Well actually, you may be joking, but you're not wrong.
This is why I stick to Windows. All that sweet, sweet telemetry and data collection. When the entire OS is malware, you don't have to worry about malware!
Lol
As an Android user, same.
Nice review of the issue.
PS - What do you know about the BSOD? Did you have anything to do with that?
The xc backdoor was likely done by a nation state.
This has implications for all Open Source, not just Linux. A colleague got this backdoor on his Mac computer! An upgrade did update the offending libxzma component, though. Did you know that a lot of open source software is used also in Windows? It could have happened to another component as well.
These sorts of things always get called "Linux" problems - pretty sure this would be possible on BSDs using OpenSSH as well
@@etherweb6796 Nope! systemd does not exist on BSD's. Thank god.
Your colleague must have put in a lot of effort to get this backdoor into his Mac, considering how few linux distros adopted the change. I mean, it popped up in the nobody-should-be-using-this testing release of Debian, and some really bad timing from Manjaro got it in there, but the number of actually compromised linux systems was probably a few dozen.
@@oasntet Bvp47 was in Linux for 10 years (even after being submitted to "Virustotal" in 2013)...
@@Munenushi BVP47 was not _in_ Linux. It never shipped with a linux distro, because it is a rootkit (not a backdoor) that needs to be installed via some other vulnerability.
When Pangu found it, it was notable not because it was there in plain sight for ten years, but because they finally found it in the wild and also identified the parties responsible.
In the mean time and inbetween time, hope to see you next time Dave 😊
Maybe this is a dumb question, but doesn't ssh have to be running, and your network allow inbound ssh connections for it to work? So for example if you had Fedora with ssh blocked on the device and network level, you'd be safe right?
Oversimplification and missing key points. SystemD was about to implement changes that would negate this backdoor.
Very important note: the code was NOT checked in. The problem is the fact that the release package was crafted with makefiles and other scripts. So this was clearly a bad intention. The only thing checked in were the compressed files (binary files - hard to review, especially since they were used in tests) containing the rogue scripts inside of them. Without the custom scripts and makefile from the crafted release package they were pretty much useless.
So those changes in makefile and scripts were made outside the git source control.
And even worse, the files in question are specifically *corrupted* compressed files (corrupted by design, as they are supposed to be test cases). That means, the file can't even be inspected as is. To inspect it, one would have to read those very obfuscated scripts which un-corrupt the files before unzipping them.
Google xkcd Nebraska.
That comic was dead on
Another interesting video and I have to say your little cuts are just perfect, lols!
"No backdoors to Windows" XD.
1. Boot Ubuntu Live from usb drive, mount the drive where Windows is installed.
2. Replace the executable which opens a screen keyboard on a login screen with cmd.exe.
3. Fun fact: the screen keyboard runs with admin rights for no reason.
4. Instant root access without logging in.
How backdoors are inserted in windows:
- NSA asks to Microsoft
- done
So true, and we would never know
The damage wasn't really that bad since most people doesn't use Debian Sid It would have been much worse if it were in the Debian Release repo or if Ubunto had actually shipped it.
09:23 and following - That was VERY carefully worded.
From what I understand the Linux vulnerability is with distros that are rolling, beta, experimental and not the LTS & current stable versions. With that said, it is important to install updates as they are issued to keep your system secure.
The thing is it was found because it was open source, close source is mystery and or minefield.
That’s not true. As Dave mentions in the video, the attack code isn’t actually in the source code. It’s in a precompiled binary. And it was caught not in some open source code review, but through plain old benchmarking. The CPU was pinned at 100% for a trivial task, and the benchmark reviewer started asking questions because it didn’t make sense. If the attacker hadn’t got careless and instead limited CPU cycles or network traffic to something resembling ‘normal’ activity (say 10%), this likely wouldn’t have ever been caught despite being open source.
@@benjaminlynch9958 sure it was hidden, that's was the point. It would have never been trace to the hacked binaries without OS. It shows nothing is infallible. In closed source like MS may find a rogue independent programmer but we will never known if it management approved backdoor.
@@benjaminlynch9958If it wasn't open source the engineer wouldn't really be able to see why ssh was causing so much CPU use. He would have only been able to file a bug report likely to ssh which isn't even the root of the issue. So someone working on the ssh package would have to get the bug report and diagnose the issue, but if systemd and xz-utils were also closed source those devs couldn't do anything but forward the issue. Now even if the issue managed to make it to the proper maintainer, he's one guy and he's burnt out on the project so there's always the chance that he can't figure it out easily or promptly. It is kind of nice that the engineer who knew what he was doing could follow the chain on his own without hitting roadblocks due to proprietary black boxes.
@@theodis8134 It's easy to detect the cause of high CPU usage in Windows by using Sysinternals Process Explorer connected to MS public symbol server.
It was pure chance that a developer working on another database project noticed a discrepancy in connect times via ssh after an update.
AH! THERE'S A BACKDOOR THAT CAN TAKE OVER ANY LINUX COMPUTER IN THE WORLD!!!! (as long as it's running the very latest version of SSH, which it most likely isn't, and definitely better not be if it's a production server.)
Once a liar, always a liar. Dave (and his company) literally admitted "making false claims that the computer is at risk" in the court among other things. That was not the only thing, but I'm not going to test if he blacklisted keywords related to his heroic deeds, just google quoted text (include quotes) or his name on hackernews or reddit.
I'm not even sure Hanlon's razor is applicable to this video.
The difference is that it was detected and fixed. In closed source the NSA or any other agency can tell Microsoft to add a backdoor and nobody would ever notice it.
Dave, your quick ending worked. Your latest video finally appeared in my youtube feed, not some 2 year old video like I normally see.
For a very long time, it was possible to send a specifically crafted path to SMB and alter files you normally couldn't. Not really sure if it was a deliberate or accidental back door.
I think the point is that the back door was found and removed…because it was open source.
The vulnerability was with some code in one package that wasn’t everywhere. Microsoft has backdoors that are still there to this day…we ALL know this.
And don't forget closed source projects heavily depend on open source projects. They are equally exposed
"With something like Windows NT, you don't just check in code and wait for it to show up in the build"
That's an interesting comment coming from the guy who accidentally gave us the 32G FAT limit.
Good point. That was kind of the wild west. Still, I'm not sure I'd change that by the way, and they haven't seen fit to either, so I could argue it was the right choice :-)
@@DavesGarage The FAT (File Allocation Table) design is brilliant; extremely efficient and even possible to manually fix some problems in the case of cross-linked files. This was at a time when a "full gallon" computer had 64 kilobytes of RAM and the operating system needed to function in about 16 kilobytes or less. Anyway, this is why many embedded systems such as my oscilloscope still use FAT formatted USB storage. It is simple and efficient (and the patent expired! Anyone can use it; it is nearly universal).
@@thomasmaughan4798 and fun for a joke.
First FAT Table Corrupted. Wanna Try The Skinny One?
@@spvillano "First FAT Table Corrupted"
While a frequent occurrence, it was also fairly easy to manually fix. Of course, by then one of the cross-linked files was demolished but you could decide which one to keep.
It seems to be a deliberate and coordinated act of sabotage exploiting human trust and not some vulnerability specific to any operating system or software. So in short exploiting human trust is not a Linux OS limitation. Having said that the open source eco system may be more vulnerable to rouge actors than proprietary systems but the openness is also the factor that allowed the exploit to be caught so quickly. So there is a trade off. So maybe this happens less in proprietary systems (I can't really say) but if it does happen then only the proprietor can be relied upon to expose (and confess) the problem.
you may or may not have done a video on this, but what is your sense of Open Source vs. Closed Source? With your experience at MS, is it really safe, and the rumors you hear about back doors and bad code, etc true or just that rumor. Basically is Windows safer than Linux? or.. Good video, thanks for explaining.
The title of this video completely cracked me up! "When Penguins Cry" As a cross platform animal... supporting SBCs, Windows, Linux, and various type-1 hypervisors... I am routinely surprised how people believe false truths of the various platforms.... no platform is 'safe' in context to all threats. This issue and how it happened should be take as a major wake up call. On par with key CVEs that appeared in various type-1 hypervisors, which I was doing infrastructure/enterprise design for a fortunate 10... before I retired in 2018. The CVEs being found was one thing, but what we knew was possible, probable, existing as zero-day exploits would freak out the public, if they had any true understanding of the risks! Any malware being found, has an element of luck!
Not to mention that IP accessable boards are in many servers, and some of those have holes you can drive trucks through.
@@javabeanz8549 Yes, Dell iDRAC for example, had some major issues early on, as did HP iLO did in its initial variants. This is why the such 'lights out' access methods of servers has to always be (recommended) on an 'administrative' rail that is highly audited/monitored, just as access to VM management infrastructure (now) is.
And just think, even our cars are getting connected now. Isn't it great?
@@darkprinc979 not only are they connected, they are becoming tattletales... soon the cars will be able to check the speed limit of the zone you are in, and prevent you from exceeding that limit by more than ten miles per hour, and the cops will be able to just shut off your car, so you can't run from them. So what happens when criminals figure out how to hijack these "features?"
@@javabeanz8549 Don't forget "anti-drunk driving measures". I'm sure nothing could possibly go wrong with all of these "safety" features. Oh, and what happens when big brother doesn't like the opinions you've been expressing on the internet?
Didn't make it into build and not many if any big businesses use the latest and greatest. They are all most likely on stable but yeah this is pretty dam nuts. Someone put a back door into the make build script!
This didn't even affect Arch Linux, one of the most up to date distros
And I think that make build script wasn't even on the repo. So no commit. Afaict.
Hmmm the argument by @Dave's Garage that closed source development processes would have prevented this actually have little to do with closed vs open source. But instead into comprehensive testing and verification prior to a new release.
I would suggest that closed source projects are not necessarily better at this in most cases.
Also, I am sure it's worth pointing out Releasing a new OS version != releasing a utility in the area of testing and verification.
SO:
1. NOT a successful point toward closed source being more secure.
2. Definitely a point toward full testing and verification being valuable as one tool to help detect similar security holes/backdoors, (the kind that cause performance degradation anyway)
I like the quick stop... Thanks for trying something new!
For the ultimate vulnerability see Ken Thompson's Turing Lecture "On Trusting Trust," for why reading the source code doesn't help.
let me summarize your thoughts:
implications are catastrophic - only bleeding edge unstable distros are affected
discovered only by accident - but very quickly in a package few people are caring about
backdoor with private key access gives ultimate access to any linux system, obfuscated as test data, uses its own state machine for injection - it could have been easily engineered to not peak cpu usage during BVT
hacker used years of social engineering to develop trust in the community - same vulnerability is there in closed source code
The exploit was so sophisticated I'm willing to bet no Microsoft engineer is sharp enough to have caught it. The regression testing might have flagged it but then again I doubt that MS engineers are paying that much attention.
You're wrong actually, the vulnerability was not to ssh into any computer, but to get remote root command execution on any computer serving sshd.
The command that can be run has to be small, part of the handshake payload.
Technically, no... not ANY system. It is specific to sshd. And, it didn't affect all distros.
You and LowLevelLearning have both earned my sub this week. Thanks to both of you for being awesome.
I wonder if Microsoft would have recognized the backdoor if it had trickled down into Windows Subsystem for Linux and ended up in a lot of Windows developer workstations and possibly servers with Openssh inside....
The backdoor wasn't checked into git at all; it was snuck into the release tarball. Building xz from git was still perfectly fine. The release tarball did not reflect the state of the git repo, hence this was called a "supply chain" attack.
Oh boy, "Assuming that Microsoft is more careful today". The comedy writes itself... Dream on, Dave.
As far as I am aware, this didn't make it into any stable linux distros. It was only in some unstable / in development versions. It feels a little uncharitable to call it a "linux backdoor" just like a vulnerability in a windows test build isn't a general problem with windows.
as a Linux user I have to tell you, that you're wrong on that.
In Linux there are many distributions (distros), and each follows their own path of releasing. But generally speaking there are the so called bleeding edge distros, that don't have long testing / unstable period - and there are many such distros - like Arch, OpenSUSE, etc.
So, many of those are affected. I know a lot of people already did complete reinstall, because it is not certain yet what are the full implications of this malware, does it affect only the sshd or also other aspects of the system.
So, the affected systems are a lot, actually.
However, the other bulk of distros are the so called long-term releases (LTS) - like Ubuntu, Debian, etc. Those were not affected by the malware, because they use older versions of xz, which is not infected. Most servers on the Internet use such LTS distros, so most of the online servers are safe from this vulnerability.
@@moetocafebut Arch does not use openssh with liblzma, so arch's SSH was not affected by this. Main targets were debian-like systems (Kali, unstable debian/ubuntu etc). Yes, you still had a backdoored package, but it wasnt that dangerous if you didn't use it explicitly. Not to mention that it was very quickly updated (at least on Arch), same with downgrading, its just a one command to downgrade a package to a older version if you're paranoid. Sure we don't know everything, but thats the optimistic thoughts.
More intresting thing is that's probably one of the few known vulnerabilities, and normies are bitching on Linux, while currently daily driving their totally-non-backdoored and privacy respecting Windows or Apple products lol
@@archie-fu7jl yes, Arch sshd is presumably not affected, but it's not 100% certain yet. So better update (or downgrade) than sorry.
This kind of video is the reason I watch your content, even though I'd abandoned Windows long ago, in favor of the Penguin OS! Keep up the good work!
The closed source equivalent is an exploit that no one caught being in MS Exhange Server for years, Microsoft discovering it, silently patching Office 362, but not releasing a patch for On-Premise Exchange for months and months until the prescheduled sales of stocks for the CEO go theough first.