secret backdoor found in open source software (xz situation breakdown)

แชร์
ฝัง
  • เผยแพร่เมื่อ 23 ธ.ค. 2024

ความคิดเห็น • 1.4K

  • @LowLevelTV
    @LowLevelTV  8 หลายเดือนก่อน +1047

    Seriously guys thank you for watching, the channel has been exploding these last few days. ( come learn C at lowlevel.academy 🥺)

    • @Hhhhh-v9z6c
      @Hhhhh-v9z6c 8 หลายเดือนก่อน +12

      Keep bringing this type of videos😽😽

    • @Zaf9670
      @Zaf9670 8 หลายเดือนก่อน +20

      Seems like security issues have been exploding these last few days!

    • @TheMatias314
      @TheMatias314 8 หลายเดือนก่อน +3

      Keep up these videos, they are amazing!

    • @seriouscat2231
      @seriouscat2231 8 หลายเดือนก่อน +4

      I think Freund rhymes with Freud. Any Germans here quickly correct me if I'm wrong.

    • @covoeus
      @covoeus 8 หลายเดือนก่อน +9

      0:14 AFAIK OpenSSH does not directly depend on lzma. Some distros (e.g. Debian) patch libsystemd into OpenSSH for readyness notification, which in turn depends on liblzma. Upstream ssh should be safe. More on this in the HN thread.

  • @AlexBrugh
    @AlexBrugh 8 หลายเดือนก่อน +4142

    The guy who found this has just been elevated from "just a guy" to "security researcher" by action alone

    • @LowLevelTV
      @LowLevelTV  8 หลายเดือนก่อน +616

      100%

    • @gingeral253
      @gingeral253 8 หลายเดือนก่อน +86

      I wonder where he got the skills to understand this stiff

    • @pixelcatcher123
      @pixelcatcher123 8 หลายเดือนก่อน

      @@gingeral253internet and motivation bro

    • @innovationscode9909
      @innovationscode9909 8 หลายเดือนก่อน +194

      It’s very very very very disrespectful to say he is just a a guy. Foolish commentary in that regards

    • @Krmpfpks
      @Krmpfpks 8 หลายเดือนก่อน

      @@gingeral253hi definitely is an accomplished developer who stumbled upon the Valgrind issue and was motivated enough to investigate and then find it. This guy probably wanted to help fix a memory issue and had the perseverence to go to the bottom of it.

  • @Krmpfpks
    @Krmpfpks 8 หลายเดือนก่อน +1219

    Kudos to this „just a guy“ who probably just wanted to help out an open source project in fixing a memory leak and performance issue and then probably spent weeks of his time getting to the bottom of this.
    This is what makes all of us safer. He deserves a medal or something.

    • @XGD5layer
      @XGD5layer 8 หลายเดือนก่อน

      The last commit related to this was three weeks ago, but I can't check when the first commit was because GitHub has quarantined the repository

    • @HelloWorld-fg2nm
      @HelloWorld-fg2nm 8 หลายเดือนก่อน +23

      Nobel Prize honestly

    • @cigmorfil4101
      @cigmorfil4101 8 หลายเดือนก่อน +25

      Just think if he had found such a bug in Windwos he'd have been arrested and charged as opposed to being thanked.

    • @Krmpfpks
      @Krmpfpks 8 หลายเดือนก่อน

      @@cigmorfil4101 well, Microsoft has a bug bounty program, as most big companies have nowadays. The ‚just a guy‘ is Andres Freund, some blogs call him ‚microsoft security researcher‘.

    • @BirkinIdk
      @BirkinIdk 8 หลายเดือนก่อน +70

      @@cigmorfil4101 What is one single shred of evidence that leads you to believe that? This has gotta be one of the most absurd things I've heard.

  • @_JohnHammond
    @_JohnHammond 8 หลายเดือนก่อน +1118

    BLAZINGLY FAST NEWS REPORTING

    • @LowLevelTV
      @LowLevelTV  8 หลายเดือนก่อน +146

      BREAKING NEWS

    • @0x9D99
      @0x9D99 8 หลายเดือนก่อน +7

      Could definitely add that to title lol

    • @smnomad9276
      @smnomad9276 8 หลายเดือนก่อน +5

      @_JohnHammond Reverse engineer it and make a video about it!

    • @Purely_Andy
      @Purely_Andy 8 หลายเดือนก่อน +1

      every time i read your name i read it in the moonbase alpha tts "john madden" voice

    • @DePhoegonIsle
      @DePhoegonIsle 8 หลายเดือนก่อน +1

      X.x What interesting events, a tinfoil hat might start to think supply chain issues, and notice the same kind of ticks in other things like 'heartbleed bug' and just how 'slightly off', but not enough it was to be spotted by all but the most ocd or diligent person for their own thing.
      Just sayin.... Maybe Open source is more at danger of being infected by forign bodies than closed source, and taken for safer .. when most don't have a single clue about wtf is going on, or just why an issue or even performance hick-up exists. Who would ever be absurd enough to question 1-2 second single hick-up in a sign in process for something meant to exist on the internet where that delay would be unreasonable to impossible to spot in the live environment.
      Speaking of cool things, Isn't it great that there are so many easy to access cloud-based hosting solutions meant for personal & small business users? That convenance is great, just have to deal with a tiny bit of internet lag because tech stuff. I mean it would suck if they all used a common supplier that managed access in some way or something, or data security... *cough cough*, because some big big target would be pained on the backs of such a supplier and be closely watched.
      Man, the amount of qq from governmental bodies about data encryption and traffic routing has sure gone down, I bet they only realized how foolish it was for them to keep on that. I guess all is well, and the days of the governmental bodies pulling overly complicated & deep state affairs on the people they care about is over for our own people right?

  • @standingpad
    @standingpad 8 หลายเดือนก่อน +858

    Important Clarification (since I feel this isn't clarified): upstream OpenSSH doesn't use liblzma, however many distros like Debian patch OpenSSH to use SystemD Notifications through libsystemd, which in turn uses liblzma. Distros like Arch (which don't patch OpenSSH) or distros without SystemD like Void should be fine with regards to SSH (however most distros are already downgrading xz anyway for obvious security reasons)
    Source: the latest Arch News post regarding this backdoor
    EDIT: to quote directly from the Arch news post:
    "Arch does not directly link openssh to liblzma, and thus this attack vector is not possible"

    • @bzuidgeest
      @bzuidgeest 8 หลายเดือนก่อน +11

      ​@@mx338but if your systems are sufficiently firewalled... What is the exploit going to do?
      This is not a good situation, but it won't break the world either. Vulnerabilities are found almost every day it seems in so much software.

    • @gagagero
      @gagagero 8 หลายเดือนก่อน +14

      @@Finkelfunk Install Gentoo.

    • @epolpier
      @epolpier 8 หลายเดือนก่อน +23

      @@mx338 You did not understand a word from this right. This relates only to very latest versions, not stable ones so NO servers are not compromised...

    • @mx338
      @mx338 8 หลายเดือนก่อน +14

      @bzuidgeest you're right, ideally you do not expose SSH to the internet without restrictions, I run everything on prem for work and don't allow SSH from the internet. However in cloud computing that's often different, if you run and AWS EC2 or any other cloud VPS, SSH is going to be exposed to the internet by default.

    • @gagagero
      @gagagero 8 หลายเดือนก่อน +12

      @@Finkelfunk Is this what the children call a "skill issue"?

  • @kim15742
    @kim15742 8 หลายเดือนก่อน +831

    As horrifying as such backdoors in widely used software is, the engineering behind such exploits is insanely impressive and creative

    • @twentylush
      @twentylush 8 หลายเดือนก่อน +196

      Using a single makefile line that looks weird but not totally out of the ordinary is so creative. On the other hand how was this mf able to commit compressed binaries without getting nuked by a maintainer, i do that on accident occasionally on my branches and it draws the aggro of every devops guy in a 10 mile radius when i submit the PR

    • @SuperBotcreator
      @SuperBotcreator 8 หลายเดือนก่อน +62

      @@twentylush LMAO draws the agro bro im stealing that one

    • @fnige
      @fnige 8 หลายเดือนก่อน +30

      @@twentylush Youre just simply a devops guy magnet

    • @TheYahmez
      @TheYahmez 8 หลายเดือนก่อน

      @@SuperBotcreator MrsBest, Penguins1 & 555NipperDog's farts are better than you 😋

    • @adicsbtw
      @adicsbtw 8 หลายเดือนก่อน +56

      @@twentylush they passed it off as a test case. It makes sense at a quick glance, having a premade file that _should_ decompile properly, then using that in testing to see if everything works as expected
      it's just that the file wasn't only used for testing

  • @9a3eedi
    @9a3eedi 8 หลายเดือนก่อน +294

    I can already read online people saying that this is proof that open source code is less secure than proprietary
    On the contrary, the fact that it was caught, and caught relatively quickly, shows that open source is more secure against these kinds of backdoor attacks

    • @shApYT
      @shApYT 8 หลายเดือนก่อน +36

      Out of sight out of mind

    • @erek
      @erek 8 หลายเดือนก่อน +21

      "this" was caught but there could be many more backdoors which will never be discovered. Imagine if a war breaks out between countries. Of course no government agency would trust proprietary software during an active war. In that scenario, the country that managed to put backdoors in open source software will have immense advantage.

    • @jfbeam
      @jfbeam 8 หลายเดือนก่อน +32

      Indeed.If someone can do this in code that's opening visible, imagine the shit that goes on behind the curtain in non-opensource crap. (Solarwinds comes to mind.)

    • @aldproductions2301
      @aldproductions2301 8 หลายเดือนก่อน +10

      Also, we have a clear method to identify and track the malicious actor's influence. We know what account they're using, and can thus see every open source project they have touched.

    • @9a3eedi
      @9a3eedi 8 หลายเดือนก่อน

      @@aldproductions2301 with that particular account, maybe. People can make as many accounts as they like

  • @Airatgl
    @Airatgl 8 หลายเดือนก่อน +934

    How many more security issues are going to be found this week?!

    • @no_name4796
      @no_name4796 8 หลายเดือนก่อน +82

      Life is a security issue

    • @dkkogmaw1311
      @dkkogmaw1311 8 หลายเดือนก่อน +21

      They asked ChatGPT

    • @bigboi1004
      @bigboi1004 8 หลายเดือนก่อน +41

      security issues are just skill issues in disguise

    • @yoshi314
      @yoshi314 8 หลายเดือนก่อน +3

      all of them.

    • @thatguy7595
      @thatguy7595 8 หลายเดือนก่อน +13

      @@bigboi1004 and unsafe language issues passed off as skill issues

  • @CCCW
    @CCCW 8 หลายเดือนก่อน +1032

    It's not just SSH. The dev seems to be suspect in way more packages.

    • @StupidusMaximusTheFirst
      @StupidusMaximusTheFirst 8 หลายเดือนก่อน

      After this, he better be. This is really well designed malware. The ways they are cleverly hiding it, all the involved packages, the linker, removed the symbols from the malware in xz, using the testing branch, those mentioned in the video, or the fact that it's not even in libssh, and it only works if you run through systemd which systemd would be compiled to use xz. That's really well orchestrated, that's gov malware I think.

    • @brians7100
      @brians7100 8 หลายเดือนก่อน +54

      is his github account still active?

    • @sarah867
      @sarah867 8 หลายเดือนก่อน

      @@brians7100 Some of his suspicious push requests date back over 2 years, so its possible this isn't the first time he has tried to do something like this, just the first time he got caught.

    • @sergioezquerro
      @sergioezquerro 8 หลายเดือนก่อน

      ​@@brians7100 Even if he isn't active anymore, the damage is done and needs to be detected and reverted in each repository this guy has committed to.
      I bet it'll last some time as an active issue until things are resolved for most packages affected, and many machines will be compromised until then

    • @nikhilchouhan1802
      @nikhilchouhan1802 8 หลายเดือนก่อน

      @@brians7100recently suspended

  • @laurensdehaan2202
    @laurensdehaan2202 8 หลายเดือนก่อน +40

    Kudos to the person who found this. His modesty does him credit, but the realization that something was amiss, the desire to delve into it, the analytical process, depth of research, and the willingness to share what he discovered with the wider community - totally aligned with the OSS ethos - shows he definitely has the right mindset to be a security researcher and has hit the ground running. Hats off, sir; bravo!

  • @kuhluhOG
    @kuhluhOG 8 หลายเดือนก่อน +180

    as a sidenote from what I gathered: The person who "contributed" this backdoor was not just some person who randomly came out of nowhere with a Merge Request. It was someone who contributed to the project for an extended period of time to a point where they themselves became a maintainer (not the main one, but projects like this often have multiple).

    • @JordanPlayz158
      @JordanPlayz158 8 หลายเดือนก่อน +19

      What a poor misuse of trust and time to achieve this

    • @wasd____
      @wasd____ 8 หลายเดือนก่อน +21

      @@JordanPlayz158 The contributor probably thought someone would pay them a lot of money to achieve this.

    • @themoviesite
      @themoviesite 8 หลายเดือนก่อน +24

      @@wasd____ It was his mission.

    • @Pharoah2
      @Pharoah2 8 หลายเดือนก่อน +34

      @@JordanPlayz158they were obviously being paid already for this. Plenty of maintainers are on the payroll of Google, Microsoft, Redhat etc. This is just the first time a malicious group decided to go that route as well

    • @fabriziob969
      @fabriziob969 8 หลายเดือนก่อน

      @@Pharoah2 just the first time? Lol do you think that between the thousands of FAANG+ employess none of them is on the payroll of some 3 letters agency?

  • @bjduncc
    @bjduncc 8 หลายเดือนก่อน +3285

    damn, NSA taking a bunch of loses recently

    • @heroclix0rz
      @heroclix0rz 8 หลายเดือนก่อน +406

      They'll always have windows!

    • @kreuner11
      @kreuner11 8 หลายเดือนก่อน +74

      Bro not every vulnerability ever is the NSA?

    • @rstewa35
      @rstewa35 8 หลายเดือนก่อน +273

      @@kreuner11 lol you’re taking his comment too literally

    • @user-lj4lo7cx7m
      @user-lj4lo7cx7m 8 หลายเดือนก่อน

      ​​@@kreuner11
      ______
      Joke ---/ you \--->

    • @sirrah9533
      @sirrah9533 8 หลายเดือนก่อน +344

      NSA is a reasonable suspect, in all honesty.

  • @swannie1503
    @swannie1503 8 หลายเดือนก่อน +164

    Idk if I would call Andres Freund “some guy” haha he’s a Postgres contributor/developer and Principle SWE at Microsoft. I get your point though. Not technically a security researcher.

  • @Neura1net
    @Neura1net 8 หลายเดือนก่อน +466

    This channel is great for my daily dose of anxiety

    • @haifutter4166
      @haifutter4166 8 หลายเดือนก่อน +5

      Yeah. Just got the news about it 9h ago. My Manjaro installation has the aforementioned versions installed and now I don't know what to do. Rollback, or directly wipe the system and return to some more newbies OS like Mint.

    • @hthring
      @hthring 8 หลายเดือนก่อน

      lol

    • @xCheddarB0b42x
      @xCheddarB0b42x 8 หลายเดือนก่อน

      You are not nearly anxious enough.

  • @cole.maxwell
    @cole.maxwell 8 หลายเดือนก่อน +53

    Loving these security vulnerability breakdowns man! I can only imagine the effort and background knowledge it takes to put content like this together, thank you.

  • @PavelMalinov
    @PavelMalinov 8 หลายเดือนก่อน +5

    Thanks! Please make a video on the topics of object file/linker crc and most of the topics you talked about, A deep dive will be amazing!

  • @basicallyeveryone
    @basicallyeveryone 8 หลายเดือนก่อน +282

    1:12 "He's not a security researcher, he's not a malware reverse engineer, he is just a Freund."

    • @xlxl7866
      @xlxl7866 8 หลายเดือนก่อน +10

      He is definitely an experienced developer

  • @ivanov83
    @ivanov83 8 หลายเดือนก่อน +7

    That is definitely one of the most crazy, complex, and sophisticated backdoor injection attempts I ever seen in my life. The engineering behind it is very impressive. The guy who discovered that deserves a reward, he just literally saved the world

  • @jimmykochi6442
    @jimmykochi6442 8 หลายเดือนก่อน +89

    After this report there were some extra findings that people might feel valuable.
    1. Some time ago the only mantainer got a lot of pressure from different users to accept another mantainer to the project.
    2. The binary files in question were compressed files that are usually made to test that the decompressing tool was working, in this case the malicious tests were two, one for a large file, and another for a corrupted file. As you can see both made for making it difficult to find that there was hidden code there.
    3. The malicious mantainer got enough trust to be able to sign the distributed tars with malicious code and to contact linux distro mantainers to presure them to update to the backdoored version, the attacker even sent a patch to the google repo that was harmless by itself but a requirement for the exploit.
    4. After the backdoor was created random accounts submitted prs to
    different projects to update to the vulnerable version.
    This was a large well orquestated attack that was most likely planned by more than one person and only discovered due to it having performance problems and certain bugs, otherwise we might have never noticed.

    • @Dratchev241
      @Dratchev241 8 หลายเดือนก่อน +7

      honestly sounds like a state actor to me.. and with the names being used I suspect CCP.

    • @05Matz
      @05Matz 8 หลายเดือนก่อน +20

      @@Dratchev241 Eh, if you're going to do an intelligence operation involving names but no face-to-face contact, you'd probably use a foreign name. I'd definitely say a state actor, but frankly, it could be anyone. I was leaning towards someone Western but smaller, but I really have nothing to base that on.

    • @warhawk_yt
      @warhawk_yt 8 หลายเดือนก่อน +12

      Yeah this was definitely orchestrated and planned for a long time and took a lot of planning. They just got unlucky that someone noticed the performance decrease and investigated and found it by pure accident.

    • @Bramble20322
      @Bramble20322 8 หลายเดือนก่อน +1

      @@Dratchev241 Hah, because obviously Chinese intelligence agencies would use a fucking Chinese name to do this, right?
      If anything this was probably done by american intelligence agencies using an asian name to generate propaganda headlines if it didnt work.

    • @christianbarnay2499
      @christianbarnay2499 8 หลายเดือนก่อน +6

      @@Dratchev241 You don't need a state actor to create and coordinate multiple accounts. There is no proof that those multiple accounts requesting the update on client programs belonged to different people. It only requires coordination. But one person can easily coordinate with themselves.
      Your point 1(pressure on the single maintainer of a core library to accept other contributors) is actually what you should expect. You don't want a single unsupervised individual to be the alpha and omega of such a central library with zero backup. That's what actually scares me about OSS and Linux. There are so many core technical libraries that are maintained by a very small group of persons. Any rogue in one of those core groups can cause wide damage. There way too little people interested in working in those dark places where you get no recognition when everything works and all the blame when something goes bad in another software just because someone else used your library in a way that it was not supposed to be used.
      Point 2 is the actual issue: porous build process.
      Test files stored in the test directory should never find a way into the compiled binary. There should be a stronghold protecting the source path and compilation tools so that nothing outside of that scope can get into the binary files that are signed and published.

  • @daniels-mo9ol
    @daniels-mo9ol 8 หลายเดือนก่อน +205

    Bullet dodged. For real. This could have been the worst money grab backdoor by far. It's literally in every system. It's especially scary that the project owners approved the compiled binaries. Hopefully it's not a maintainer behind this.

    • @kienanvella
      @kienanvella 8 หลายเดือนก่อน

      It's literally the project owners or a representative of the project owners org.
      Both of the main devs that are the contact points for package maintainers have had their GitHub accounts suspended reportedly

    • @trofl
      @trofl 8 หลายเดือนก่อน +127

      It sounds like the original, single maintainer was overloaded and started passing off duties to another contributor who "helpfully" stepped in recently to reduce the load. Fast forward a few months and it looks like this helpful contributor was (likely) a state agent...

    • @Dratchev241
      @Dratchev241 8 หลายเดือนก่อน +21

      @@troflquestion is which state agent... I would say in likely by order Mossad, then the NSA/CIA/MI6 then FSB and or MSS-CCP

    • @jsrodman
      @jsrodman 8 หลายเดือนก่อน +56

      They aren't compiled binaries. They're compressed files. Which is totally normal for testing a compression tool.
      What was missed was potentially looking into the contents of the compressed files, which I know very few people would bother to do, or looking at the quality of the new tests. If you're just making existing code paths faster, why do you need new compressed files?
      Edit: auto-incorrect

    • @9a3eedi
      @9a3eedi 8 หลายเดือนก่อน +36

      You realize that this backdoor was caught but so many other could be successfully installed in open source code without our knowledge and without extremely deep code reviews. Though I would expect that other project would start scrutinizing things more deeply in the next couple of weeks and we'll be hearing more surprises

  • @mrtnsnp
    @mrtnsnp 8 หลายเดือนก่อน +194

    It is slightly more subtle, from what I understand. It is not that openssh uses liblzma, but liblzma is used in systemd. On systems where openssh is patched to use systemd as well, you end up with a security issue. This appears to be limited to the combination of x86_64 and linux and systemd. That is still a significant fraction of all linux systems.

    • @HUEHUEUHEPony
      @HUEHUEUHEPony 8 หลายเดือนก่อน +13

      openrc saved the day

    • @damouze
      @damouze 8 หลายเดือนก่อน +33

      That does not surprise me at all.
      Systemd is one giant security issue. The biggest security issue about it being the megalomanic ego of its creator.

    • @damouze
      @damouze 8 หลายเดือนก่อน +1

      @@thelvadam5269 Ever heard of black humour? 👹

    • @samuelwaller4924
      @samuelwaller4924 8 หลายเดือนก่อน +2

      ​@ok-tr1nwthis is a very specific backdoor that only affects software that uses lzma and does RSA with this specific library. How many programs fit that description...

    • @Axman6
      @Axman6 8 หลายเดือนก่อน +4

      From the Openwall discussion:
      I think that distros should be very careful when they start patching openssh in general.

  • @sanctionedforce1868
    @sanctionedforce1868 8 หลายเดือนก่อน +57

    so if I'm understanding this, the intent is to be able to use a specific key that doesn't have to be installed [legitimately] on a system (no direct attack necessary) to effectively gain "authorized" access to a system. to simplify it, this behaves like a master key or a master lock (iykwim).

    • @drarko91
      @drarko91 8 หลายเดือนก่อน +5

      Exactly

    • @SmokeyCosmy
      @SmokeyCosmy 8 หลายเดือนก่อน +8

      Not exactly technically speaking, but the exact same effect in practice. The difference, fortunately, being that this way the attack can only happen under specific conditions. For example, some environment variables can make this not run, since the attack depends on specific code being injected. Of course, even with this stroke of "luck" it still is one of the most serious attacks/vulnerabilities found in the last couple of years.

    • @statelessdev
      @statelessdev 8 หลายเดือนก่อน +2

      ikwym wrt lpl reference :)

    • @IlCapodOro
      @IlCapodOro 8 หลายเดือนก่อน +2

      Apparently, this leads to RCE rather than just auth bypass

  • @timgerk3262
    @timgerk3262 8 หลายเดือนก่อน +50

    The malicious commit was made Feb 23 and entered xz-utils stable on Feb 24. Detected in Debian-unstable.
    5 points for early detection, before it should have been distributed to production anywhere?

    • @jfbeam
      @jfbeam 8 หลายเดือนก่อน +10

      There was no "commit". The hack was only placed in a non-rev controlled tarball that other projects then imported.

  • @Cobinja
    @Cobinja 8 หลายเดือนก่อน +35

    Andres Freund, who found and reported this, should be called "The XZorcist".

  • @cherubin7th
    @cherubin7th 8 หลายเดือนก่อน +148

    Crazy to think the test can change the build. This should be clearly separated.

    • @alsweetex
      @alsweetex 8 หลายเดือนก่อน +19

      I agree. Even the build systems I maintain for PHP code run tests and then (if the tests passed) checks out the Git repo again in order to do final deployment actions on a clean slate.

    • @DevynCairns
      @DevynCairns 8 หลายเดือนก่อน +14

      it doesn't, really... the test fixture just exists in the repo doing nothing, but with the modified build script, that test data is used to reconstruct the malicious code. in fact the test fixture in there isn't even used at all in a test (which should have been sus)

    • @platinummyrr
      @platinummyrr 8 หลายเดือนก่อน +14

      the actual git tree of the project doesn't have the M4 line which triggers this. The M4 line was added to the public source tarball post release by someone who had keys to do that.

    • @gosnooky
      @gosnooky 8 หลายเดือนก่อน +2

      Yeah, for sure a no-no, but perhaps this was by design. Harder to notice things when you assume everyone follows the rules.

    • @isbestlizard
      @isbestlizard 8 หลายเดือนก่อน +5

      Right??? Build processes shouldn't allow random extra .o's not generated from sources to appear :O

  • @plateoshrimp9685
    @plateoshrimp9685 8 หลายเดือนก่อน +126

    Apologies if I missed it, but it should be clarified that this did not make it into any production releases. The fact that it was caught before release is a demonstration of the strength of the open source model.

    • @haifutter4166
      @haifutter4166 8 หลายเดือนก่อน +8

      What about Manjaro?

    • @gregoryreimer869
      @gregoryreimer869 8 หลายเดือนก่อน +31

      I'd say we got very, very lucky that someone noticed some odd behavior and went 10 extra miles tracking it down.
      In fact I'd say this was a pretty big failure in terms of the model. As it looks like there is there was no verification that the source being used was even the same as the source in the repository(You know, the stuff people usually check for things like this). If it had been complied with the repo then it wouldn't have been an issue at all(vestigial, non executing code aside).
      Kind of clever really. Leave out the easiest to see bits from the source. Slap them in the release tarball and as long as the behavior isn't too different from the original nobody notices(and if it is you can probably just patch and blame it on something else like the did in 5.6.1)
      So all the normal checks and balances that would normally catch this in OSS(as far as I know) just don't work here. Only weeks after release and by blind luck did this get caught and that's not something to celebrate.
      Say what you will about closed source, but at least the back doors in their software are intended by the company ;)

    • @danbopes6699
      @danbopes6699 8 หลายเดือนก่อน +7

      OpenSUSE was vulnerable. MicroOS was vulnerable as a result. It's an os built for containers targeted at scalable services and bigger entities. Curious how many kube clusters got taken over.

    • @nikhilchouhan1802
      @nikhilchouhan1802 8 หลายเดือนก่อน

      @@haifutter4166 As someone stated here Arch doesn't use patched OpenSSH and since manjaro upstreams directly from Arch, I would say it didn't affect it. For Fedora on the other hand, it affected Fedora 40 beta in a few flavours, rawhide and silverblue (iirc?) being two of those. Older versions did not get affected

    • @XGD5layer
      @XGD5layer 8 หลายเดือนก่อน +18

      @@gregoryreimer869 A rogue developer could do the same in closed source software, and we'd have even less of a chance to catch them.

  • @libredove
    @libredove 8 หลายเดือนก่อน +30

    OpenSSH doesn't use liblzma directly. Some GNU/Linux distros patch it to use libsystemd and that uses liblzma.

    • @rj7250a
      @rj7250a 8 หลายเดือนก่อน +5

      And most stable distros like Debian and Ubuntu LTS, do not use the latest version of programs, so not affected.

    • @libredove
      @libredove 8 หลายเดือนก่อน

      @@rj7250a And the OpenSSH upstream is unaffected since systemd is not present there. :) (OpenBSD)

    • @DDracee
      @DDracee 8 หลายเดือนก่อน

      @@rj7250a it was officially in debian-unstable, so it was gonna be in the next main release

    • @_garicas
      @_garicas 8 หลายเดือนก่อน +2

      Well noted

  • @feelsunbreeze
    @feelsunbreeze 8 หลายเดือนก่อน +2

    Dude I'm loving this exploits news and guide. You're really helping me get a better understanding of these things!

  • @geno755
    @geno755 8 หลายเดือนก่อน +9

    Wow - this is absolutely crazy. Never would have thought that this passes a review. Obfuscated code… I have thought several times - e.g. when people talk about hardware wallets’ software being open source - you dont know what ends up on the device, and even if the source is fine, you dont know what happens during the build process. A case like this shows me this suspicion isnt that unrealistic. Thank you for the video - I subscribed.

  • @StingSting844
    @StingSting844 8 หลายเดือนก่อน +135

    These videos are great. But please add more context or explanation about how these exploits could be used. I forwarded your previous video on kernel exploit to the management and they came back saying "we dont understand the implications from this"

    • @italianbasegard
      @italianbasegard 8 หลายเดือนก่อน +13

      It’s pretty simple, they could ssh into your server. What you have running on that server and whether or not they could they could perform privilege escalation would be required knowledge to determine what the bad actor “could do”, and from that would you determine the implications.

    • @pavek1865
      @pavek1865 8 หลายเดือนก่อน +4

      both of these are only doable if the attacker already has access to the network and most companies have an encrypted secure vpn and the ssh would be in the vpn right.. ?, so both are relatively harmless from external attacks, they do have to worry about sabotage from inside. Even then records would be kept and the culprits found.

    • @arthurmoore9488
      @arthurmoore9488 8 หลายเดือนก่อน +3

      How does management feel about everyone in the company who uses those servers having access to all of their files, including customer data? From an insider threat perspective, that's what we're talking about. A larger concern would be if one site gets hacked, then everything on that server would be hacked.

    • @JuanWayri
      @JuanWayri 8 หลายเดือนก่อน +8

      @@italianbasegard that's doesn't provide enough actionable context. It should include "in bold" the identifiers of the compromised releases. It wouldn't take too much effort to highlight that this only about "RSA's implementation" + "liblzma [...] tarballs for 5.6.0 and 5.6.1" and NOT just "SSH libraries".

    • @jeffreyblack666
      @jeffreyblack666 8 หลายเดือนก่อน +2

      @@italianbasegard I don't think they would require privilege escalation.
      From my understanding of the video, they could hypothetically log in as root, using their key, and get root access.

  • @TheDeveloperGuy
    @TheDeveloperGuy 8 หลายเดือนก่อน +95

    That’s why I say code (and commits) must be as dumb, clean and understandable as possible. No overcomplication.

    • @CFSworks
      @CFSworks 8 หลายเดือนก่อน +35

      The backdoor payload was hidden among dozens of innocent compressed test files, and the hook to execute the script that embedded the payload was slipped into a makefile in the release tarball but not committed to Git. We absolutely should be in the habit of diffing our release tarballs vs. the Git tags from now on, but we're going to need to come up with a smart way of analyzing any high-entropy files checked into Git for potential hidden payloads.

    • @twentylush
      @twentylush 8 หลายเดือนก่อน +6

      @@CFSworks I feel like its a failure of keeping up with the times though! this exploit is a complete dead-end if you policy-out artifacts in code repos and instead only pulling them in containerized environments when the "tests" are actually run. Not to say its all the maintainers fault, I mean the OG seemed pretty burnt out and overloaded, and there's not a lot of up-to-date devops guys with a golden-heart and clear schedule to bring these OS projects to the modern ci/cd era

    • @CFSworks
      @CFSworks 8 หลายเดือนก่อน +3

      ​@@twentylush I suspect that an attacker trying to pull off an attack like this in the future would just switch to hiding the payload in a non-artifact binary file instead. Perhaps a base64 chunk hidden in the metadata of the project's logo, or a loose object under .git. There are too many opportunities for outright steganography here. :/

    • @absurdengineering
      @absurdengineering 8 หลายเดือนก่อน +7

      @@CFSworksContinuous integration should be the source of tarballs FFS. Nobody should be uploading those manually.

    • @CFSworks
      @CFSworks 8 หลายเดือนก่อน +3

      ​@@absurdengineering Definitely. But here we had a malicious maintainer, so even with auto-upload of the git-archive output, they would have just secretly replaced the tarball 3 seconds afterward with the backdoored version.

  • @dhillaz
    @dhillaz 8 หลายเดือนก่อน +100

    If it was missed in popular OSS, imagine how easy it is for skilled bad actors to sneak these things into closed source products!

    • @plateoshrimp9685
      @plateoshrimp9685 8 หลายเดือนก่อน +38

      Especially when the bad actors are the companies who make the products!

    • @balsalmalberto8086
      @balsalmalberto8086 8 หลายเดือนก่อน +18

      Windows has left the chat.

    • @DeathSugar
      @DeathSugar 8 หลายเดือนก่อน +3

      I'm not sure if xz could be considered popular OSS. Popular package, maybe, yes. But not project itself.

    • @Bramble20322
      @Bramble20322 8 หลายเดือนก่อน +1

      Microsoft employees have names and adresses, they'd be found on an audit, fired, blacklisted and possibly jailed for something like this. Open source maintainers? Good luck going after a dude you're not even sure exists on the other side of the world.

    • @christianbarnay2499
      @christianbarnay2499 8 หลายเดือนก่อน +7

      That is actually a huge problem of OSS. Many "popular OSS libraries" (aka core building blocks used by hundreds of projects) are actually not popular at all. They are maintained by 1 or 2 people at best and everyone else just blindly integrates those libraries into their project trusting that some magic being has actually independently reviewed and validated what those guys did. And for that exact reason those libraries are very easy to infiltrate and alter for rapid damage on a lot of projects at once. There is no actual incentive outside of pure benevolence for someone to actively participate in securing some of the most technical and sensitive core blocks of the OSS architecture.
      Being open or closed software is irrelevant here. It's an organisational issue of the dev teams. Everybody wants reviewed binary that they can trust but nobody wants to be the one that spends time doing the actual review.

  • @mathgeniuszach
    @mathgeniuszach 8 หลายเดือนก่อน +58

    The problem here is accepting commits that run binary executables. That should have been suspicious from the get go.

    • @SmokeyCosmy
      @SmokeyCosmy 8 หลายเดือนก่อน +20

      No. The problem is patching security sensitive programs to use 3rd party libraries and, of course, not letting the most important daemon on the system (systemd) that handles everything using 3rd party libraries. There's a reason people are monitoring openssh a lot more then some compression library and this problem wouldn't have been found for who knows how long if a persons OCD hadn't kicked in to thinking a program takes too long to open (half a second).
      It shouldn't have happened and on systems that didn't try and screw around just to make a config file look prettier, it didn't.

    • @MrFujinko
      @MrFujinko 8 หลายเดือนก่อน +2

      Yeah. Why not commit a BIOS firmware update at this point? lmao

    • @Sypaka
      @Sypaka 8 หลายเดือนก่อน +4

      @@MrFujinko Actually... I wonder, if someone could Chainfuck EFI. You just need a lib, which is most likely run as root. So the same crap, but write code, which drops a EFI binary into /boot and hooks that into NVRAM and the loader, which in turn reloads or mimics GRUB. Part of this EFI binary goes straight to Ring -3 on AMD or Ring -1 on Intel, acting as a hypervisor. The PC boots normally, but the Chainfucker is in control of the system by opening ports outside linux control.
      It's just a theory though, I am no expert.

    • @bl8de3
      @bl8de3 8 หลายเดือนก่อน +4

      Yea. I even look ten times if someone adds a png or font to some kind of front end environment. Never not be paranoid.

    • @jbutler8585
      @jbutler8585 8 หลายเดือนก่อน

      @@bl8de3 TBH with how easy it is to Steghide data inside an image file, that's not unfounded to suspect malicious info could be hiding anywhere.

  • @s8r4
    @s8r4 8 หลายเดือนก่อน +190

    With every passing day, templeOS is starting to look more and more reasonable

    • @rusi6219
      @rusi6219 8 หลายเดือนก่อน

      Boundless switch

    • @jakedrake99
      @jakedrake99 8 หลายเดือนก่อน +3

      You’ll pass up the BSDs?

    • @ChrisWijtmans
      @ChrisWijtmans 8 หลายเดือนก่อน +3

      yes there is a lot to learn about templeOS, for example auditing the code that runs is easy and fast to do. no compiling.

    • @CEOofGameDev
      @CEOofGameDev 8 หลายเดือนก่อน +5

      they said that terry was mad, but who is going crazy is society...

    • @rusi6219
      @rusi6219 8 หลายเดือนก่อน +1

      @@CEOofGameDev society was always crazy you're just accelerating at a slower rate than the people around you that's why you can actually see it

  • @deep_space_dave
    @deep_space_dave 8 หลายเดือนก่อน +34

    I think I just got a good reason why I stopped using a rolling release! Thanks for this info!

    • @Cobalt985
      @Cobalt985 8 หลายเดือนก่อน +16

      Arch was not affected.

    • @toddtroll2220
      @toddtroll2220 8 หลายเดือนก่อน

      @@Cobalt985That's not the whole truth. Arch was affected but OpenSSH on Arch was not affected. Quote from Arch news: "Arch does not directly link openssh to liblzma, and thus this attack vector is not possible". But in xz package history you can clearly see that affected upstream versions 5.6.0 and 5.6.1 has been released by Arch.
      Still and forever user of Arch 😍

  • @vnc.t
    @vnc.t 8 หลายเดือนก่อน +36

    unpopular opinion: workflow 1 pulls the repository, builds everything and then tests everything, workflow 2 pulls the repo without using workflow 1's output, builds and publishes, that way malicious code must be in the build commands or toolchain where it will be noticed, tests just test and cannot affect the release

    • @CFSworks
      @CFSworks 8 หลายเดือนก่อน +16

      The malicious code is in the build commands, added to a random .m4 file in the release tarball (not Git) only. The payload was just hidden among the test files since that's a non-suspicious location to put a large binary blob.

    • @bigpickles
      @bigpickles 8 หลายเดือนก่อน +1

      Just read this as I squeezed a gripping fart. Contemplating this binary

    • @1495978707
      @1495978707 8 หลายเดือนก่อน +4

      ​@@CFSworksYou can still implement segregation of testing and building. Which to me as a simple physicist seems like the natural thing to do if you care about security. Same reason the manhattan project was broken up, as well as any other modern project with high security

    • @CFSworks
      @CFSworks 8 หลายเดือนก่อน

      @@1495978707 For sure, but I think that's done already. The tests are primarily run by the XZ project's developers and the CI system when each change is made. The backdoor is only targeting RPM+DEB package builders, and usually those buildsystems don't run the tests.

    • @CFSworks
      @CFSworks 8 หลายเดือนก่อน

      @@1495978707 Absolutely, but I think that's been done already. The RPM/Debian buildsystems don't typically run the test suites.

  • @JM-is1vf
    @JM-is1vf 8 หลายเดือนก่อน +27

    When was the backdoor mechanism committed?

    • @Rozenmorte
      @Rozenmorte 8 หลายเดือนก่อน +18

      Looks like it was added in Feb 24, 2024 for version 5.6.0 and 5.6.1 of xz

    • @felixjohnson3874
      @felixjohnson3874 8 หลายเดือนก่อน +27

      ​@@Rozenmorteso it lasted barely a month. Is it tasteless to make an infant mortality joke about backdoor malware?
      Eh, eitherway just remember that next month someone tries to say open source is less secure when even the pretty bloody advanced backdoors like this don't even last long enough to be picked up by a single release cycle.

    • @dragoscosma84
      @dragoscosma84 8 หลายเดือนก่อน

      ​@@felixjohnson3874 nah he found it by luck

    • @asgacc8789
      @asgacc8789 8 หลายเดือนก่อน

      ​@@felixjohnson3874 moving forward I feel like we need more eyes looking at the skeletons, we kinda got lucky "just a guy" stumbled upon this one

  • @filker0
    @filker0 8 หลายเดือนก่อน +24

    I found a keylogger in some custom keyboards back around 1990. Some of the extended function keys were producing the wrong scan-code combinations. While working out what was going on, I found it.
    It was a bit of a rabbit-hole.

    • @ricardf1857
      @ricardf1857 8 หลายเดือนก่อน +2

      Wow that sounds very interesting, could you give us more details?

    • @AllAmericanGuyExpert
      @AllAmericanGuyExpert 8 หลายเดือนก่อน

      That sounds implausible. When keyboards weren't even USB yet, and the Internet didn't have a WWW yet, I don't see how a keylogger would do much good on a peripheral unless it was combined with a sneakernet.

    • @Dpx008Music
      @Dpx008Music 8 หลายเดือนก่อน +3

      @@AllAmericanGuyExpert The methods of data retrieval for keyloggers in pre-USB and pre-WWW era would differ from those today, but the fundamental utility of capturing keystrokes covertly does not. If such keyboards existed, they would be used much differently, focusing more on targeted data collection from specific individuals or systems, while relying on physical access for data retrieval.
      I understand the skepticism but the absence of the WWW doesn't necessarily render the concept of hardware keylogging implausible!

    • @tzm1843
      @tzm1843 8 หลายเดือนก่อน

      @@AllAmericanGuyExpert There was Novell etc.

  • @adjbutler
    @adjbutler 8 หลายเดือนก่อน +4

    It appears that this backdoor fails to run properly on NixOS due to the non-standard FHS and the dynamic linker by default does not work as expected compared to Vanilla Linux. But steps are being taken in nixpkgs to downgrade xz anyway.
    Malware targeting vanilla linux has a hard time working on NixOS without careful forethought.
    I know that I am talking about security through obscurity, but sometimes obscurity just works.
    Also, in NixOS liblzma in not used as a dependency for sshd or openssh.

  • @Animaniac-vd5st
    @Animaniac-vd5st 8 หลายเดือนก่อน +1

    Thanks for putting this all into language that normal IT guys can understand without needing to have much experience with Linux library management and deep security knowledge.

  • @virtualinfinity6280
    @virtualinfinity6280 8 หลายเดือนก่อน +29

    Some more additions: As far as investigation of this problem went to this point, this backdoor was introduced around the 5.6.0 release. You can run "xz --version" to check, what version liblzma you run. If it is pre-5.6.0 you _may_ be safe (pending final analysis). But if you are on >= 5.6.0, you got the backdoor. At this point, it is recommended to downgrade to a version prior to 5.6.0. Regard this post as an immediate stop-gap measure for your system, not a proper security fix of the issue.

    • @iamweave
      @iamweave 8 หลายเดือนก่อน +4

      A relief if true. Ubuntu 22.04 is 5.2.5

    • @haifutter4166
      @haifutter4166 8 หลายเดือนก่อน +3

      Allready checked. Fck
      I use Manjaro. Maybe should return to something like Mint.

    • @JosephStory
      @JosephStory 8 หลายเดือนก่อน

      @@haifutter4166 Manjaro already has an update for the affected packages.
      Download of lib32-xz (5.6.1-2) finished
      Download of xz (5.6.1-2) finished

    • @NameUserOf
      @NameUserOf 8 หลายเดือนก่อน +1

      Manjaro received a fix. Also, do you use it on the server? Regular desktops aren't effected as tehre's a little reason to use ssh server there and with rsa key.

    • @flamethrower883
      @flamethrower883 8 หลายเดือนก่อน +1

      I read that you shouldn't run `xz --version` or `xz -V`, use brew command instead

  • @cusematt23
    @cusematt23 8 หลายเดือนก่อน +7

    im gonna be honest ... i am subscribed to like 200 youtube channels. i am starting to get the most excited when you have an upload. super interesting stuff, and some of it above my head. but i have a feeling little things like this are going to be of outsized importance in the years to come.

  • @puceno
    @puceno 8 หลายเดือนก่อน +346

    This doesn't seem like coincidence anymore

    • @rabbitcreative
      @rabbitcreative 8 หลายเดือนก่อน +49

      Never was.

    • @philadams9254
      @philadams9254 8 หลายเดือนก่อน +127

      What? Why would it be? A backdoor is intentional.

    • @terminalvelocity4858
      @terminalvelocity4858 8 หลายเดือนก่อน +50

      There is no coincidence, just malicious people out here. Nothing new. Thankfully, there are plenty that aren't and will counter it, every time.

    • @daniel29263
      @daniel29263 8 หลายเดือนก่อน

      ​@@terminalvelocity4858not just people, countries too.

    • @kmcat
      @kmcat 8 หลายเดือนก่อน +4

      I've fellt like the heart beat / bleed was delibret in opensll

  • @spookycode
    @spookycode 8 หลายเดือนก่อน +14

    Holy shit, get ghidra we are going for a wild ride

  • @dannywidjaya7943
    @dannywidjaya7943 8 หลายเดือนก่อน +20

    This is the equivalent of bystanders watching comic book heroes and villains fight it out in the sky

  • @Real-Name..Maqavoy
    @Real-Name..Maqavoy 8 หลายเดือนก่อน +2

    Found a similar issue (in our Librarys) from 2017, in the City, Which is owned by the State.
    Doesn't help when the Admins are using out of Date Software to *Secure* those Computers.

  • @eljuano28
    @eljuano28 8 หลายเดือนก่อน +26

    Great! Now it's back to smoke signals!

    • @9a3eedi
      @9a3eedi 8 หลายเดือนก่อน +3

      Ehh but smoke signals aren't exactly encrypted

    • @jittertn
      @jittertn 8 หลายเดือนก่อน

      They can be ​@@9a3eedi

    • @asgacc8789
      @asgacc8789 8 หลายเดือนก่อน

      @@9a3eedi blackmailed by little timmy next door

  • @doce3609
    @doce3609 8 หลายเดือนก่อน +40

    Damn what is happening this week.

  • @aminebahlaouane3604
    @aminebahlaouane3604 8 หลายเดือนก่อน +90

    shit's going down this week

    • @terminalvelocity4858
      @terminalvelocity4858 8 หลายเดือนก่อน

      Like...?

    • @imacmill
      @imacmill 8 หลายเดือนก่อน

      @@terminalvelocity4858 And Subscribe.

    • @aminebahlaouane3604
      @aminebahlaouane3604 8 หลายเดือนก่อน +1

      bug in apple computers, linux exploit

  • @xQcWilliam
    @xQcWilliam 8 หลายเดือนก่อน

    Thanks for breaking down this issue so fast! I was made aware of it but wanted to understand how it was done and had a hard time doing that just reading the sources.

  • @H1kari_1
    @H1kari_1 8 หลายเดือนก่อน +12

    How can a public library get a commit of a non-readable binary file approved???

    • @joost00719
      @joost00719 8 หลายเดือนก่อน +16

      If I understood it right, it was a random compressed file which was used in a unit test to "test" some stuff. But instead it executes some code, which changes the build process (unit tests are usually ran during the build process), which then changes the linker, which hooks into a call to a third party library, which then can grant access to the malicious user.
      It wasn't even clear that it was a non-readable binary file. At first glance it just looked like a compressed file for a specific unit test which tests uncompressing.

    • @H1kari_1
      @H1kari_1 8 หลายเดือนก่อน

      Sneaky.@@joost00719

    • @luigicorciulo8190
      @luigicorciulo8190 8 หลายเดือนก่อน +3

      Do you know that Linux is full of binary blobs and there was a project called Linux libre that tries to exclude all the binary blobs from linux

  • @felixjohnson3874
    @felixjohnson3874 8 หลายเดือนก่อน +31

    Question : when was this backdoor added originally?
    It'd be quite interesting to compare how long advanced backdoors like this exist in Open Source compared to known backdoors added to closed source software
    Edit : couldn't find a date skimming through the video, but XZ 5.6 was released in February from a quick search, so at least about a month

    • @alanmckinnon6791
      @alanmckinnon6791 8 หลายเดือนก่อน +5

      Presumably the lzma project is in git so you could clone it and find the precise commits. Then you would know the dudes handle too

    • @Spiker985Studios
      @Spiker985Studios 8 หลายเดือนก่อน +8

      Correct, and the "binary tests" that were added weren't even used
      It's quite ingenious

    • @daniel29263
      @daniel29263 8 หลายเดือนก่อน +12

      About 3 weeks ago the relevant xz file was updated.

    • @Bramble20322
      @Bramble20322 8 หลายเดือนก่อน +1

      Dont worry, the backdoors you should worry about wont even be found, only honeypots like this to keep the open source community smug, overconfident and blind to the real threats.

    • @felixjohnson3874
      @felixjohnson3874 8 หลายเดือนก่อน

      @@Bramble20322 Grandpa, take your meds. The nurses said they'd press charges if you have another episode.

  • @SuperBotcreator
    @SuperBotcreator 8 หลายเดือนก่อน +18

    So this dude is listed as a developer officially for the project: "The current project members are Lasse Collin and Jia Tan. Jia became a co-maintainer for the XZ projects in 2022." Unless he's got some info on how his account got compromised, everything in the history of mankind he's ever touched needs to be reverted and his name plastered on security blogs to the top of google results.
    In either case, an obvious laugh to OSS maintainers in general.

    • @luizfernandes1149
      @luizfernandes1149 8 หลายเดือนก่อน +4

      It is hard to believe that his account got compromised, the oldest commit related to this incident is the one that introduced the compiled binaries, and it comes from January 23, it is a bit hard to believe that him, a co-maintainer did not notice strange commits coming from his account during the last 2 months.

  • @sigstackfault
    @sigstackfault 8 หลายเดือนก่อน +8

    Good thing i haven't updated any of my VMs in years

  • @TRex-fu7bt
    @TRex-fu7bt 8 หลายเดือนก่อน

    I saw a few videos about something called xz pop up on my feed and I thought, I gotta get a proper rundown from Low Level Learning. Thanks for being a go-to

  • @jurajmihaly9315
    @jurajmihaly9315 8 หลายเดือนก่อน +7

    Marvelous exploits, I should learn to exploit testing.

  • @deez_narts
    @deez_narts 8 หลายเดือนก่อน +1

    Exceptional videos and shorts, man. Keep up the great work!

  • @uuu12343
    @uuu12343 8 หลายเดือนก่อน +30

    Disguising the backdoor as a files in the tests directory is...pretty new and kinda scarily big brain wtf
    Obfuscation by Obscurity

    • @Sypaka
      @Sypaka 8 หลายเดือนก่อน

      After this, you can be sure, people will verify all tests now, to see if that lib was "infected" the same way. Expect some funny updates the next week.

  • @LHCB6
    @LHCB6 8 หลายเดือนก่อน

    Saw the article and wasn't sure if I should click on a video about an article I read, and security issues that I've already taken care of, but this is a really great informative video. Thanks for making it!

  • @pulgamecanica
    @pulgamecanica 8 หลายเดือนก่อน +32

    OpenSSH doesnt use directly this library, this video is a bit misleading... but other than that, nice vid

    • @Kirillissimus
      @Kirillissimus 8 หลายเดือนก่อน +6

      Yes, it is that damn SystemD crap breaking subsystem isolation and hoping for the best again. But the core problem of patch reviews in open software projects and maintainer responsibility remains valid regardless.

  • @lefteryx
    @lefteryx 8 หลายเดือนก่อน +1

    andres freund is tbf not "just a guy", he's a microsoft principal software engineer and one of the main contributors to postgresql

  • @fe911s
    @fe911s 8 หลายเดือนก่อน +4

    Great stuff , i want to see your full break down when you upload it .

  • @philippgampe8120
    @philippgampe8120 8 หลายเดือนก่อน +1

    That's a crazy backdoor. Wouldn't surprise me to see some other popular packages compromised in a similar way.

  • @terminalvelocity4858
    @terminalvelocity4858 8 หลายเดือนก่อน +7

    Doesn't affect most distros (Debian, etc) unless on testing branches or rolling release. Arch Linux has already released an updated patch to repos. Sad to see this happened, but glad it was found and fixed before damage was done.

  • @Rasterizing
    @Rasterizing 8 หลายเดือนก่อน

    Awesome report! The best way I was able to explain this to a non technical person (and I'm barely able to follow this myself) was as follows;
    Imagine the index at the back of a book - the entry you want is actually on page 5 but the index has been changed to say page 11. That would be obvious, because the index page isn't in the same font, etc - so you'd be instantly suspicious. Imagine if someone then hacked the printer to print "11" instead of "5" - you'd never know and would go there instead of the "real" destination.
    I realise this is a massive simplification and makes it a lot more trivial (whilst also not being exactly correct) but it's the best anaolgy I could draw.

  • @shirro5
    @shirro5 8 หลายเดือนก่อน +6

    There is going to be a very long conversation about a lot of issues raised by this incident. Retirement and transition of FOSS project leadership, trust of upstream build systems and lots more. I don't have answers to any of that but I do have a simple request for distro maintainers. Can we not take a well engineered, mission critical bit of code from the security focussed OpenBSD folks and then add a bunch of extra library dependencies that most people don't need, want or expect. sshd is the last bit of software I expect maintainers to be adding random patches and additional library dependencies. ldd on arch lists 12 less dynamic libraries than on my debian stable systems but for my purposes the arch package lacks nothing. I think distro users need to push back on this stuff and not demand their pet feature be added to core libs.

  • @croissantwrenchn
    @croissantwrenchn 8 หลายเดือนก่อน +2

    My man, you’re the best. Love your videos

  • @heyjoeway
    @heyjoeway 8 หลายเดือนก่อน +13

    Shit like this is why I stopped exposing any ports aside from Wireguard to the internet.

    • @fwfy_
      @fwfy_ 8 หลายเดือนก่อน +15

      BREAKING NEWS: new wireguard RCE exploit found

    • @Phroggster
      @Phroggster 8 หลายเดือนก่อน

      @@fwfy_ Sigh... Stuff like this really needs a /s on the internet, but thanks for making me look/hyperventilate. There are 15 CVE's mentioning Wireguard, three from the year 2024, all of which are tagged as vulns in Cilium specifically.

    • @StupidusMaximusTheFirst
      @StupidusMaximusTheFirst 8 หลายเดือนก่อน

      @@fwfy_ 🤣

    • @Interpause
      @Interpause 8 หลายเดือนก่อน

      you made sure to disable ipv6 right?

    • @AD34534
      @AD34534 8 หลายเดือนก่อน +1

      What about malware that's already in your network? There could be a lot more that we just haven't found yet.

  • @sharkinahat
    @sharkinahat 8 หลายเดือนก่อน +2

    I'm kind of impressed how they managed to sneak that in.

  • @troyhamilton1889
    @troyhamilton1889 8 หลายเดือนก่อน +12

    I’ve been writing some ssh code in C using their header files and stuff. I have always said that OpenSSh is wayy too big. It is over 130,000 lines of c code. For reference Amazon was able to make a ssh library that’s as good as openssh and it was only 9000 lines. Let this be a lesson that bloated software is hard to manage and that complexity allows hackers to take advantage of it.

    • @dj_chateau
      @dj_chateau 8 หลายเดือนก่อน +6

      It wasn't an SSH library though, so your point isn't really relevant here. Complexity does create security challenges, but this wasn't complexity that contributed to this issue.

    • @gaborm4767
      @gaborm4767 8 หลายเดือนก่อน

      I am against the use of shared libraries, I think nowadays they cause more problems than benefits.

  • @sec2john
    @sec2john 8 หลายเดือนก่อน +1

    Crystal clear explanation thank you!

  • @nanolith
    @nanolith 8 หลายเดือนก่อน +32

    The title is misleading. The back door was not in anything relating to OpenSSH upstream. Distros patch the OpenSSH server code to include support for notifying systemd of logins. This is not built into OpenSSH, as the OpenSSH devs would never include such a silly feature. This is Linux distros shoehorning insecure systemd nonsense into everything, and yet again, it has burned users.

    • @jedi10101
      @jedi10101 8 หลายเดือนก่อน

      how to learn/understand all these stuff?

    • @nou712
      @nou712 8 หลายเดือนก่อน

      It's not just a systemd issue, it's an issue for PAM as well which doesn't just affect linux either.

    • @nanolith
      @nanolith 8 หลายเดือนก่อน

      @@nou712 openssh should be built without PAM support if you intend to use it on a remote accessible machine. Linux and FreeBSD PAM are both hot messes.

    • @OliNorwell
      @OliNorwell 8 หลายเดือนก่อน +1

      Yeah, it's a great video but that title makes me sad, it's misleading to get more clicks, I mean I know he wants to grow the channel and all, but throwing SSH under a bus to get them?... The OpenSSH project did nothing wrong here.

  • @michaeldeloatch7461
    @michaeldeloatch7461 8 หลายเดือนก่อน

    Just found your channel thanks YT algo. Your presentation is excellent.

  • @Jarikraider
    @Jarikraider 8 หลายเดือนก่อน +3

    *Claims to not be a security researcher or a reverse engineer*
    *Proceeds to research security and discover an open source backdoor exploit*

    • @Mercurio-Morat-Goes-Bughunting
      @Mercurio-Morat-Goes-Bughunting 8 หลายเดือนก่อน

      And therein lies the magic of open source; the opposite of security by obscurity.

  • @jrkorman
    @jrkorman 8 หลายเดือนก่อน

    And that friends was why in our shop, build files had to be as transparent as possible. They did get checked over to make sure they did ONLY what was needed and NO tricks, etc.
    Now the fun part was FOS libraries.

    • @absurdengineering
      @absurdengineering 8 หลายเดือนก่อน +1

      Where I work, all oss dependencies are built with cmake. If they only use autotools, we port the build to cmake. It’s crazy how much cruft is there for irrelevant platforms of the past. Truth is, most gnu code that’s “autotools only” does just fine with rather simple cmake scripts that anyone can understand.

    • @acters124
      @acters124 8 หลายเดือนก่อน

      The weird thing is that the malicious stuff is only found in the dev's hosted archive file of the source code, it is not found on github or other locations. Very odd.

  • @test40323
    @test40323 8 หลายเดือนก่อน +4

    Huh, who is allowed to change the make file? Is that traceable?

  • @FedtTony
    @FedtTony 8 หลายเดือนก่อน

    What a clever way to inject a backdoor! Everyone is focused on reviewing the source code, nobody expects the build process itself to be altered

  • @sarundayo
    @sarundayo 8 หลายเดือนก่อน +15

    Who TF commited that?! Let's start there >_>

    • @aeebeecee3737
      @aeebeecee3737 8 หลายเดือนก่อน +3

      Committed by Mr. N$4

    • @DDracee
      @DDracee 8 หลายเดือนก่อน

      one of the main maintainers lol

  • @tecsmith_info
    @tecsmith_info 8 หลายเดือนก่อน

    This kind of stuff is more popular than you think. Thank you for covering this.

  • @ZeroUm_
    @ZeroUm_ 8 หลายเดือนก่อน +4

    The build process is too open if the code being built can affect it that way.

  • @paulfloyd9258
    @paulfloyd9258 8 หลายเดือนก่อน

    Valgrind isn't just about leak detection. The main tool, memcheck, also validates that memory is initialized when it is read and that reads and writes are to valid memory. Both of these are usually far more serious than leaks. In this case it detected an invalid write below the stack pointer.

  • @JEffinger
    @JEffinger 8 หลายเดือนก่อน +34

    Some nation state is pissed

    • @meepk633
      @meepk633 8 หลายเดือนก่อน

      Ohh "nation state". This guy is sophisticated.

    • @deletevil
      @deletevil 8 หลายเดือนก่อน

      kremlin or ccp?

  • @manoelBneto
    @manoelBneto 8 หลายเดือนก่อน

    The fact this was caught by sheer chance makes me wonder how many more backdoors using similar injection techniques are out there in the wild.

  • @Krmpfpks
    @Krmpfpks 8 หลายเดือนก่อน +12

    If I read it right this ‚just a guy‘ actually was a maintainer of the fedora linux distribution and the malicious actor managed to become the maintainer of the XZ library by posing as an active developer for years.
    The ‚just a guy‘ did valgrind memory tests to verify if the library is safe to include and it failed. The malicious actor tried to convince everyone the valgrind errors are GCC bugs to hide this, and ‚just a guy‘ even helped fixing the memory errors before finding out the true nature of the patch.
    Edit: I can no longer find the mailing list post that confirms the version above. The newer posts by Andres Freund (the ‚just a guy‘) tell the story a little bit differently. See my response below for sources

    • @luigicorciulo8190
      @luigicorciulo8190 8 หลายเดือนก่อน

      How do you know this

    • @JustLennyBenny
      @JustLennyBenny 8 หลายเดือนก่อน

      @@luigicorciulo8190 He writes "see my responses below for sources" but I see nothing so consider this bs.

  • @jdotseven
    @jdotseven 8 หลายเดือนก่อน

    Amazing find. Not to mention the genius of the back door itself!

  • @DMSBrian24
    @DMSBrian24 8 หลายเดือนก่อน +29

    Source and software distribution platforms should run hash checks on releases to avoid those situations, it's insane that this is not the case yet.

    • @DMSBrian24
      @DMSBrian24 8 หลายเดือนก่อน +2

      If it's too expensive to implement globally, it could instead be done once a project becomes big enough, it could earn some sort of verified status.

    • @RandomGeometryDashStuff
      @RandomGeometryDashStuff 8 หลายเดือนก่อน

      what is hash checks?

    • @vvstwo
      @vvstwo 8 หลายเดือนก่อน +5

      this is happening at compile time tho most people are going feel completely safe after compiling the lib themselves.

    • @benfowler1134
      @benfowler1134 8 หลายเดือนก่อน +9

      Part of the exploit was rolled into released source tarballs on the Github website (but not the Git repo IIRC). If upstream package maintainers used the provided tarballs to build the binary packages and then made the signatures available, they would be 'trusted' regardless. In which case, you're screwed. There are multiple issues the perpetrator exploited to plant this backdoor; there's going to be a ton of lessons learned.

  • @jamesbond_007
    @jamesbond_007 8 หลายเดือนก่อน +1

    Do I recall correctly that early versions of the C compiler had a cooperating backdoor with the login program that would install a hard to spot backdoor into the login program? This was in the early, early UNIX systems. The hack you mentioned in the video is incredible. Dynamic linked libraries can have all sorts of interceptors & interposers placed on the, by simply having suitable LD_LIBRARY_PATH setting that refer to your version of an API. I used this trick onetime to create an interposition on malloc & free that would add additional header and footer memory to malloced storage, so the memory had sentinels at both ends to detect overwrites, and I think I also had it link all of the malloc'd memory together in a list so an analyzer could scan through the malloc list and find any areas that had been overrun, and compute some statistics about how much memory was allocated, maybe how long it was allocated for, etc. Probably around 1992 or so.

    • @05Matz
      @05Matz 8 หลายเดือนก่อน

      At least theoretically the self-propagating backdoor you're describing was described way back when in an academic paper I remember being told about, "Reflections on Trusting Trust", or something like that, and it's been a sort of... matter of conspiratorial debate whether or not it actually happened, at least that's how I remember it.

    • @jamesbond_007
      @jamesbond_007 8 หลายเดือนก่อน

      @@05MatzYes, it was from long ago -- maybe the 80s when the description came out. The source code for Version 6 and Version 7 UNIX is available (including for "cc'"), so if the hack was still there , it should be observable. It was one of the original UNIX guys, I think Brian, but maybe Dennis who talked about it and provided some details about how it worked. [as I recall]

  • @serhii-ratz
    @serhii-ratz 8 หลายเดือนก่อน +34

    The complexity of projects raises, but this backside is peace of art… 🖼️

  • @tankman1301
    @tankman1301 8 หลายเดือนก่อน +1

    nice video with good incentives to further dive into

  • @davejoseph5615
    @davejoseph5615 8 หลายเดือนก่อน +3

    I thought "open source" meant that obfuscated code was never accepted or allowed?

    • @bl8de3
      @bl8de3 8 หลายเดือนก่อน

      it was a compressed test archive. it was, in fact, very well hidden.

  • @transientaardvark6231
    @transientaardvark6231 8 หลายเดือนก่อน

    I love your past-participle of "to dig"

  • @noodlish
    @noodlish 8 หลายเดือนก่อน +13

    Description is from previous vid still. Also I hear Valgrind isn't pronounced -grind (like coffee) but -grinned (like the Cheshire cat).

    • @LowLevelTV
      @LowLevelTV  8 หลายเดือนก่อน +7

      fixed thank you

    • @kricku
      @kricku 8 หลายเดือนก่อน +3

      Yeah, it means gate

  • @gotj
    @gotj 8 หลายเดือนก่อน

    Thanks for this great video, 0% fat and straight to the point. Cheers.

  • @wonderstruck.
    @wonderstruck. 8 หลายเดือนก่อน +20

    Andres Freund isn’t a security researcher, but he’s not “some guy” either 😅 He’s a principal engineer at Microsoft and a PostgreSQL developer.
    (Not a developer who uses Postgres, but a developer who builds and maintains Postgres itself.)

    • @xlxl7866
      @xlxl7866 8 หลายเดือนก่อน +2

      He is definitely an experienced developer

  • @9a3eedi
    @9a3eedi 8 หลายเดือนก่อน

    This feels like it'll be such a shock to open source projects... I would expect that other project would start scrutinizing their code and contributions more deeply in the next couple of weeks and we'll be hearing more surprises

  • @oglothenerd
    @oglothenerd 8 หลายเดือนก่อน +3

    At least it was caught and is open source.

  • @edelzocker8169
    @edelzocker8169 8 หลายเดือนก่อน +2

    Thanks! Installed 5.6.1-2...

  • @geerliglecluse5297
    @geerliglecluse5297 8 หลายเดือนก่อน +4

    Seems almost as if someone or a group of very knowledgeable engineers is deliberately targeting open source software. And it's not just technical, this also targets the way changes to open source software are usually review, validated and merged into the code base as well. The whole change process for open source software seems to be in need of an upgrade.

    • @joshallen128
      @joshallen128 8 หลายเดือนก่อน

      But it's open that's the problem and benefit

    • @05Matz
      @05Matz 8 หลายเดือนก่อน +4

      Oh, certainly. I'd bet that this was the work of some kind of three-letter-agency* from some country somewhere. Probably better than even money that it was a nominally 'friendly' country to you and/or me (in my case Canada, so the whole usual suspects of Five Eyes/etc.)! Not so friendly when they show their hand at their persistent efforts to make everyone in the world unsafe in the name of their stupid squabbling power games.
      *Name may not be exactly three characters, but is probably an acronym.

    • @Bramble20322
      @Bramble20322 8 หลายเดือนก่อน +1

      @@05Matz More like 6 letter agency tbh. They do be trying to spy on everyone all the time.

    • @bobmorgan1575
      @bobmorgan1575 8 หลายเดือนก่อน

      @@05Matz This is something that I wouldn't put past MS as a way to eliminate platform competition.

    • @05Matz
      @05Matz 8 หลายเดือนก่อน

      @@bobmorgan1575 I don't think they'd do something with this much risk of getting caught, just to besmirch the security of a competing platform. I think whoever did this planned to USE the backdoor, not for it to only exist as FUD.

  • @osvaldogalvez4776
    @osvaldogalvez4776 8 หลายเดือนก่อน

    Dude I love you for these videos. Thank you

  • @CU.SpaceCowboy
    @CU.SpaceCowboy 8 หลายเดือนก่อน +4

    is it only when you enable compression or just in general. if it compromises the binary in the build process does that mean it only effects ssh thats recompiled from source code or does it effect every binary that comes with every system? (sorry im unfamiliar with how that kinda stuff works)

    • @chri-k
      @chri-k 8 หลายเดือนก่อน +1

      every binary that was compiled after the change was introduced is compromised, which may or may not include that one in your system depending on how old this is

    • @mcarpenter2917
      @mcarpenter2917 8 หลายเดือนก่อน +1

      If the source code was compromised then binaries where also comprised. But the issues was only created 3 weeks ago so a lot of systems will never have received the update, also you would have to be a ssh user to be affected, because ssh is not something that is enabled by default. This is my take anyways from what little info was in the video.