@@SenulaDilasara It might happen because missing drivers of the distro you try to install. Windows, have same issue. Try to install Zorin OS 17.1 It recognized the NVME drive with no problem and the installation was fast. It's an awesome distro with the feel of Windows and it's bombarded with list of software if you want to install, and you can still use the terminal if you want to.
I love to hate on Microsoft but Crowdstrike literally crashed RedHat servers less than a month ago and Debian servers back in April. I think the common denominator here is Crowdstrike.
@@lazymass if I give my house keys to a third party and they cause a problem, I bear some responsibility because I allowed them access. try reading about, "trust, but verify" & "swiss cheese model"
For thoae who don't know, an airline in the U.S. (Southwest) wasn't affected by this Crowdstrike BSOD issue because it was running in top of _Windows 3.1 & 9.5._ Yes, seriously!
Yeah, I know "if it ain't broke don't fix it", but come the f**k on, there's a point where maintaining those obsolete systems probably costs more than swtiching to something else or upgrading...
Windows 9.5? Surely you mean '95 ;-) ? Well, a lot of banks still run code from the 60's (COBOL), but granted, the (virtual) machines running that code are not directly connected to the internet. They must take the mantra "If it aint't broken, don't fix it" very seriously, but they are forgetting the boatloads of known vulnerabilities on those ancient graphical DOS-Shells.
@@LarixusSnydes Yes, I did mean Windows 95 (but if I edit my comment, the heart from Nick goes away), but yeah, there is a point where you HAVE to move on because it's too risky if something DOES happen and it breaks.
The CrowdStrike problem really doesn't have anything to do with Windows. CS has also made a bunch of Debian servers kernel panic in april. This is genuinely a problem with CS not testing their updates.
@@nicejungle It's not crowdstrike messing up the kernel, it's a kernel module, going back a kernel version still loads the kernel module and crashes the system. They also didn't magically just brick windows, it was a messed up kernel driver that caused this.
@@nou712 No. Recently I completely messed up my kernel with a bad update of nvidia drivers. I've rebooted, select previous kernel, problem fixed. This kind of trick is impossible on windows because it's not a professional OS
You can boot windows to a mode where drivers aren't loaded. It's not exactly the same but it's pretty close. Really, I will choose linux every time I can but from a user point of view they aren't so different..
I saw a commenter on Ars Technica say that the big issue is Habsburg Syndrome. Like Habsburg dynasty, the modern tech industry is so inbred that a small issue can quickly become catastrophic.
Lolol I agree mostly. There is the extra element to outsourcing security or services to other companies because you can't afford to keep them in house. That sort of element makes it entirely dependent on the service provider. Maybe this means we need more service providers? Or maybe regulation of different type and size of service providers? Or we enforce standardized process and open interfacing for different types of service proders and services (think on the ANSI/NIST/ISO level)
TBH, the fact that this didn't happen on Linux & MacOS doesn't mean it couldn't. Falcon sensor on Linux is a kernel module as well. The real problem is any OS allowing this sort of nonsense...
well, it's an antivirus so why is it weird that it needs ring 0 driver to work correctly? if you could easily disable the driver from loading at boot, that would be massive security hole, isn't? I think IT guys shouldn't allow critical level software like antivirus to update itself without testing in safe environment first.
@@Space_Rat1 ig crowdstrike pushed the update stating that they fixed some critical vulnerability, so damned if you do damned if you don't. EDIT: the update wasn't critical. my bad. IT teams of the enterprises are to be blamed too!
It wasn't that much of a monopoly, we are talking about something like a 20% market share, but 20% of all computers using windows globally is a big big number
web (internet) is ruled by linux, enterprises by microsoft. those enterprises dug their own grave by continuously over the years keeping developing softwares (eggs) only for windows (basket) and created a SPOF
@@0x4C3DD the answer is simple for this case... support. Most companies don't have linux experts to deal with problems locally, they simply call support when an issue occurs.
My poor help desk is having to work with remote workers to enter Bitlocker recovery codes and walk them through getting into safe mode to delete the files that are causing the blue screen.
Yeah, feels insane even to have billboards paying a windows license. Run a Linux box, if it’s just to automatically play a freaking video or display a slideshow…
That is the key word. Server Most business use Windows for their "terminals" aka what they employees use to do their work because most people isnt familiar with a linux distro And tbh with the quintillion flavors and 300000 million different application packaging formats i dont blame then You would have to train people with using the terminals and that is cash money companies are not willing to fill
In the corporate world, all the workstation pcs including most POS, billboards, ATM, etc. run some version of Windows. We are talking about licensing schemes that are aimed at companies so it's not as expensive as one might expect. It is a lot of money but companies prefer to pay and offload a lot of responsibility to vendors. Also businesses always prefer to have a supplier that can take responsibility from eventual failures, they don't like open source OS because the company itself would have to step up. For a long time businesses would prefer HP-UX and Solaris to Linux or OpenBSD. That only changed in the last ten years.
the point is why is windows all around the world crashing because kids today cannot program like their fathers and grand fathers could they are spoiled babies plain and simple🤪🤪 and switching to Linux would change nothing because the Linux devs are not going to fix the crashes when they will not give support now with existing linux let alone handle the complexity's of business software to boot I mean why do you think Linux has no native awesome games made on it while windows can run tons of bad ass games🤔
My main question is why the fuck are they not using staggered rollout? Like first roll it out to 0.01% of customers, see if crash rates spike or if you get a thousand angry phonecalls. If not, then roll it out to 5%, then 30%, then 50, 80 and 100% like every company I’ve worked at does this, even ones with like only 1000 customers
Honestly I think what SUSE could probably do to make this rebranding request go a whole lot smoother is offer an olive branch and assist openSUSE in developing their own recognizable branding. They don't _have_ to do it, and I'm not expecting nor demanding that they do, but it would be pretty cool if SUSE _did_ do that as a gesture that their request for openSUSE to rebrand was not made in bad faith and that they're willing to continue supporting the community distro in its new identity.
openSUSE community has already chosen a new logo and most likely we'll go with Tumbleweed, Leap and Aeon etc. as separate distribution brands. The project name is still in the open as far as I know although that won't really matter as its most likely only going to be a legal entity in the background.
They know that they have given permission for these updates. IT departments don't have as many people as they used to and are increasingly dependent on cloud and vendor direct updates, they don't have the resources to vet the updates and roll them out on their own.
It was explained to me, that this is the entire point of CrowdStrike. The business model is the outsourcing of security-vetting and "cutting-edge" live-monitoring. A lot of the fortune 500 companies dont have a good excuse, but smaller companies dont have the resources to vet updates to the degree necessary, so they pay CrowdStrike for it and CrowdStrike gets Kernel-access in order to do that job. Essentially, you can cross your fingers that your IT team, amidst all the daily nonsense they are dealing with, also pay 24/7 attention to zero days potentially affecting your company or you pay CrowdStrike to do that for you. Main issue is really that companies that could afford their own teams still choose to save money and thereby create an industry-wide, single point of failure. Imagine CrowdStrike got compromised by a malicious actor... CrowdStrike is really the main culprit here for not properly testing their updates before final release... Mastodon seems convinced that they just laid off a big part of their QA team...
@@tj2375 I work on a 4 person IT team for over 4,000 employees. We rely heavily on vendors and none of ours push OTA updates before they can be tested.
@@dolgorukysvyatoslav Windows still sucks and is using Windows Recall in the next update to spy on its users it's not just about Crowdstrike it's starting to suck in general the last good Windows will be Windows 10 I tried Windows 11 and the start menu is abomination and injected with ads why do you still defend them?
@@dolgorukysvyatoslav Also lucky for you I don't use crowdstrike on Linux thankfully it's only an option they seem to make Norton look innocent just now.
A lot of tech evangelists are using the Crowdstrike crashes to raise their flags. You'll hear that the crash wouldn't have happened if they used Linux instead of Windows, or Rust instead of C++, or whatever. They are all wrong. The problem happened because an antivirus company with shady QA practices was given remote kernel access to thousands of machines. Their crappy update system was fed garbage and entered an unforeseen state. That's the kind of situation that can easily brick a machine regardless of the safety nets your software has in place, because preventing malicious software from running is what the entire thing is intended to do. The solution is not Linux, or Rust, but not to give kernel access to shady companies.
Actually, the issue wouldn’t have happened on Linux anymore, because KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen
Windows also has such an API but a lot of security vendors decline to use it for unclear reasons. It was introduced in the Vista era and has been actively expanded and updated since. The best performing products typically use it.
I love Linux and I love the open source community but in the crowdstrike incident I think that you're totally right about it being a QA problem and bashing Windows without knowing the details is just Linux evangelists preaching for their own church . Bad security practices and bad QA testing can and has happened in the open source world and can be just as bad and destructive, on top of my head OpenSL's Heartblead in 2012 or Linux mint's compromised iso in 2016. Software is developed by humans and humans make mistakes, no software will ever be perfect. I'm against Microsoft for their practices but this crowdstrike incident is not a reason to parrot Linux as the ultimate solution for security and pristine software devoid of bugs, such software doesn't exist.
@@nicejungle Safe Mode is a thing on Windows. It was part of the guidance for reverting the breaking change manually with the Crowdstrike disaster in fact.
Don't twist things. The global crash happened on Microsoft OS, but is solely the fault of Crowdstrike - not Microsoft. This could happen to anybody that carelessly install 3rd party kernel components, being Microsoft, Linux or Mac.
Photoshop 2021 runs under WINE, Mattscreative has done videos on this. He also has one on Photoshop 2023, I think that required using a VM but it was still running the app outside the VM. He has one on Illustrator too.
The thing is, from what I hear, Safe Mode _did_ work to enact the workaround. Minimal drivers, and all that jazz. It should also be noted that this is a driver file. Somehow, there wound up being a long string of zeroes where there was supposed to be actual code. How did that even happen?
How wonderful it is - a program to protect against cyber attacks led to a complete collapse of the system. It’s logical - if your system doesn’t work, no one can hack it. In any case, the need to reboot to solve essentially any problem is something that has not changed since Windows 95. Just reboot it until it works! Even if you have to do it 15 times, that's absolutely normal, that's how computers are supposed to work! It’s funny that this problem has practically not affected Russia, since the services addressing the problem do not work in Russia due to sanctions.
If you could get into Safe Mode then you could remove the corrupted file from Crowdstrike. However, if you use Bitlocker then you need a recovery key and that might be dependent on your IT team being able to access that on a Windows machine which might have the same issue. Additionally I ran into needing local admin credentials for my work laptop to finally delete the file.
Regarding the system level access. You need that level as a protection. How do you know there is a rootkit unless you are already sitting where it wants to go? I do agree though that there was no "insecure" fall back boot mechanism is pretty poor.
We should learn from what happened here and work towards shielding Linux in such a way that third party software for safety is not nescessary. Considering that Linux has been growing in the desktop space, we should take safety more seriously.
You can't shield anything against companies like Crowdstrike and people who hire them. The problem is this: they need to be subjected to criminal prosecution. They keep doing this and no punishment. Crowdstrike CEO did it years ago when he was at McAfee. It's similar to how Unity CEO made a career of destroying companies / tanking stocks. (Did it before in EA) This is all very deep corporate gameplays. The only shield is... community distros that are free from corporations.
It's not the low level access crowstrike sw works or that this update crash windows during boot. There are plently of similar driver/firmware/sw that act in similar way on linux and macos, (nvidia drivers ?) that also can make the os crash or halt on boot. The problem relies in who pushed the update non testing it fully (many w machine with different hw versions and software crashed, not specific ones), and also IT department not managing those updates in test machines or postpone its installation for 1 or 2 days.
@@TheLinuxEXP Yeah, I hope some of their work will make it into development for the whole community. I honestly, think that building up Linux systems for enterprises takes a little more work, but it's more secure and stable in the long run. I taught English to a programmer at ROSCOSMOS (the Russian Space Agency), he said they use a custom designed Debian derivative called AstraLinux. Beyond, this there's a good chance that computer labs in public schools will now have Ubuntu or something in them all soon. I could see a good knock on effect for the whole Linux user base from this massive move away from Windows.
Basically OpenSUSE is Fedora, but I worry that changing branding loses so much steam, this already not too popular distro might lose more popularity if not done right, and in a recognizable way. I don't know how much SUSE actually cares about Fedora, like if they see it as a way to get early bug reports, contributions etc, but I would think a small marketing push about the inevitable name change would show their support well, while accomplishing their goals. It really should be a thoughtful switch.
Suse is even a bit older than Redhat. It is diappointing that they finally almost completely follow the redhat trail, even whith crap software like network manager and gnome. But that happens when you go from a small german geek company to a big international (=U.S.?) one.
As long as the rebrand keeps the chameleon named Geeko, then I’m fine with the rebrand! If it does not have a chameleon, I’d still be alright with it, but something would feel different every time I boot into openSUSE.
I don't mind OpenSuse being rebranded. It combines being very stable with fairly up to date packages and I have very little to complain about ever since I switched to it from Debian. I have an HDR monitor and did not want to wait 2 years for Debian to update Plasma. Having the YaST applications next to the KDE defaults is a little weird at times but some of the provided tools are great, once you understand how they work.
@@loz9324 SuSE dropped the Chameleon. Maybe the (former) OpenSuSE team can still reclaim the beloved mascot. One is sitting on my screen upstairs in my office right now ;-).
You didn't need a bootable USB drive to fix the crash, just start the computer and turn it off as soon as it started going into windows. This kicked off the special menu on next boot that let you select Safemode boot on next boot where you could then delete the bad file(s). But you had to do this for EVERY machine. Also if you had bitlocker enabled with an encrypted drive you also needed to get your recovery key (hope you had that handy). The biggest issue here is that too many companies cut down on their IT staff and outsourced what some of them did to a global company that then could screw over a LOT of machines in one go.
The cause of the recent CrowdStrike for Linux issues was actually a bug in eBPF, not the CrowdStrike software. Apparently eBPF gives kernel access from software running in user-space. The issues only occurred on certain kernel versions. A simple reboot to the previous kernel worked around the problem as did a change in the configuration of CrowdStrike to let it run in kernel-mode.
Interesting. I would like to point out that loading eBPF programs is a privileged operation and needs root on newer distros. You can however load cBPF programs from unprivileged processes through SECCOMP.
หลายเดือนก่อน +7
Windows has nothing to do with CrowdStrike shipping a poorly tested update with a faulty driver. Any operating system can crash if a program with privileges in the kernel space does things it shouldn't. Also, the same company crashed Debian and Rocky Linux. So yes, it did happened to Linux too.
KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen
Windows has suffered from viruses and other security threats for years due to users deciding not to apply updates as provided. Slowly and surely, automatic updates have become the norm. Software updates roll out without local release cycle tests on a smaller number of systems. Are you surprised?
and the reluctance to update was partially caused by undesired behavior after updates, like UI elements being rearranged etc. seems like a lot of practices need to be re-evaluated and change for change's sake needs to go unless you opt into insider builds.
Linux is also not free from CrowdStrike problems Debian 12 + crowdstrike caused kernel panics in April RedHat + crowdstrike caused kernel panics in June
Thank you! This isn't a windows issue. This is an issue on the nature of EDRs themselves. Windows just happens to be the endpoint of this particular disaster
This CorwdStrike Issue had a broad impact - we were on a late evening call reviewing a proposal that was due in the AM and 1/3 of the people on the call started reporting blue screens of death and we almost lost our latest version of the proposal. The next morning - half of our company was impacted.
Microsoft doesn't understand that Windows is used by users. They are more focused toward their AI and Azure Cloud computing services than the actual usability of their own operating system. I'm also tired of companies forcing us to log-in into accounts for work that do not necessarily need network connectivity. I had to install Windows on a VM to create ISOs for separate machines for testing or development purposes, and the fact that I need to work around that pesky OOBE screen to get to a desktop with a local user is astonishing. This should be illegal. Fortunately I've been a Linux user for the past 8 years, and full-time for the past 4 years now. I'm happy to see that year after year, Linux improves in usability and ease of use while still keeping power of choice to users. So I'm glad that more people realise that they have better choices than Windows, at least for those who aren't locked in with "services" and software that is exclusive to certain operating systems.
I haven't used windows on my machines for the past 9 years and I recently had to troubleshoot a windows 11 pc for family... Boy it's so bad, the UI is bad and you get ads in places where you wouldn't expect any... Like why would anyone put ads for Microsoft 355 in the frigging settings of all places???
@@dudmanjohn You're probably one of the rare ones who think this. A lot of people think otherwise, unless if they use Apple products, and even that is becoming less true with the amount of customisability that was brought to iOS recently, for example.
The difference between Windows and Linux, when windows crashes (whatever the reason why) it affects the whole world and when Linux crashes (whatever the reason why) nobody hears about it.
I don’t see the crowdstrike update breaking windows as a “windows” problem per-se. It installs a kernel driver, doesn’t matter if that’s Mac, Windows or Linux. If you’re a system admin and you install some 3rd parties kernel driver into your servers, you are opening yourself up to this. Mac and Linux customers likely got lucky with this one. Wire a cloud service directly into your kernel, expect it to break at some point. Treat it like any other update and at least have a canary process.
Also your claim that this hasn't happened on linux is just wrong, it has happened before with RHEL systems, the software has its own self-updating kernel module on linux as well.
And I thought that running essential infrastructure applications on Windows platforms is a class A felony in the US. If it isn't, it should be. And it appears in the interest of clarity and transparency "MicroSoft 365" will be renamed to "MS 364 and 1/3".
5:00 Sad to disappoint you on that. A Crowdstrike update lead to kernel panic on Debian in April. So it has the same problems on any platform, and just this time it was only Windows, but was Linux before.
I'm slowly migrating to Linux. I'm starting by installing Fedora on my 2nd gaming PC; the one connected to the TV. I'll migrate my main PC to one distro or another, once I get more comfortable with the differences/nuances. It's tough because I've been using Windows for almost 30 years, so I know where everything is and how everything works, so I can easily resolve technical issues. It's comfortable. Whereas Linux was more of a hobby and a small part of my degree, so I'm still very much a newb.
XDR/antivirus software is one of the few exceptions where kernel level access is justified. Because XDR needs to be able to see and be able intervene with activities on any level of the operating system.
This outage just demonstrates that a lot of critical infrastructure runs not only on Linux, but also on Windows. It's never a good idea to put all your eggs in one basket, therefore, it is good that Windows is present at the enterprise level. And don't forget, this was not a Windows issue, but a 3rd party product.
You're just wrong about the Crowdstrike stuff. As an example, calling it 'a single point of failure' is just a fundamental misunderstanding of what that means. By your definition, the opersting system itself would be considered a 'single point of failure.' The problem is a lack of testing. Updates to critical systems should be done in a test environment before it is ever pushed to production. That doesn't happen for a few reasons but, most often, because management won't pay for it.
I don't think he's wrong about anything here, I think it's just a different way to analyze the situation. He's analyzing it from a diversity point of view. In agriculture, for example, the practice of monoculture is risky because a disease can wipe out an entire season of crops, starving the dependant community, so polyculture is far more common. Another example can be seen within communities: communities with more genetic divergence tend to be more resistant to infections. Yes, the operating system is also a "single point of failure" in the same sense. When a day-0 kernel exploit for Windows or Linux is discovered, it's a big deal that quickly becomes a race to patch everything up with security fixes before the situation can get out of hand. Remember the xz utils backdoor a couple months ago? That was also a major "single point of failure" threat. The exploit was discovered quite quickly, which is why there was so little damage, but if it hadn't been discovered, nearly all Linux machines would have become vulnerable, which could have ended disastrously, depending on the attacker's motivations. Of course, as you said, better testing of updates could have also alleviated this problem. This is the ideal solution, but it is a flawed approach because it is simply impossible to test for 100% of situations. Now I don't know that I can really defend CrowdStrike in this situation where they somehow messed up nearly all Windows machines, but that doesn't invalidate other forms of damage control.
Linux and mac OS didn't crash because they don't receive the .SYS file, which was completely zeroed out. Had their driver files been zeroed out instead, they most certainly would have crashed. Dereferencing a null pointer that deep in the kernel in unguarded code will *definitely* crash any OS.
In the eternal cat and mouse game of black hat vs security businesses, I'm not surprised that Crowdstrike has pushed their code to lower and lower levels of the OS. Security software needs to be at a level lower than whatever malware is out there. But that comes with an increasing responsibility for the stability of the users' systems, so they need to design so that this kind of crash isn't possible with an update. They didn't, so they need to make architectural changes so make sure this can't happen again. Furthermore, this is why updates are tested, beta tested, rolled out as a staggered release, etc. Pushing updates to everybody all at once isn't a great move when your organization is used by such a large number of systems including critical systems for many businesses. I'm not very impressed by Crowdstrike.
Idk what you mean that „no program should have that level of access“ with the whole crowdstrike situation They have to have that access bcs the are intercepting calls made by all processes Unprivileged processes can’t do that so they have to run with higher privileges than the process being monitored Also if there was a way to crash my edr I would prefer I takes the whole system with it instead of allowing the malware to do what it wants I’d rather be offline than infected
I will not accept that there is no way for such a program to monitor my system without having the ability to make it unbootable if something goes wrong. That’s horrible design.
@@TheLinuxEXPThat is just how it works, even on Linux it is like this. Take SECCOMP for example: - It can be used by unprivileged programs - It loads a program INTO the kernel to intercept syscalls - It runs a verifier to check if it terminates (aka formal verification) It can still break the kernel, but the verifier ensures that it is pretty good to run. So we need better verification and testing.
Well, guess it's slightly time to look for another distro that supports Plasma just as well and that is actually not requiring me to use their Wiki to get even the boot process done.
I would love to switch, but lack of support for a few peripherals that I need stops me. I can almost find workarounds for my software needs. I just wish HW vendors took Linux seriously. In meantime its Windows with WSL on my desktop and Linux on laptop.
My dad says the computers at his office were out for a short while, but the IT department got them up and running again within a couple of hours. But his personal laptop was fine, it doesn't use CrowdStrike. The laptop I'm typing this on is Ubuntu, so I was safe. And so was almost everyone at my university; like 90% of them own Macs, in my stats class there are zero- I repeat, ZERO- Windows laptops. 99% of the students in that class own MacBooks, and I'm the 1% who own a PC (that isn't even a Windows one)
The second biggest party in Greece asked of the government to move everything to Linux before the situation gets even worse. We were not even affected mostly. I know who I'm voting for next time! :)
I tried Red Hat and then Caldera way back in the day. Not so fun. I found Mandrake. It went to Mandriva and now Mageia. Wonderful. Easy. Since the 'recall' thing, I have made this my daily driver.
I know you hate Windows and is warranted whomever wrote your script about BSOD and Safe Mode needs to reread what Safe Mode does, the only issue was remoting into a server and having someone physically there to do the deletion. A Live USB is the last resort but Safe Mode was a solution and the easier of them if your drive was encrypted
Linux and macOS weren't crashed by CrowdStrike because they obviously don't use the Kernel driver for Windows that contained quite the beginner's fault. Also, at least the Linux version is said to be able to use eBPF instead of some shady and untrustworthy closed source Kernel driver. macOS doesn't allow for third party Kernel drivers alltogether. EDIT: they could have very easily broken their Linux Kernel Mode drivers too, and I kinda doubt any Linux distro has that many capabilities helping in such a circumstance. Also, not sure if the driver crashed the running system or only crashed every time someone tries to boot Windows afterwards.
This is such a display of incompetence, I expect Crowdstrike will soon go out of business. Probably there will be massive lawsuits for damages. This bug was, by all appearances, very easy to catch by Crowdstrike. Instead they continued to push it to customers for many hours. That is a massive display of incompetence. That is even apart from the question why this software was tied so closely to the Windows kernel.
Hi Nick, why don't you mention how firefox 128 did introduce a "privacy preserving ad feature" that looks very like an integrated data collection system for ad based companies more or less as google's "privacy sandbox" does?
I use the Mainline Linux Kernel ppa to be able to update my Linux kernel in Kubuntu and the highest that it is currently showing for Linux kernels is 6.9.9, which I have had installed for some time now as my active kernel in use. Something that is weird is when I go to uninstall the older Linux kernels that were used by Cannonical to create the 24.04LTS, they will not uninstall, I can uninstall any other older kernels, just not the ones that Ubuntu installed with.
15:30 if I understood correctly, you can’t install anz adobe stuff because that needs integration old internet explorer, but if you installed it some other way than you can start the programs
@@cameronbosch1213 because Adobe has a hard dependency for the actual Internet Explorer, which wine doesn’t have the rights to use, I think but as I said, I only saw that in a video of someone and sadly I don’t remember whose it was
when i was a kid, i thought that hospitals, airlines and major infrastructure had their OWN closed source OS, and that they didn't use windows, mac or linux at all, but today i see that if they were to actually try to make their own OSes, it'd be a major waste of time and resources
I wish to correct something. The Crowdstrike tool update seems to have been a distribution corruption. I am not sure a similar thing couldn't happen on Linux. according to Low Level Learning this was a null pointer dereference clobbering registers. What protections does the Linux Kernel have against a dimilar issue. You would still have to touch a linux box to boot into a recovery environment if a proprietary driver caused a kernel panic durring bootup.
This only affected people using this software. I've been in IT 25 years and this is the first time ever hearing about Crowdstrike. This is the kind of thing that makes me dubious how such a "mistake" escaped from the developers department, to QC departments, to beta testers and so on before ever becoming a release candidate? In this day and age how can one not consider a nefarious act? Yes if you have been in IT as long as I have or longer you must question. Windows has been an unfinished product since it was launched close to 40 years ago (hence the constant updates/security fixes etc.) Also none of the systems I am around daily, or sites I visit (like Amazon) had any issues. So this issue probably was a limited issue around the US and not some world wide massive outage where people scream like they are on fire. Large corporations are mostly affected because IT IGNORES the warning signs (seen it in real time!)
My laptop brightness hotkeys will freeze until it wake from suspend. I had 6.1 kernel version. With 6.5 the issue was resolved. I hope 6.10 will help many.
Is it probably because the kernel was only null for Windows, but not for other OSes? Like, the developer who compiled the driver didn't notice the null error since the file size is not showing any error, but it was compiled correctly for Linux and macOS.
Even if they switch to Linux someone can still push some "security driver" updates that crashes Linux just the same. It's heavily misleading to say it's Windows instability this time. It really is just a faulty driver. Of course this usually doesn't happen on Linux because most of the drivers are reviewed and pushed to the kernel source. But any third party driver can crash the kernel just as easy as Windows, if not easier.
Big difference is that drivers aren't forcefully pushed to Linux machines. Any decent linux admin will only pull updates, making this sort of bug localized and much more avoidable.
@@mytech6779 I'd argue given it's Crowdstrike specifically. What makes you think that automatic updates aren't also turned on in Linux by default as well? This time it's just the Windows driver that is broken. This can totally happen on Linux as well.
Yeah, shoddy programming is at fault here. There are probably some cost-cutting protocols at play here. Understaffed and under-qualified programmers no doubt. What the penny pinching accountants at CrowdStrike managed to save will probably hurt them in the long run.
@@shamringo7438 I didn't say anything about defaults, I said forcefully pushed. Though on that tangent, many distros do not have auto update as a default.
As a person who has been teaching OS for more then 10 years, I would give my 2 cents about CloudStrike issue in Windows - People blame microsoft for this and they say it is their fault because of the bad design of the OS. The reality is that this problem could happen to any modern general purpose OS. In modern CPU you have a mechanisum called isolation rings. For example in x86/x64 you have 4 - from 0 to 3. Linux, FreeBSD, Windows and MacOS uses only two of them 0 and 3. When the boot loader loads the OS kernel it gives the control to it and it runs the core subroutines and most of the drivers/modules in this ring 0. This ring is highly privileged and everything which runs there has full acess to full CPU instructuon set and full memory. When the OS pass the control to the user space processes it sets the CPU in ring 3 and then pass the control. Then the user process has the ilusion that it has full access to the hardware but thats not the case. When a user space process needs to something with the OS (e.g. open file) it use a core library of the OS. Behind the call of the function to open the file there an interupt instruction that triggers a mechanisum in the CPU to wake a specific subroutine of the OS. This way the OS will take the control and do whatever is requied (e.g. validate permissions, use the fs code to read the blocks from the storage, loads the data in memory and etc.) Using this isolation rings you can ensure that user space process cannot interfere or break another process. However, when the code runs in ring 0, it can break everything intensionally or unintentionally. This works the same for all modern general purpose OS. So it doesnt metter if you have a buggy driver in linux, windows or macos, all of them can fail. In this specific issue, CrowdStrike have a bug in their driver which is there probably for a long time. They realese a new definition file that unlocks this bug. Since the driver is kernel mode (ring 0) it demages the whole OS. Why they need ring 0 acesss - they hook their code in diffrent calls in subroutines in order to collect audit data and later analyze it for different patters. And this works the same in all OS. Furthermore, CrowdStrike did few similar problems with Linux and MacOS but with less impact. Also M$ ans MacOS enforce only singed drivers to be loaded in kernel mode exactly because of this impact. So we have a well known trusted 3rd party company which is using signed driver with their certificate and they just messed it up. I like the idea to blame Microsoft but lets not blame them for something that is not thier fault. P.S. there some projects like eBPF which exists in Linux and even Windows that can help by doing static code analysis verification before opening the door to the kernel space. However, they have some limitations and it is up to the vendor to develop thier programs...
AFAIK, Crowdstrike on Linux now uses eBPF, meaning that this issue wouldn't have happened on Linux, at least on systems with relatively recent kernels. The issue isn't necessarily that MS didn't prevent this from happening, because bugs happen, it's that the OS has no mechanism in place to recover from this. Why isn't Windows rebooting automatically in safe mode after 2 or 3 failed boot attempts? Why didn't it offer to boot while disabling the Crowdstrike kernel driver, as an option? It feels like either a cascade failure of various recovery systems that didn't trigger, or an oversight in terms of OS design, where they just don't have a solid recovery option. Booting 15 times in a row into a BSOD shouldn't happen :)
There is also what MacOS does: system extensions API. The kernel essentially asks for a process in userspace if something should be allowed (it reminds me of a micro kernel design).
Head to squarespace.com/thelinuxexperiment to save 10% off your first purchase of a website or domain using code thelinuxexperiment
That's why blocking Windows updates is important and Zorin OS is a fine replacement against Win11 Copilot AI and Recall spyware.
i want to install linux on my laptop but my ssd is not showed even by gparted. Do you have anyway to fix that?
@@SenulaDilasara
It might happen because missing drivers of the distro you try to install. Windows, have same issue.
Try to install Zorin OS 17.1
It recognized the NVME drive with no problem and the installation was fast.
It's an awesome distro with the feel of Windows and it's bombarded with list of software if you want to install, and you can still use the terminal if you want to.
I love to hate on Microsoft but Crowdstrike literally crashed RedHat servers less than a month ago and Debian servers back in April. I think the common denominator here is Crowdstrike.
They crashed but they were not unbooteable after the crash
Strikes (no pun intended) me, that someone needs a change of career, enforced if necessary.
@@lien-san3347so what? Everyone blaming windows here is completely out of their mind... Linux feet lickers
True and real, that company is a pile of crap.
@@lazymass if I give my house keys to a third party and they cause a problem, I bear some responsibility because I allowed them access.
try reading about, "trust, but verify" & "swiss cheese model"
Opensuse should change their name to chameleon
That would be the master reply back!
That sounds like a neat name for the openSUSE rebrand!
There is already software used for stock keeping called Chameleon, don't know if that would interfere
or LizardedLinux
@@skelebro9999 And have a green lizard 🦎
For thoae who don't know, an airline in the U.S. (Southwest) wasn't affected by this Crowdstrike BSOD issue because it was running in top of _Windows 3.1 & 9.5._ Yes, seriously!
Hahah wow
Yeah, I know "if it ain't broke don't fix it", but come the f**k on, there's a point where maintaining those obsolete systems probably costs more than swtiching to something else or upgrading...
Windows 9.5? Surely you mean '95 ;-) ? Well, a lot of banks still run code from the 60's (COBOL), but granted, the (virtual) machines running that code are not directly connected to the internet. They must take the mantra "If it aint't broken, don't fix it" very seriously, but they are forgetting the boatloads of known vulnerabilities on those ancient graphical DOS-Shells.
@@LarixusSnydes Yes, I did mean Windows 95 (but if I edit my comment, the heart from Nick goes away), but yeah, there is a point where you HAVE to move on because it's too risky if something DOES happen and it breaks.
Dos saves the world
The CrowdStrike problem really doesn't have anything to do with Windows. CS has also made a bunch of Debian servers kernel panic in april. This is genuinely a problem with CS not testing their updates.
Except in linux, you can go back to the previous kernel after a bad update. Down time : one reboot
@@nicejungle It's not crowdstrike messing up the kernel, it's a kernel module, going back a kernel version still loads the kernel module and crashes the system. They also didn't magically just brick windows, it was a messed up kernel driver that caused this.
@@nou712
No.
Recently I completely messed up my kernel with a bad update of nvidia drivers.
I've rebooted, select previous kernel, problem fixed.
This kind of trick is impossible on windows because it's not a professional OS
@@nou712 iirc, most systems will just build the kernel module dkms/akmods for the latest kernel installed so older kernels use the older version
You can boot windows to a mode where drivers aren't loaded. It's not exactly the same but it's pretty close. Really, I will choose linux every time I can but from a user point of view they aren't so different..
I saw a commenter on Ars Technica say that the big issue is Habsburg Syndrome. Like Habsburg dynasty, the modern tech industry is so inbred that a small issue can quickly become catastrophic.
Haha that’s a great way to put it
Lolol I agree mostly. There is the extra element to outsourcing security or services to other companies because you can't afford to keep them in house. That sort of element makes it entirely dependent on the service provider. Maybe this means we need more service providers? Or maybe regulation of different type and size of service providers? Or we enforce standardized process and open interfacing for different types of service proders and services (think on the ANSI/NIST/ISO level)
Most fastest "super computers" run Linux.
"Don't put all of your eggs in one basket" still rings true.
@@markluxton3402 true but irrelevant.
TBH, the fact that this didn't happen on Linux & MacOS doesn't mean it couldn't. Falcon sensor on Linux is a kernel module as well. The real problem is any OS allowing this sort of nonsense...
That’s what I said :)
rpm-ostree for the win !!
well, it's an antivirus so why is it weird that it needs ring 0 driver to work correctly? if you could easily disable the driver from loading at boot, that would be massive security hole, isn't? I think IT guys shouldn't allow critical level software like antivirus to update itself without testing in safe environment first.
@@Space_Rat1 ig crowdstrike pushed the update stating that they fixed some critical vulnerability, so damned if you do damned if you don't.
EDIT: the update wasn't critical. my bad. IT teams of the enterprises are to be blamed too!
@@0x4C3DD Which was probably true, but just not tested enough what the impact was before mass-deploying said update.
This is another example of problems in Monopoly. Single company providing security solution to lot of major companies
It wasn't that much of a monopoly, we are talking about something like a 20% market share, but 20% of all computers using windows globally is a big big number
web (internet) is ruled by linux, enterprises by microsoft.
those enterprises dug their own grave by continuously over the years keeping developing softwares (eggs) only for windows (basket) and created a SPOF
@@0x4C3DD the answer is simple for this case... support. Most companies don't have linux experts to deal with problems locally, they simply call support when an issue occurs.
My poor help desk is having to work with remote workers to enter Bitlocker recovery codes and walk them through getting into safe mode to delete the files that are causing the blue screen.
@@0x4C3DD but linux is not one coherent product, dude.
Crowdstrike protected windows computers from *all* the threats. 100% effectiveness.
Even better, it protected planes from crashing by preventing them from flying. Amazing protective software, I'll tell you that!
@@Linux_ASMR I'm glad you get my drift :)
What shocked me is critical businesses, hospitals, airlines use Windows. I thought they use Linux. Linux dominates in the server world.
Desktops aren’t Linux.
I'm more shocked they don't have fucking backups.
Yeah, feels insane even to have billboards paying a windows license. Run a Linux box, if it’s just to automatically play a freaking video or display a slideshow…
That is the key word. Server
Most business use Windows for their "terminals" aka what they employees use to do their work because most people isnt familiar with a linux distro
And tbh with the quintillion flavors and 300000 million different application packaging formats i dont blame then
You would have to train people with using the terminals and that is cash money companies are not willing to fill
In the corporate world, all the workstation pcs including most POS, billboards, ATM, etc. run some version of Windows. We are talking about licensing schemes that are aimed at companies so it's not as expensive as one might expect. It is a lot of money but companies prefer to pay and offload a lot of responsibility to vendors. Also businesses always prefer to have a supplier that can take responsibility from eventual failures, they don't like open source OS because the company itself would have to step up. For a long time businesses would prefer HP-UX and Solaris to Linux or OpenBSD. That only changed in the last ten years.
the point is, you can't get a malware if your computer isn't working 😂
the point is why is windows all around the world crashing because kids today cannot program like their fathers and grand fathers could they are spoiled babies plain and simple🤪🤪 and switching to Linux would change nothing because the Linux devs are not going to fix the crashes when they will not give support now with existing linux let alone handle the complexity's of business software to boot I mean why do you think Linux has no native awesome games made on it while windows can run tons of bad ass games🤔
My main question is why the fuck are they not using staggered rollout? Like first roll it out to 0.01% of customers, see if crash rates spike or if you get a thousand angry phonecalls. If not, then roll it out to 5%, then 30%, then 50, 80 and 100% like every company I’ve worked at does this, even ones with like only 1000 customers
Good point. Besides, I suspect they did not test this properly in-house before deployment.
Id like for Opensuse to keep the chameleon, but changing the name is perfectly good with me
What's so bad about the Chameleon anyway? I don't get that part at all. It's been the logo for so many years now, and I love it.
@@JammetIt's SUSE's trademark, they have the final say who uses it.
Maybe if they changed it to a green cute dragon that kinda looks chameleon-ish so you know it’s related but it’s not the same thing.
@WilburJaywright or a newt or an axolotl, something similarly cute
Or salamander@@xXx_Regulus_xXx
the rebranding is about brand identity not association
SEO for SUSE/openSUSE is a nightmare
True
I would laugh really hard if the code that caused the outage was generated by an LLM. That would be chefs kiss.
What is a LLM?
@@eeaotly Large Language Model, in other words an A.I. language model. Some companies have used them to generate code cheaply.
@@WaterShowsProd Thank you!
Does not seem clear, but there's a driver file full of zero causing the infinite boot crash.
Seems like to me there was a build or deployment failure.
@@nicejunglebummer. I guess we'll have to wait for the first "A.I." generated crash.
Honestly I think what SUSE could probably do to make this rebranding request go a whole lot smoother is offer an olive branch and assist openSUSE in developing their own recognizable branding. They don't _have_ to do it, and I'm not expecting nor demanding that they do, but it would be pretty cool if SUSE _did_ do that as a gesture that their request for openSUSE to rebrand was not made in bad faith and that they're willing to continue supporting the community distro in its new identity.
openSUSE community has already chosen a new logo and most likely we'll go with Tumbleweed, Leap and Aeon etc. as separate distribution brands. The project name is still in the open as far as I know although that won't really matter as its most likely only going to be a legal entity in the background.
The fact that with Windows a vendor can push an OTA update like that without the IT teams knowing about it is an issue.
Yep
They know that they have given permission for these updates. IT departments don't have as many people as they used to and are increasingly dependent on cloud and vendor direct updates, they don't have the resources to vet the updates and roll them out on their own.
Exactly, this is the real issue.
It was explained to me, that this is the entire point of CrowdStrike. The business model is the outsourcing of security-vetting and "cutting-edge" live-monitoring. A lot of the fortune 500 companies dont have a good excuse, but smaller companies dont have the resources to vet updates to the degree necessary, so they pay CrowdStrike for it and CrowdStrike gets Kernel-access in order to do that job. Essentially, you can cross your fingers that your IT team, amidst all the daily nonsense they are dealing with, also pay 24/7 attention to zero days potentially affecting your company or you pay CrowdStrike to do that for you. Main issue is really that companies that could afford their own teams still choose to save money and thereby create an industry-wide, single point of failure. Imagine CrowdStrike got compromised by a malicious actor... CrowdStrike is really the main culprit here for not properly testing their updates before final release... Mastodon seems convinced that they just laid off a big part of their QA team...
@@tj2375 I work on a 4 person IT team for over 4,000 employees. We rely heavily on vendors and none of ours push OTA updates before they can be tested.
"fuck you, i use linux" ahh moment
@@yoClohrine Crowdstrike is also used on Linux, and it was their fault, not Windows
@@dolgorukysvyatoslav Crowdstrike did also not fail on Linux. So the issue seems to be that there's no reliable testing on Windows.
Also the main cause of issues was becuase azure did not test updates before pushing to production
@@dolgorukysvyatoslav Windows still sucks and is using Windows Recall in the next update to spy on its users it's not just about Crowdstrike it's starting to suck in general the last good Windows will be Windows 10 I tried Windows 11 and the start menu is abomination and injected with ads why do you still defend them?
@@dolgorukysvyatoslav Also lucky for you I don't use crowdstrike on Linux thankfully it's only an option they seem to make Norton look innocent just now.
Crowdstrike did strike the whole crowd.
AFAIK the creative cloud applications mostly *do* run on wine, but the installer does not.
Interesting!
A lot of tech evangelists are using the Crowdstrike crashes to raise their flags. You'll hear that the crash wouldn't have happened if they used Linux instead of Windows, or Rust instead of C++, or whatever. They are all wrong. The problem happened because an antivirus company with shady QA practices was given remote kernel access to thousands of machines. Their crappy update system was fed garbage and entered an unforeseen state. That's the kind of situation that can easily brick a machine regardless of the safety nets your software has in place, because preventing malicious software from running is what the entire thing is intended to do. The solution is not Linux, or Rust, but not to give kernel access to shady companies.
Actually, the issue wouldn’t have happened on Linux anymore, because KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen
Windows also has such an API but a lot of security vendors decline to use it for unclear reasons. It was introduced in the Vista era and has been actively expanded and updated since. The best performing products typically use it.
Wrong.
On Linux, if you messed up your kernel in any way, you reboot and you select the previous kernel
On Windows, you're screwed
I love Linux and I love the open source community but in the crowdstrike incident I think that you're totally right about it being a QA problem and bashing Windows without knowing the details is just Linux evangelists preaching for their own church .
Bad security practices and bad QA testing can and has happened in the open source world and can be just as bad and destructive, on top of my head OpenSL's Heartblead in 2012 or Linux mint's compromised iso in 2016.
Software is developed by humans and humans make mistakes, no software will ever be perfect.
I'm against Microsoft for their practices but this crowdstrike incident is not a reason to parrot Linux as the ultimate solution for security and pristine software devoid of bugs, such software doesn't exist.
@@nicejungle Safe Mode is a thing on Windows. It was part of the guidance for reverting the breaking change manually with the Crowdstrike disaster in fact.
Don't twist things. The global crash happened on Microsoft OS, but is solely the fault of Crowdstrike - not Microsoft. This could happen to anybody that carelessly install 3rd party kernel components, being Microsoft, Linux or Mac.
Yeah but how can you trust anything Microsoft saids bill gates has lied a lot over and over the past years no one should trust what they say
Photoshop 2021 runs under WINE, Mattscreative has done videos on this. He also has one on Photoshop 2023, I think that required using a VM but it was still running the app outside the VM. He has one on Illustrator too.
"The best way to keep your computer safe, is to turn it off" 🤷♂
The thing is, from what I hear, Safe Mode _did_ work to enact the workaround. Minimal drivers, and all that jazz. It should also be noted that this is a driver file. Somehow, there wound up being a long string of zeroes where there was supposed to be actual code. How did that even happen?
How wonderful it is - a program to protect against cyber attacks led to a complete collapse of the system. It’s logical - if your system doesn’t work, no one can hack it.
In any case, the need to reboot to solve essentially any problem is something that has not changed since Windows 95. Just reboot it until it works! Even if you have to do it 15 times, that's absolutely normal, that's how computers are supposed to work!
It’s funny that this problem has practically not affected Russia, since the services addressing the problem do not work in Russia due to sanctions.
If you could get into Safe Mode then you could remove the corrupted file from Crowdstrike. However, if you use Bitlocker then you need a recovery key and that might be dependent on your IT team being able to access that on a Windows machine which might have the same issue. Additionally I ran into needing local admin credentials for my work laptop to finally delete the file.
I hope this outage will convince companies to finally switch to Linux for non-employee computers!
Regarding the system level access. You need that level as a protection. How do you know there is a rootkit unless you are already sitting where it wants to go? I do agree though that there was no "insecure" fall back boot mechanism is pretty poor.
We should learn from what happened here and work towards shielding Linux in such a way that third party software for safety is not nescessary. Considering that Linux has been growing in the desktop space, we should take safety more seriously.
if only I could "retweet" a youtube comment
See also the almost Flatpak virus that luckily got caught
You can't shield anything against companies like Crowdstrike and people who hire them.
The problem is this: they need to be subjected to criminal prosecution.
They keep doing this and no punishment.
Crowdstrike CEO did it years ago when he was at McAfee.
It's similar to how Unity CEO made a career of destroying companies / tanking stocks. (Did it before in EA)
This is all very deep corporate gameplays.
The only shield is... community distros that are free from corporations.
@@pyepye-io4vu Sorry but I do not get how this relates to what I said.
Everything should be run in a sandbox. Everything.
All these companies installing updates on production systems without testing them first???? Start printing the pink slips...
Are you suggesting that humans installed the crowdstrike update? Because that's not how the update worked. It was pushed from crowdstrike
It's not the low level access crowstrike sw works or that this update crash windows during boot.
There are plently of similar driver/firmware/sw that act in similar way on linux and macos, (nvidia drivers ?) that also can make the os crash or halt on boot.
The problem relies in who pushed the update non testing it fully (many w machine with different hw versions and software crashed, not specific ones), and also IT department not managing those updates in test machines or postpone its installation for 1 or 2 days.
Russian airports were reported to be functioning just fine yesterday. They recently switched over to Linux due to Microsoft withdrawing support there.
I guess the sanctions Russia was handed had at least some positive effects!
@@TheLinuxEXP Yeah, I hope some of their work will make it into development for the whole community. I honestly, think that building up Linux systems for enterprises takes a little more work, but it's more secure and stable in the long run. I taught English to a programmer at ROSCOSMOS (the Russian Space Agency), he said they use a custom designed Debian derivative called AstraLinux. Beyond, this there's a good chance that computer labs in public schools will now have Ubuntu or something in them all soon. I could see a good knock on effect for the whole Linux user base from this massive move away from Windows.
Yeah but they didn't have any flights anyway.
@@mytech6779BS
False. The reason they were not affected is not because they don't use Microsoft software but instead because they don't use Crowdstrike.
Basically OpenSUSE is Fedora, but I worry that changing branding loses so much steam, this already not too popular distro might lose more popularity if not done right, and in a recognizable way.
I don't know how much SUSE actually cares about Fedora, like if they see it as a way to get early bug reports, contributions etc, but I would think a small marketing push about the inevitable name change would show their support well, while accomplishing their goals.
It really should be a thoughtful switch.
Suse is even a bit older than Redhat. It is diappointing that they finally almost completely follow the redhat trail, even whith crap software like network manager and gnome. But that happens when you go from a small german geek company to a big international (=U.S.?) one.
As long as the rebrand keeps the chameleon named Geeko, then I’m fine with the rebrand! If it does not have a chameleon, I’d still be alright with it, but something would feel different every time I boot into openSUSE.
I don't mind OpenSuse being rebranded. It combines being very stable with fairly up to date packages and I have very little to complain about ever since I switched to it from Debian. I have an HDR monitor and did not want to wait 2 years for Debian to update Plasma. Having the YaST applications next to the KDE defaults is a little weird at times but some of the provided tools are great, once you understand how they work.
Losing my beloved chameleon is devastating 😢
@@loz9324 SuSE dropped the Chameleon. Maybe the (former) OpenSuSE team can still reclaim the beloved mascot. One is sitting on my screen upstairs in my office right now ;-).
You didn't need a bootable USB drive to fix the crash, just start the computer and turn it off as soon as it started going into windows. This kicked off the special menu on next boot that let you select Safemode boot on next boot where you could then delete the bad file(s). But you had to do this for EVERY machine. Also if you had bitlocker enabled with an encrypted drive you also needed to get your recovery key (hope you had that handy). The biggest issue here is that too many companies cut down on their IT staff and outsourced what some of them did to a global company that then could screw over a LOT of machines in one go.
The cause of the recent CrowdStrike for Linux issues was actually a bug in eBPF, not the CrowdStrike software. Apparently eBPF gives kernel access from software running in user-space. The issues only occurred on certain kernel versions. A simple reboot to the previous kernel worked around the problem as did a change in the configuration of CrowdStrike to let it run in kernel-mode.
Interesting. I would like to point out that loading eBPF programs is a privileged operation and needs root on newer distros.
You can however load cBPF programs from unprivileged processes through SECCOMP.
Windows has nothing to do with CrowdStrike shipping a poorly tested update with a faulty driver.
Any operating system can crash if a program with privileges in the kernel space does things it shouldn't.
Also, the same company crashed Debian and Rocky Linux. So yes, it did happened to Linux too.
KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen
Windows has suffered from viruses and other security threats for years due to users deciding not to apply updates as provided. Slowly and surely, automatic updates have become the norm. Software updates roll out without local release cycle tests on a smaller number of systems. Are you surprised?
and the reluctance to update was partially caused by undesired behavior after updates, like UI elements being rearranged etc. seems like a lot of practices need to be re-evaluated and change for change's sake needs to go unless you opt into insider builds.
crowdstrike software decided that their software will run on kernel level.
Time to upgrade to Linux!
Gnome REALLY needs to add HDR. It's the single feature that keeping me from using Gnome on my desktop computer.
Linux is also not free from CrowdStrike problems
Debian 12 + crowdstrike caused kernel panics in April
RedHat + crowdstrike caused kernel panics in June
Thank you!
This isn't a windows issue. This is an issue on the nature of EDRs themselves. Windows just happens to be the endpoint of this particular disaster
This CorwdStrike Issue had a broad impact - we were on a late evening call reviewing a proposal that was due in the AM and 1/3 of the people on the call started reporting blue screens of death and we almost lost our latest version of the proposal. The next morning - half of our company was impacted.
Microsoft doesn't understand that Windows is used by users. They are more focused toward their AI and Azure Cloud computing services than the actual usability of their own operating system.
I'm also tired of companies forcing us to log-in into accounts for work that do not necessarily need network connectivity. I had to install Windows on a VM to create ISOs for separate machines for testing or development purposes, and the fact that I need to work around that pesky OOBE screen to get to a desktop with a local user is astonishing. This should be illegal.
Fortunately I've been a Linux user for the past 8 years, and full-time for the past 4 years now. I'm happy to see that year after year, Linux improves in usability and ease of use while still keeping power of choice to users. So I'm glad that more people realise that they have better choices than Windows, at least for those who aren't locked in with "services" and software that is exclusive to certain operating systems.
I haven't used windows on my machines for the past 9 years and I recently had to troubleshoot a windows 11 pc for family... Boy it's so bad, the UI is bad and you get ads in places where you wouldn't expect any... Like why would anyone put ads for Microsoft 355 in the frigging settings of all places???
We don't want choice.
@@dudmanjohn You're probably one of the rare ones who think this.
A lot of people think otherwise, unless if they use Apple products, and even that is becoming less true with the amount of customisability that was brought to iOS recently, for example.
Live, Laugh, Linux. 🙃
I hope they rename it SusLinux.
That would be too Sus ;-).
The difference between Windows and Linux, when windows crashes (whatever the reason why) it affects the whole world and when Linux crashes (whatever the reason why) nobody hears about it.
I don’t see the crowdstrike update breaking windows as a “windows” problem per-se. It installs a kernel driver, doesn’t matter if that’s Mac, Windows or Linux. If you’re a system admin and you install some 3rd parties kernel driver into your servers, you are opening yourself up to this. Mac and Linux customers likely got lucky with this one.
Wire a cloud service directly into your kernel, expect it to break at some point. Treat it like any other update and at least have a canary process.
They pushed a corrupted windows kernel driver, it caused a nullptr dereference and then boom bsod.
Also your claim that this hasn't happened on linux is just wrong, it has happened before with RHEL systems, the software has its own self-updating kernel module on linux as well.
I didn’t say it hasn’t happened before, I said it didn’t happen for this specific time
@@eps-nx8zgbut on linux you can easily disable kernel modules
I mean, it is CrowdStrike. Crowd Strike.
Oh, my! It's genius!
And I thought that running essential infrastructure applications on Windows platforms is a class A felony in the US. If it isn't, it should be. And it appears in the interest of clarity and transparency "MicroSoft 365" will be renamed to "MS 364 and 1/3".
5:00 Sad to disappoint you on that. A Crowdstrike update lead to kernel panic on Debian in April. So it has the same problems on any platform, and just this time it was only Windows, but was Linux before.
Best advert for Linux as an OS yet!
I'm slowly migrating to Linux. I'm starting by installing Fedora on my 2nd gaming PC; the one connected to the TV. I'll migrate my main PC to one distro or another, once I get more comfortable with the differences/nuances. It's tough because I've been using Windows for almost 30 years, so I know where everything is and how everything works, so I can easily resolve technical issues. It's comfortable. Whereas Linux was more of a hobby and a small part of my degree, so I'm still very much a newb.
XDR/antivirus software is one of the few exceptions where kernel level access is justified.
Because XDR needs to be able to see and be able intervene with activities on any level of the operating system.
I think the key factor is that this has to have low level kernel access to actually do its job, and I believe the same is true for linux.
Huh Photoshop 2024 works in wine thats insane because i remember that i can't even run the installer
This outage just demonstrates that a lot of critical infrastructure runs not only on Linux, but also on Windows.
It's never a good idea to put all your eggs in one basket, therefore, it is good that Windows is present at the enterprise level. And don't forget, this was not a Windows issue, but a 3rd party product.
You're just wrong about the Crowdstrike stuff. As an example, calling it 'a single point of failure' is just a fundamental misunderstanding of what that means. By your definition, the opersting system itself would be considered a 'single point of failure.'
The problem is a lack of testing. Updates to critical systems should be done in a test environment before it is ever pushed to production. That doesn't happen for a few reasons but, most often, because management won't pay for it.
I don't think he's wrong about anything here, I think it's just a different way to analyze the situation. He's analyzing it from a diversity point of view. In agriculture, for example, the practice of monoculture is risky because a disease can wipe out an entire season of crops, starving the dependant community, so polyculture is far more common. Another example can be seen within communities: communities with more genetic divergence tend to be more resistant to infections.
Yes, the operating system is also a "single point of failure" in the same sense. When a day-0 kernel exploit for Windows or Linux is discovered, it's a big deal that quickly becomes a race to patch everything up with security fixes before the situation can get out of hand.
Remember the xz utils backdoor a couple months ago? That was also a major "single point of failure" threat. The exploit was discovered quite quickly, which is why there was so little damage, but if it hadn't been discovered, nearly all Linux machines would have become vulnerable, which could have ended disastrously, depending on the attacker's motivations.
Of course, as you said, better testing of updates could have also alleviated this problem. This is the ideal solution, but it is a flawed approach because it is simply impossible to test for 100% of situations. Now I don't know that I can really defend CrowdStrike in this situation where they somehow messed up nearly all Windows machines, but that doesn't invalidate other forms of damage control.
Linux and mac OS didn't crash because they don't receive the .SYS file, which was completely zeroed out. Had their driver files been zeroed out instead, they most certainly would have crashed. Dereferencing a null pointer that deep in the kernel in unguarded code will *definitely* crash any OS.
I have absolutely had thies party software nuke my linux desktops ability to boot properly, this isnt a "windows" issue alone.
In the eternal cat and mouse game of black hat vs security businesses, I'm not surprised that Crowdstrike has pushed their code to lower and lower levels of the OS. Security software needs to be at a level lower than whatever malware is out there. But that comes with an increasing responsibility for the stability of the users' systems, so they need to design so that this kind of crash isn't possible with an update. They didn't, so they need to make architectural changes so make sure this can't happen again. Furthermore, this is why updates are tested, beta tested, rolled out as a staggered release, etc. Pushing updates to everybody all at once isn't a great move when your organization is used by such a large number of systems including critical systems for many businesses. I'm not very impressed by Crowdstrike.
Is that your cat in the Tuxedo ad? Looks like my Max, who I miss, he passed away at age 13.
It's ridiculous how to many infrastructure stuff don't have automated fallback backup systems and just stay down if something goes wrong....
Idk what you mean that „no program should have that level of access“ with the whole crowdstrike situation
They have to have that access bcs the are intercepting calls made by all processes
Unprivileged processes can’t do that so they have to run with higher privileges than the process being monitored
Also if there was a way to crash my edr I would prefer I takes the whole system with it instead of allowing the malware to do what it wants
I’d rather be offline than infected
I will not accept that there is no way for such a program to monitor my system without having the ability to make it unbootable if something goes wrong.
That’s horrible design.
@@TheLinuxEXPThat is just how it works, even on Linux it is like this.
Take SECCOMP for example:
- It can be used by unprivileged programs
- It loads a program INTO the kernel to intercept syscalls
- It runs a verifier to check if it terminates (aka formal verification)
It can still break the kernel, but the verifier ensures that it is pretty good to run.
So we need better verification and testing.
@@TheLinuxEXPMicrokernels are probably what you're looking for. But the big OSes are all monolithic.
@@1.0 GNU HURD is a microkernel, I think...
Doesn't this prove Windows should not force bitlocker
Well, guess it's slightly time to look for another distro that supports Plasma just as well and that is actually not requiring me to use their Wiki to get even the boot process done.
Hey Nick thanks dude, last time I asked you to post those screenshot articles in dark mode and you did it.
6:44 this makes perfect sense. I thought SUSE and OpenSUSE were the same thing before today.
I would love to switch, but lack of support for a few peripherals that I need stops me. I can almost find workarounds for my software needs. I just wish HW vendors took Linux seriously.
In meantime its Windows with WSL on my desktop and Linux on laptop.
My dad says the computers at his office were out for a short while, but the IT department got them up and running again within a couple of hours. But his personal laptop was fine, it doesn't use CrowdStrike.
The laptop I'm typing this on is Ubuntu, so I was safe. And so was almost everyone at my university; like 90% of them own Macs, in my stats class there are zero- I repeat, ZERO- Windows laptops. 99% of the students in that class own MacBooks, and I'm the 1% who own a PC (that isn't even a Windows one)
The second biggest party in Greece asked of the government to move everything to Linux before the situation gets even worse. We were not even affected mostly. I know who I'm voting for next time! :)
There is a video on TH-cam that shows how to install photoshop on linux. You basically have to install on windows then copy it over...
That is a bunch of great Linux improvements. Love it!
IT had a hard day. Some are remote and live hundreds of miles away and had to drive hours away to be on site to fix some of this.
Ah, bless.
but which animal should they choose? I hope they stay green.
The SUSE thing is strange to me, like. Why now? It's reasonable, but why did they take so long to ask
I tried Red Hat and then Caldera way back in the day. Not so fun. I found Mandrake. It went to Mandriva and now Mageia. Wonderful. Easy. Since the 'recall' thing, I have made this my daily driver.
I know you hate Windows and is warranted whomever wrote your script about BSOD and Safe Mode needs to reread what Safe Mode does, the only issue was remoting into a server and having someone physically there to do the deletion. A Live USB is the last resort but Safe Mode was a solution and the easier of them if your drive was encrypted
Linux and macOS weren't crashed by CrowdStrike because they obviously don't use the Kernel driver for Windows that contained quite the beginner's fault. Also, at least the Linux version is said to be able to use eBPF instead of some shady and untrustworthy closed source Kernel driver. macOS doesn't allow for third party Kernel drivers alltogether.
EDIT: they could have very easily broken their Linux Kernel Mode drivers too, and I kinda doubt any Linux distro has that many capabilities helping in such a circumstance. Also, not sure if the driver crashed the running system or only crashed every time someone tries to boot Windows afterwards.
This is such a display of incompetence, I expect Crowdstrike will soon go out of business. Probably there will be massive lawsuits for damages. This bug was, by all appearances, very easy to catch by Crowdstrike. Instead they continued to push it to customers for many hours. That is a massive display of incompetence. That is even apart from the question why this software was tied so closely to the Windows kernel.
Hi Nick, why don't you mention how firefox 128 did introduce a "privacy preserving ad feature" that looks very like an integrated data collection system for ad based companies more or less as google's "privacy sandbox" does?
I use the Mainline Linux Kernel ppa to be able to update my Linux kernel in Kubuntu and the highest that it is currently showing for Linux kernels is 6.9.9, which I have had installed for some time now as my active kernel in use. Something that is weird is when I go to uninstall the older Linux kernels that were used by Cannonical to create the 24.04LTS, they will not uninstall, I can uninstall any other older kernels, just not the ones that Ubuntu installed with.
15:30 if I understood correctly, you can’t install anz adobe stuff because that needs integration old internet explorer, but if you installed it some other way than you can start the programs
Aaaaah interesting!
Then why not install a web view for Internet Explorer via Wine Tricks or Bottles?
@@cameronbosch1213 because Adobe has a hard dependency for the actual Internet Explorer, which wine doesn’t have the rights to use, I think
but as I said, I only saw that in a video of someone and sadly I don’t remember whose it was
I wonder whether this will give some of them the idea of dumping Windwoes for something more stable.
Yes, stable like Debian or RHEL, that had exactly the same issue this year, courtesy of Crowdstrike.
I pray for the sanity of every IT people all around the world right now. 😔
when i was a kid, i thought that hospitals, airlines and major infrastructure had their OWN closed source OS, and that they didn't use windows, mac or linux at all, but today i see that if they were to actually try to make their own OSes, it'd be a major waste of time and resources
1st time understood the importance of safe boot of windows !!! 😆
I wish to correct something. The Crowdstrike tool update seems to have been a distribution corruption. I am not sure a similar thing couldn't happen on Linux. according to Low Level Learning this was a null pointer dereference clobbering registers. What protections does the Linux Kernel have against a dimilar issue. You would still have to touch a linux box to boot into a recovery environment if a proprietary driver caused a kernel panic durring bootup.
openSUSE name suggestion:
- Gecko Linux or Geeko Linux
- openGecko or openGeeko
Geeko is the name of the chameleon on the logo I think.
This only affected people using this software. I've been in IT 25 years and this is the first time ever hearing about Crowdstrike. This is the kind of thing that makes me dubious how such a "mistake" escaped from the developers department, to QC departments, to beta testers and so on before ever becoming a release candidate? In this day and age how can one not consider a nefarious act? Yes if you have been in IT as long as I have or longer you must question. Windows has been an unfinished product since it was launched close to 40 years ago (hence the constant updates/security fixes etc.)
Also none of the systems I am around daily, or sites I visit (like Amazon) had any issues. So this issue probably was a limited issue around the US and not some world wide massive outage where people scream like they are on fire. Large corporations are mostly affected because IT IGNORES the warning signs (seen it in real time!)
Can confirm lightroom and Photoshop work with the wine update
6:45 the issue is: why complain about it NOW? after so many years ?
Strategies change!
My laptop brightness hotkeys will freeze until it wake from suspend. I had 6.1 kernel version. With 6.5 the issue was resolved. I hope 6.10 will help many.
Is it probably because the kernel was only null for Windows, but not for other OSes? Like, the developer who compiled the driver didn't notice the null error since the file size is not showing any error, but it was compiled correctly for Linux and macOS.
iirc, someone told me that Debian and RHEL-based distros earlier this year were crashing due to Crowdstrike.
Even if they switch to Linux someone can still push some "security driver" updates that crashes Linux just the same.
It's heavily misleading to say it's Windows instability this time. It really is just a faulty driver.
Of course this usually doesn't happen on Linux because most of the drivers are reviewed and pushed to the kernel source. But any third party driver can crash the kernel just as easy as Windows, if not easier.
Big difference is that drivers aren't forcefully pushed to Linux machines. Any decent linux admin will only pull updates, making this sort of bug localized and much more avoidable.
@@mytech6779 I'd argue given it's Crowdstrike specifically. What makes you think that automatic updates aren't also turned on in Linux by default as well?
This time it's just the Windows driver that is broken. This can totally happen on Linux as well.
Yeah, shoddy programming is at fault here. There are probably some cost-cutting protocols at play here. Understaffed and under-qualified programmers no doubt. What the penny pinching accountants at CrowdStrike managed to save will probably hurt them in the long run.
@@hotrodjones74Nah, they will just (insert euphemism for firing people here) lower staff instead rather than touch the senior or even medior managers.
@@shamringo7438 I didn't say anything about defaults, I said forcefully pushed.
Though on that tangent, many distros do not have auto update as a default.
As a person who has been teaching OS for more then 10 years, I would give my 2 cents about CloudStrike issue in Windows - People blame microsoft for this and they say it is their fault because of the bad design of the OS. The reality is that this problem could happen to any modern general purpose OS. In modern CPU you have a mechanisum called isolation rings. For example in x86/x64 you have 4 - from 0 to 3. Linux, FreeBSD, Windows and MacOS uses only two of them 0 and 3. When the boot loader loads the OS kernel it gives the control to it and it runs the core subroutines and most of the drivers/modules in this ring 0. This ring is highly privileged and everything which runs there has full acess to full CPU instructuon set and full memory. When the OS pass the control to the user space processes it sets the CPU in ring 3 and then pass the control. Then the user process has the ilusion that it has full access to the hardware but thats not the case. When a user space process needs to something with the OS (e.g. open file) it use a core library of the OS. Behind the call of the function to open the file there an interupt instruction that triggers a mechanisum in the CPU to wake a specific subroutine of the OS. This way the OS will take the control and do whatever is requied (e.g. validate permissions, use the fs code to read the blocks from the storage, loads the data in memory and etc.)
Using this isolation rings you can ensure that user space process cannot interfere or break another process. However, when the code runs in ring 0, it can break everything intensionally or unintentionally. This works the same for all modern general purpose OS. So it doesnt metter if you have a buggy driver in linux, windows or macos, all of them can fail. In this specific issue, CrowdStrike have a bug in their driver which is there probably for a long time. They realese a new definition file that unlocks this bug. Since the driver is kernel mode (ring 0) it demages the whole OS. Why they need ring 0 acesss - they hook their code in diffrent calls in subroutines in order to collect audit data and later analyze it for different patters. And this works the same in all OS. Furthermore, CrowdStrike did few similar problems with Linux and MacOS but with less impact.
Also M$ ans MacOS enforce only singed drivers to be loaded in kernel mode exactly because of this impact. So we have a well known trusted 3rd party company which is using signed driver with their certificate and they just messed it up.
I like the idea to blame Microsoft but lets not blame them for something that is not thier fault.
P.S. there some projects like eBPF which exists in Linux and even Windows that can help by doing static code analysis verification before opening the door to the kernel space. However, they have some limitations and it is up to the vendor to develop thier programs...
AFAIK, Crowdstrike on Linux now uses eBPF, meaning that this issue wouldn't have happened on Linux, at least on systems with relatively recent kernels.
The issue isn't necessarily that MS didn't prevent this from happening, because bugs happen, it's that the OS has no mechanism in place to recover from this. Why isn't Windows rebooting automatically in safe mode after 2 or 3 failed boot attempts? Why didn't it offer to boot while disabling the Crowdstrike kernel driver, as an option?
It feels like either a cascade failure of various recovery systems that didn't trigger, or an oversight in terms of OS design, where they just don't have a solid recovery option. Booting 15 times in a row into a BSOD shouldn't happen :)
There is also what MacOS does: system extensions API.
The kernel essentially asks for a process in userspace if something should be allowed (it reminds me of a micro kernel design).