Crowdstrike Falcon takes down EVERYTHING
ฝัง
- เผยแพร่เมื่อ 15 ก.ย. 2024
- Massive screw up.
Official Discord Server - / discord
Follow me on X - / atericparker
Disclaimer: The content in this video is for education and entertainment purposes to showcase the dangers of malware & malicious software. I do not encourage any form of illegal hacking, nor do I encourage the usage of game cheats, cracks or hacks.
Cracks are sometimes shown to highlight the dangers of software piracy, my content is not intended to teach anybody how to pirate, or maliciously hack.
More Malware Investigation Videos:
→ The latest "NORD" Malware - Nordsecured: • The latest 'NORD' Malw...
→🧧VIRUS WARNING🧧 NEW Optifine for Minecraft 1.16 SCAM: • 🧧VIRUS WARNING🧧 NEW Op...
→ The wilkreate TH-cam stealer virus that started this whole trend: • Fake sponsor DESTROYS ...
(C) Eric Parker 2024
Thanks for quick mention Eric -- very glad everyone is springing into action to chat about it and get the word out!!
John, you made the video talking about this like 3 hours after the incident. Do you sit and check every minute did some crazy stuff happen?
John Almmonds
f
That is what you get when you trust ring 0 to random proprietary bullshit.
But the thing is you cannot hack windows if it is already crashed.
It's the ultimate secure mode, CrowdStrike are actually brilliant security innovators!
what are you talking about?
it's not about this at all...
@@AmlalElMahroussit literally says they pushed a bugged driver. Drivers work at ring 0. So this has everything to do with that.
@@hashtag9990 huh, what does kernel level drivers have to do with proprietary software?
@@AmlalElMahrouss if they were open source, maybe someone would've picked it up before going to every machine. See how xz vulnerability got caught recently? People would just voluntarily test their software even if the company couldn't do basic QA, before pushing the update. It could still have gotten out, but the chance decreases because more eyes will be watching it.
[Update]
It turned out that the weird looking files were some definition files instead of executable drivers for CrowdStrike. And the problematic file is one of them being filled with nothing but zeros. It wasn't those files with random-looking bytestreams that caused the problems, but the one bad apple filled with zeros.
[Original]
Our team also got the bad "drivers" files and yep, they are not even drivers. Just nonsense bytestreams. Why would such an obvious error not being caught in the first hand is quite jaw dropping ngl. Even most malware samples we get do sanity checks to make sure expected PE files have valid PE headers. This is just insane. Moreover, why Microsoft chooses to force crash Windows instead of not loading nonsensical sys files is also inconceivable to me.
How does Windows bootloader not just log errors if it sees a nonsensical .sys file instead of immediately panicking? It's harder _to_ screw up here than not to. It has to read the PE header to load it into NTOS anyways, if it doesn't see one it should just move on instead of immediately dying.
Something ring 0 ? If it's a "Critical process died" error, it may seems like a driver loader (maybe the falcon subsystem?) fails because random bytestream, thus exiting, and the kernel just panics as a safety (which is stupid but oh well)
the reason windows crashes is because the driver is loaded with the kernel (ring 0) and if a driver crashes (since it is ring 0 it is a part of the kernel) the kernel crashes
Microsoft can't do anything about it, not even implement a fix because the only reason to avoid this is to just load the drivers as ring 1
I guess the person responsible for this "driver" was replaced by "cheaper" alternative (fired)
@@EmayeahThe driver loader should have mechanism to ensure it is a driver and not a meme image renamed with a .sys extension
Funny that I literally left work yesterday for a week of holiday and then this goes down, feel like I dodged a bullet lol
lol same here 🤣
It seems we bother dodged the same ICBM.
luckyyyy, I was stading at the front of the store and watched each register blue screen one by one then looked own ay my scanner to see "cannnot connect to server". it was a long shbift
During the Java Log4j panic, I got the notification the night I was going on vacation for 4 weeks. It was on a Friday late afternoon too so I emailed the details to my team and logged out.
Same here, I've been on vacation for the past week so I may have my work cut out on
Monday
My dad's work laptop was affected by this and I tried to help him fix it, but booting into safe mode requires the Bitlocker recovery key, which his company's IT department did not provide him. So in summary, his laptop got bricked remotely with no way of fixing it 😂
Even worse. They use Azure to get the bitlocker recovery key. So if that is also affected now not a way to get the bit-locker key
@@ToxicRoachX50 I'll bet the only Azure machines affected are Windows VMs where the customer has installed this AV software on there. Same as with AWS. MS has long been known to dogfood their own software, and using CrowdStrike seems like a crazy decision.
@@ToxicRoachX50 He got it fixed now. Apparently his IT guy just happened to leave his computer on so it didn't install the update or something. All the other IT specialists at the company could not get in their work PCs.
The most secure laptop ever
@@henson2k lol
It's always gotta be someone on the inside screwing it all up
and code that was written by a first-day-on-the-job person was deployed automatically worldwide without any kind of testing
AI wrote and approved it. humans not needed anymo. tecnologi 🤖
7 layers of user role including “approval” and still this happens 🤦🏼♂️
Just on your last point - this doesn't just affect end user devices - it also affects servers.
There is event a "Worst Case" where those servers are also using Bitlocker, with the recovery keys stored in Active Directory... meaning thye can't even get their recovery keys to repair servers
Only servers running this particular AV. Which, given corporate policy would be all of them...
Bitlocker is enabled on my device.
My device does not have this problem but I am afraid if this happens to me.
@@Goku789 then don't use bitlock
Form the Wikipedia article, Microsoft being Microsoft: "Microsoft's Outage Tip for Customers: Try Rebooting Your System 15 Times" 😑
Nah that's clearly not right. You must of did something wrong, take it to an expert before you make your computer sad, you're scaring it
I haven't checked, but it seems likely that it could fall back to automatically booting to safe mode after so many failed attempts, like it will try to boot to recovery on the third boot after two failed ones.
combined with the search results being absolutely flooded with the always useless "run sfc /scannow and pray" responses
aksually startup repair should fix it /s
Apparently crowdstrike has a 5% chance of downloading the fix between loading and blue screening, so the 15 times is a play at averages
What a day! The CBC testimonial appears to be from "Commercial Bank of California" and not "Canadian Broadcasting Corporation", but good coverage on this morning's events none the less :)
Honestly, when I first woke up; I couldn't believe it and when my friend who isn't the tech saviest told me "An antivirus did it" I scoffed and mocked them like "nahh, I can't connect to my azure cloud so therefore it's microsoft's datacenter outage you bozo. The software might be struggling to connect to Azure and they're BSODing because of that" without actually looking into it because I live a arrogant and ignorant lifestyle of not reading/watching the news.
Oh how very wrong I was; I can't believe this. Everyone feared Y2K and yet all we had to fear was ""updates"" this whole time :')
I'm a sysadmin and my company uses Crowdstrike... Fun day for me so far!!
Me too. Well, rather a fun night. I was up all night recovering our servers. Hope it’s not too bad over there for you!
@@justseny thank you!!!!!
@@justseny Thanks! Only a hundred computers and 1 server we had to do the manual fix on. We have about 300 endpoints and 20 VMs at my site, so it could have been a lot worse
Lets hope all IT/sys admins get a raise to fix the damage done by just 1 company. I hope your company is allowed to send them a huge fine to pay you guys off. Gl!
"The best way to protect your files is destroying them" CROWDSTRIKE
clownstrike 🤡🤡
This is actually a pretty ingenious cyber attack vector in the event that an apt simply got access to a given cybersecurity company and forced a faulty update. Any system running auto updates would be hit immediately
What I don't understand is the rollout they follow.
Pushing an update all at once, without any kind of structure? It's not like they have 100 users!
Precisely. Where was the change management and gradual rollout? This is the number one question, because bad updates will happen!
3:50 that’s not just any game dev, that’s Masahiro Sakurai. He responsible for the Kirby games and Smash Bros
Yup. He's a household name
I’m a network engineer and had to go into work at 1 AM this morning because about 80% of all of our windows servers and PCs were blue screened. This is about 5500 Windows machines that we had to touch.
I work in IT. My whole company was down today. That was a nightmare walking people who don't know how to use computers through boot menus and such to get safe mode enabled
Just tap F8 after the bios, right???
4:45 - For my area the 911 and Prison Services and were the only functional part of the government infrastructure being on private servers to keep them up in case of an incident like this
Also no Airport Delays because we only have 3 flights and the incident was resolved a few hours before the first one
Funny enough, this is veryyy interesting from an OSINT perspective
Now everyone knows what security software the affected companies use because of this. Whoops!
@@VideoGameSmash12 thats what ive been saying lol
This is what happens when you outsource your updates.
"Just push the 'Update All Windows Devices' button, Akimov! How hard can it be?"
All i can see is a future Kevin Fang video about how a faulty kernel driver took down the internet…
When covering the services affected you left out the one I’d say is among the worst, this took out the epic system across America, this outright crippled hospital function. You can’t file or retrieve any medical data on patients
This. Imagine a patient in the middle of open heart surgery and the screen displaying all the medication levels, vitals and micro cameras just blue screens.
Thousands of bills
@@thahrimdonmost of those devices are not open to the internet, and also hopefully not using windows
It is more that retrieving patient records might no longer be possible, so the hospitals might miss out on critical information
@@jan_the_man tbh with the way things are today I wouldn’t be surprised if the scalpels are Bluetooth
@@thahrimdon huh? what is that even supposed to mean??
We got the bad drivers at work today as well. Was shocked to see my station working and a station next to me that was functioning yesterday normally at work is having BSOD. Tried to restart and same thing. Until i realized most PCs at work nearby are having the same issue, then I was told an outage to Microsoft computers are affected and turns out airlines cancelled flights due to airlines and airports are both affected. Crazy to see the world was completely paralyzed because of 1 bad driver update.
Wannacry without the encryption
It's effectively wannacry when you combine this with enterprise-level Bitlocker 😂😂😂
@@emanekafecaftoggaf6893 but with no address to give away all your money to :c
this is what you get for installing kernel level malware in your system
Cat ears at 100k, don’t forget…
The Sorcery of Computers Confounds me.
All I know is that the magic screens stopped working.
Hail the Omnissiah!
"Severe lapse of judgement" ~ CrowdStide, most likely.
Our company switched to crowstrike agent, the PCs are much slower now xD + when you the user is suspicious about the file, there is an option to scan it but it does literally nothing, no popup screen with scanning progress etc.
Why was Crowdstrike picked in the first place? Did everybody glom onto these guys because they were involved in the DNC leak?
cat ears.
dont try and ignore it eric.
And this people is what happens when the whole world runs on online systems
Why always on a Friday ...
In my country, only one of the airport was effected, it's not popular Crowdstrike here.
Someone lost their job today and that's sad, but he's got a story he can tell people for years and years.
There are so many things wrong that Crowdstrike, I don't think they will ever be forgiven:
1. The system file was either formatted wrong or completely corrupted. They sent out a cfg file with incorrect content to millions if not billions of machines
2. The CEO tried to brush of the severity by stating the faulty driver isn't running in Kernel mode, which is technically true, but the kernel driver was reading and parsing the faulty content in the cfg file, so the kernel driver panicked.
3. The CEO also failed to mention that instead of getting a new driver checked and signed everytime, Crowdstrike downloads cfg files, (including the faulty one) over the internet, which is then read by the kernel driver. This bypasses ALL Microsofts checks for making sure the driver is signed and working.
4. He also failed to state the current protections they have against malicious activity abusing the way the drivers are downloaded
It's been a fun day working at an MSP😅
God speed. I spent all night getting critical infrastructure back online, I don't envy you.
Hackers creeping waiting to ... strike 😅
Somehow CrowdStrike's internal validation process and automated tests that has worked for countless updates suddenly failed? Maybe a more likely cause is having a few key disgruntled and/or compromised employees, and not having *enough* people in the validation approval chain to harden the wetware threat vector. I want to know who benefitted from the outages annd what was happening when everyone was focused on this. Watch the other hand.
Apparently Windows doesn't validate drivers before signing them, either.
"Windows" doesn't sign drivers at all. Microsoft does. But I suspect Microsoft doesn't check drivers submitted from well-trusted security companies and doesn't even get to see their source code since the MS code validation teams are largely script jockeys that don't have DOD level or equivalent security clearances that would be punishable with prison time (or worse) if they leaked something. It's clearly a failure within CrowdStrike, but it might have been a socially engineered or coerced failure.
Google is buying security agent "wiz" for a lot of money. Guess Wiz has grown in value. Wiz is direct competitor to crowdstrike
back in the day when I worked in a large multinational, in IT department, we always, and I mean always, tested any updates offered to our software first before deploying it on the actual workstations, but I guess IT departments in the companies that got hit hard by this, had been shrunk to a point where they simply cannot do that, or more likely, the IT support is outsourced to some people in India
this is what careless "lean strategy" implementation leads to, those people you never see, I mean the IT tech support guys, ARE working, that is why you don't see them, at least that is how it was when I was working in IT support, then some fresh Harvard graduate comes along and says - why do we need them, everything just works anyway!
crowdstrike seems to be on autoupdate with out IT needing to push out updates
@@JoebDragon and that is the whole problem, windows is also like that for wast majority of cases, but you can (or could rather, haven't done any enterprise management for over a decade) configure in enterprise environment to use your own update servers, servers you manage, and let them push out updates that you tested yourself in your environment, but that was 15 years ago
basically, workstations in the domain were set to autoupdate, but all the serving of those updates was done by us, not Microsoft or whoever
IIRC for ChatGPT 3.0 academic paper writers would occasionally report "hallucinations" - when asked to write on a topic too far of ChatGPT's comfort zone it would try do AI's analogue of bullshitting your way out of answering homework you did not prepare. I wonder whether that's what happened here, combined with cutting costs on quality control because it's "AI-automated" (the company does sound the type).
Right this needs saying now, fixing this does NOT NEED people to go to physical machines 1 by 1, the fix can be fully automated just like we did, force hosts to reboot into safe mode with an IPMI Command, and just wait, we have an API that instructs host on pending jobs, even in safe mode, add delete driver DRIVERNAME, check WSUS For any updates before auto rebooting,. THIS IS NOT A FIX THAT SHOULD TAKE DAYS. Hours at most.
We had 3500 servers fixed in just under 2 hours.
Will take days to fix my ass. The only people complaining are those who have to work overtime to fix it, cos there own systems and balances have clearly failed, me and my time clocked of at 5pm as normal, and no one back till Monday, and non of us will stress on it, because were fully prepared for things like this.
And even if we couldn't fix it like that, we can redeploy all 3500 servers from backups in just under 4 hours, as per our disaster recovery plan.
The incompetence isn't just crowd strike, ALOT of IT teams gonna have some explaining to do here, and if they don't have to explain it there bosses must be more inept than a flat earther.
Best mass update screw-up since defective Win-10 Network “Class” Drivers in 2018. That time we had to fire up a Win7 machine and wait to download the updated update, then apply to each inaccessible Win-10 box thru USB. (I’m not CompTIA Net+ only Prntr+). I heard they could have fixed that one by adding Rem in front of one line, but nnooo, they cleaned it up by adding a few thousand more lines of code
this is almost funny until you realize the huge impact this has had..
How did it happen? Painful incompetence. It wasn't checked, therefore it could have just as well been malware, which makes this a security incident as well, no matter what they're telling people. The waffling about AI doesn't exactly inspire competence either.
was this the world wide tech issue that was on all the popular news channels this afternoon?
Yes.
I went to bed around 2/3 am ,woke up almost at 12pm , didn't notice anything till later on news 😅
You can't do proper antivirus without running it on the kernel level, there's nothing weird about that. XDR/antivirus needs to see and be able to interfere with malicious activities at any privilege level.
The problem in this case is Crowdstrikes lack of quality assurance for such crucial software.
Sean K testing in production again.
That was exactly my question - did nobody have to approve the update before it's pushed to millions of devices?
3:59 all australian companies affected. Love how this affects government services here such as Centrelink (welfare) and NBN (National Broadband Network)
How don't these companies have manual review for any updates from internet with HW samples matching production machines? Like do you just trust that any update won't fuck up your machines and possibly lose you millions of dollars? It's not like you can sue issuer lol.
I worked on production line for printer ink cartridges, and this is what we did. Despite that, every fuckin' time there was windows update one of our machines stopped working. So we have to load backup without update and forbid installing it.
We fixed our servers by deploying winpe and booting them into it. There they just delete the file that is causing this issue and reboot. This solved our server side Problem quickly. But clients with Bitlocker will require hands on solving... :(. Boot into winpe or recovery if possible, unlock the bitlocker drive (manage-bde.exe) with recovery and than removing the CrowdStrike driver.
Falcon isn’t an outstanding product, they just strongly push to AI/ML instead of relying on signatures (so far the ML strategy proves to be inferior) and focus on good admin webUI (hence why it’s used by so many companies)
I wouldn’t run this kind of software over a conventional EDR, but hey, something something cutting costs :3
I'm 20 and I just finished a course on I.T, I'm a complete novice in this area and even I know that your should never, EVER, update something without testing it first.
this is why linux is seriously ahead of windows, in what world would a software product even have the position and file-rights to fuck up the core OS this hard? it isn't even something integral like drivers for core hardware like drives/graphics card, yet it still has this much control, it truly speaks to the risks of the "standard" that software should have excessive rights on your computer AND be proprietary...
yes
Well the reason the crowdstrike agent needs privileged access is because it needs to place integrate itself in the network stack so it can scan for malware. To do that the only way is to use driver. All drivers no matter the functionality run at the same level as the kernel. When the kernel makes an error that is non correctable it raises a bug check and goes into BSOD. The only way that this could have been caught would be with more thorough testing and a rolling release system. Personally, almost all of my machines run Linux and I would like to believe that this could have been prevented with Linux and FOSS but it still has its flaws and this same scenario is probably still possible.
No it's not, DKMS on Linux has the potential to fuck up your system the same way. Linux is not a microkernel.
I feel bad for one of my old job's customers...
My old job, which may be the biggest Microsoft-simp imaginable, kept pushing them into ditching Linux for the "vastly superior Windows experience", they have NEVER had any software issues until that part of the contract was filled, then they had BSOD after BSOD after BSOD after BSOD, and it was through that customer that I knew about this incident.
This is not a cyberattack but damage is comparable or worse.
The generous reading here is that they did test it but accidentally pushed the update anyway.
had a flight this morning 😭😭😭
Re, why embedded systems run Windows. I have had security people at my company tell me that they consider anything open source to be more vulnerable than closed source software. Microsoft may have changed their position, but those people internalized it two decades ago and are now in higher level positions.
Another thing is companies software and infrastrucutre is already designed for Windows, and no one has much experience with Linux and/or no one wants to shake the boat. To the point I have seen actively maintained websites and applications being written in .NET Framework. That's only officially supported on Windows, with IIS for websites. Heck, I know a senior programmer at a pretty big company who created a brand new static website in .NET Framework by cloning an existing project they're familiar with. In 2024!
Then you get the part where Microsoft does make a more slimmed down version of Windows for those applications. Plus, the company likely has an enterprise license, so it doesn't cost them much/anything in license fees to put Windows on all the boxes. Of course, then running a stupidly resource intensive AV because of policy...
Edit after having finished the video.
If CrowdStrike doesn't survive while SolarWinds is still going strong I will be surprised. That a was far worse outcome from far worse incompetence, and they still are a massive company. The only difference is SolarWinds was able to temporarily blame the vendor for their own mistake.
Even FedEX was effected by this said by a commentor on Mutahar's video on this and it effected packages and it delayed packages by like 2 weeks. Not to mention some University/College Students were infected by this just because they needed this shitty software to get on their campus' network. I know someone personally who was working on this BS yesterday due to the fact that the company he works for was also affected by this incompetence. If I were Microsoft I would sue Crowdstrike Falcon since their stock dropped in value because of this bullshit.
And I thought college campuses stopped giving away antivirus/anti-malware products to their students...huh.
I got this update automatic today and deleted the file after knowing what the issue was. 3 hours of my work time gone. They must test if it can be installed without error. Unacceptable from a cybersecurity company.
The driver wasn't just invalid, it was all null bytes from what I heard
Crowdstrike - "We're Better Than Competent - We're Diverse."
Why do you weirdos have to be racist at the most random times 😂😂
Not far off, Crowdstrike said how proud they were for having Indians working for them.
Screaming the 🤬 DEI boogyman isn't going to make you any more credible..BTW...
@@alexdhall Ok groomer
100k subs = eric will put on cat ears
I can’t understand why Production/Customer facing systems have direct Vendor driven updates. At my Bank all updates are delivered to Test systems, validated and only then rolled out to Production.
The Machine Spirits are rebelling! Praise the Omnissiah for protection.
I couldn’t buy gas this morning, probably because of it. Huh.
yup
Like 70% of screens in a factory that i work in were just bsods. I even took a couple of pics. Im not affected myself since i use Arch :P
Dealt with this all day, someone help me
we will not forget, you're halfway there
And this is exactly why forced auto updates never sat right with me. Years ago I was forced to update, you know, Windows 10 just randomly decided "Fk you you will get the latest untested trash pushed to you", so it restarted, updated and it corrupted my NVidia drivers and I had to use a bootcd to save all important files.. Ever wince this happened I have Windows 10 updates turned off.
I've been at the beach all week. Not my problem until Sunday
I hear Gary did it ... classic Gary!
Crowdstrike devs were too excited about their upcoming vacay and didn’t wanna go through and test their driver 😂 let’s see how many people will lose their job at Crowdstrike!
They need to change their name to CustomerStrike.
somebody is gonna lose their job
Only once the crunch is over 😂
Nope, Crowdstrike get mega bucks from Blackrock for hiring them. They'll keep them on and still turn a massive "profit".
i saw a video of dave plumber is the name i think who is an ex microsoft employee who wrote the original task manager and windows menu versions.
according to him how this happened is that the signed driver loads additional files in which contain then the actual code being executed. so the certificate for the driver stays valid since technically there was no change and those extra files dont get validated.
for some reason one of those files was all null and because there is no validation and error checking it crashed Falcon.
at least thats the story i remembe rbut i also saw a bit on twitter about invalid memory adress calls to a memory bit that was never allocated.
No idea of the full story and how everything ties together tho.
What a gigantic single point of failure
You should change the tense of "Crowd Strike" to "Crowd Struck".
They really did stroked the crowd
I don't think such a large portion of the tech world relying on a single company's product was very smart 😅
Y2K but 24 years later lol
right. Why people blame this company? M$ windows was successfully shut down, so the threat was neutralized. Job done
Can't feel sorry for the ones working heavily fixing it. They're getting paid big money for tapping on a kb basically.. 😂 so it works out for them.
Hey IT folks.
On the plus side. Free overtime right?
Who reimburses for this? Do the companies effected just eat the billions, or does CloudStrike need to pay up?
I totally expected them to test an AI or some new model. Like, we put all our malware and drivers and whatever into an AI and it generated us the totally best new world-wide awesome new Windows driver --> but. in reality, like with other AI models, it made crazy mistakes that even a Junior programmer would not do that way.
a blue screen is called a "bug check". Isn't it a kernel panic? or has that changed?
Kernel panic is mac / linux.
NO It's BSOD
Even Reddit got shut down
2:36 😂someone had to say it
Wow, they even fuct up with the No. of asterisks 🙂
Should have stayed with Kaspersky, eh?
This AI thing is working pretty good AMIRITE?
wait wait ... there's no way to boot windows into safemode remotely ?
Without an IPMI?
Funny shit I was yesterday to my coworker like hey my laptop crashed and laughed how our other work laptop crashed too, thought it was coincidence before shit hit the fan. Hate that software tbh. Total failure.
Checking for null-pointers in your multi-million-worth Windows Kernel driver is so like 90s or whatever. Doesn't Microsoft check them or something? ;-)
Why do it myself and spend like 5 minutes in a VM to get this crash and then avoid paying millions of damages to my customers? Weird.
windows gave me an update message this morning, thank god I ignored it lol
Poor guy who screwed everything up
"A game director" :/
Is this faulty antivirus driver? I never installed this program before and got no BSOD.