CrowdStrike Avoids Responsibility
ฝัง
- เผยแพร่เมื่อ 15 ก.ย. 2024
- Recorded live on twitch, GET IN
Article
www.crowdstrik...
My Stream
/ theprimeagen
Best Way To Support Me
Become a backend engineer. Its my favorite site
boot.dev/?prom...
This is also the best way to support me is to support yourself becoming a better backend engineer.
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-K...
Get production ready SQLite with Turso: turso.tech/dee...
"It wasn't our kernel driver. Just something our kernel driver dereferences, isn't signed, doesn't validate its value and fails critically when an unexpected value appears."
Exactly! If third parties are to be allowed to run kernel code, they at the very least shouldn't be allowed to circumvent windows signing their updates by doing them in this roundabout way. Drivers are signed for a reason.
Checksums are your friends, everybody!
"Also, it was put there by our updater process, who also didn't check its signature, or validate its contents. And the update was distrbuted by our infrastructure, which also didn't verify that the files uploaded. And while it did not stop the update automatically after multiple hosts went down, we were the good guys when we pressed the "stop" button after 8.5 million hosts were rendered inoperable. It is not our fault, you see."
@@YurgenGrimwood The issue is that if it had to go through the windows signing process, it would never be able to do so fast enough to be valuable as a security tool. Zero-day vulnerabilities are attacked sometimes within hours of release, there is no world where any OS's kernel signing process would be fast enough to validate something to protect against zero day attacks.
That said, that kernel code should have been written to validate the content update had valid data before it tried to read/run it.
I blame msft for allowing them to be “signed” and run freely updates. Without msft certification none of this would have happened.
homies didn't get the memo that "move fast and break things" doesn't mean to break entire airlines and hospitals and stock markets
Move fast to break things
I worked at Crowdstrike 6 years ago. Their motto was "two feet on the gas pedal", the CEO is a hobby race car driver (richie rich stuff).. I am not kidding. All gas, no brakes. I left after 10 months and gave up pre IPO stock. I also worked on the release engineering team and could not take the stress.
@jeezusjr did they use object oriented programming?
Move fast and break things in your development environments...
correction: "Move fast and break EVERYTHING..."
To be fair to CrowdStrike, if the system is stuck in a continuous boot loop it can't be infected with viruses and the data is protected. So mission achieved I guess
AI... Is that you?
Task Failed Successfully
You could take out the drive and just insert some malicious code but it’s definitely a security improvement
CO2 emissions were briefly down as thousands of airplanes sat grounded. We salute Crowdstrike for their environmental stewardship and decisive climate leadership.
Task failed succesfully
Typical behavior nowadays: deny everything, never take accountability for anything.
They didn't want that to be used against them in the upcoming lawsuits
Typical brainlet reaction to the reality of making statements that give you liability. I'm sure you're so honest, and accountable that when the cop pulls you over you start telling him every crime you've ever committed to take accountability!
@@oussama7132 and make no mistakes, lawsuits are coming!
Wait for the headline to blow over in a couple days and everyone will forget. 😂😂😂
@@hotrodhunk7389very true
The CEO of Crowdstrike was also CTO of McAfee back in 2010 during its global crash. Let that sink in.
This comment needs to be pinned.
@@edrocheAgreed
@@edroche prime talked about this in the video
At least McAfee has an interesting story... Had
Failing upwards, classic
Crowdstrike learned from Boeing: Say that you own it, then proceed to blame others
"The buck stops with me"
Boeing: of course we installed software in the plane and conspired and overtly lied to the FAA about it, oh but it was pilot error that brought the plane down.
The young Ethiopian pilot in the Lionair flight did exactly what he was supposed to, when unbeknownst to him, MCAS put the plane in an unstable configuration. The aircraft still slammed into the ground at Mach 1. Nice work Boeing.
Both Boeing and Crowdstrike blamed their customers.
They could have just said "we tested in production on friday"
The annual disaster recovery test resulted significant improvements....
It was just a beautiful blue screen of serene life.
LOL! That made me laugh. Thanks.
It was the serene blue sky of the Sixtine Chapel...
One of those wallpapers of a beach where the water is so clean and calm
Birth is the opposite of death.
Life…life has no opposite.
@@XeenimChoorch-nx8wx Not really. birth (or creation in general) only happens because of death.
The problem nowadays is that CEOs or other high ups don't face any repercussions when they have clearly failed at their job. What should be high risk high reward jobs, have turned into a 0 risk, insane reward jobs. Whenever they fail, they either stay in their position, or step down, receive a ton of severance money, and simply go work at the next company who for some reason will gladly hire them.
Meanwhile the poor engineer who rushed this code out will A) feel absolutely terrible (even though it's clearly a process failure), and B) likely get fired or at least get cooked by management for as long as he works there.
true, the mistakes of the CEO was to allow directors, who allowed manager to hire a engineer, who made a broken PR... actually everyone should pay the price and get fired, and never be able to get re-hired somewhere else.
to resume simply ^^
@@Mylordkaz I mean, we don't even know if engineer is to blame here. It could've been, as prime said, "management said we push now and ignore some time expensive tests" situation. We won't know and CrowdStrike surely won't tell it.
@@spl420 I'm with you on the rush and no test
You can almost feel the lawyers standing over their shoulders making sure they don't say anything that could be used in one of the many court cases that just have to come from an error this massive.
The fact that Crowdstrike has not been slammed by 100s of lawsuits so far, is due the fact that the tort attorneys' computers are still down.
When I was a child, many years ago, the common word for the kind of corporate word salad in that press release was "gobbledygook". I note that they carefully avoided revealing any of the following:
1. What, exactly, caused the crashes.
2. Who, exactly, sent the faulty file to 8M customers.
3. What, exactly, do they intend to do to prevent recurrence.
There should be a law against releasing this kind of gobbledygook, with a mandatory fine of $5000 per instance.
Why is 2 important? That would absolutely be throwing an engineer under the bus. Have you heard of blameless post mortems?
5000 whole dollars? How will they ever recover?
That's like you getting a fine for a nickel. I'm sure that will learn em.
1. Most likely, at the time they put that blog post up, they didn't know what _exactly_ was the cause, besides what was already known: somehow, the file content led to accessing unmapped memory. I'm pretty sure they were as flabbergasted as everyone else. However, they couldn't keep silence until they figure everything out.
2. This is largely irrelevant as it's not even clear what does "exactly" mean in this context. Unless this resulted from an attack on the CS infrastructure, the Event was an outcome of a long chain of events, possibly rooted somewhere at the inception of this Falcon thing itself.
3. How can one formulate a meaningful prevention strategy without knowing all the technical details of what happened? They could always spit out the standard blah-blah about "establishing guidelines", "further improvements", "carefully crafted polices" and all that. They didn't however.
$50 per word I think would be a good rate. The 'cost of doing business' would keep their nonsense appropriately short, giving people a way of determining the value of the information (If it's long-winded anyway, then they're unlikely to be breaking this proposed law; if it's kept very very short, more people are gonna get suspicious).
Problems with your proposal:
- Who decides what is gobbledegook and what isn't?
- How exactly would they determine that?
The road to tyranny is paved with perfectly reasonable laws.
Azure screen of dire misfortune.
inclusive language, +1 from the DEI police
Chartreuse screen of minor inconvenience.
Warning number #3360B0
now i check how close i guessed with the hex value
Pretty close (at least to the colour seen through a camera. I've never seen a BSOD on a monitor directly)
I worked for crowdstrike for around a year, and had to quit. Worst job I ever had, they pulled this shit all the time, and internally, they point fingers instead of looking for solutions. I remember a director calling out someone by name and berating them in front of like 50 people and NO ONE DID ANYTHING, I reported him and sent in a audio recording and nothing happened. Additionally, this screams like a manager was told by executives they needed to get this update out to look good on some arbitrary metric. They will and do threaten jobs if you don't just do what they want. I remember putting in some process improvement presentations together and I was told if I didn't stop pursuing it I would get written up. Worst experience I've ever had at a job, hands down. I literally cried every morning before signing on. I would take working a dead end job, getting paid min. wage than going back to a company like them.
This is the majority of corporate and tech jobs.
But you were one of the 50 who did nothing. Only after the fact
@@avwie132 can't u read? He reported it
@@avwie132 what would you expect to do? Berate superiors and get fired for trying to stand up to a bully with power? Only real option he had was to report it to an authority who could do something, odds are he had no way to stop this person without getting fired in the moment.
@@fredericchoppin how were they so sure nobody else filed a complaint? They claimed nobody did anything.
Their kernel driver loads unsigned files, validates nothing, interprets some part of that file as a pointer (null in this case) and dereferences it. Imagine for a moment that this file wasn't all zeros, but crafted by malware. This is a security company. Their product makes you *more* vulnerable.
Imagine every Normie in the world knows about your company for crashing the internet and your response is "there is a chance that solar radiation has hit the exact bit in our application on each machine in the word"
haha, "the Neutrino defense"! Haven't seen that tactic employed since my cousin Vinnie defended me after that spaghetti incident.
We had a company one time try to tell us their shitty software broke because of a cosmic bit flip. This was an error they were struggling to FIX, not a fluke one-time event they couldn't replicate. The server obviously had ECC memory, too.
@@jeffwells641 the fact this actually happened makes it even funnier
Tbf most media reported that this was a Windows issue so no Normie is gonna care to dig into this.
It’s a simple missing step that any competent programmer would do.
Open file.
*check the file was there* (missing step)
Try to read (Fail!)
If they publicly accept responsibility they effectively admit fault and can be sued by 15 different industries.
Although they will probably still be sued into oblivion (and hopefully they will).
I hope so too. My old job had them and i took noticed and let bosses know this software is garbage. Not sure what happened during the issues but i was a happy camper. A company with contracts based on lies should (a company) not exist
They won't. The terms of service state that they only refund fees at most. They are not liable for any losses above paid fees.
@@_sneer_ ToS is not the be all end all, if i slip in that i can fuck your mom every night into a ToS that you sign, can i then fuck your mom every night? Don't answer, i'm already doing that.
@@_sneer_This will certainly be an interesting case - if they can proofe that they followed best practices and implemented industry standards and it still happened they are probably fine. But if there is even a hint of management ignoring concerns or intentionally disabl
ling safety measures you are getting into malpractice and intentionally risking to damage critical infrastructure is nothing you can grant yourself in the terms of service.
@@robstamm60 Oh sweet child, ToS/EULA is only as binding as the court decides it is on the day.
Most of the time a companies ToS/EULA doesn't hold up in ANY court as they are usually unethical or immoral.
Unironic "I'm sorry you feel this way" response.
Imagine being the engineer pushing a change on Friday evening, then getting stuck in public transport because the train has just been upgraded to Windows ME and has been taken out by your fix just like half the rest of the universe.
Move fast, break everything.
This kind of 'soft' language is pretty much de rigueur these days in all mediated communication. Advanced persuasion techniques are at play that seek to apply deep level of knowledge about human psychology and language to influence how the message is received, and how the messenger is perceived. A crude example of this is how a certain conflict gets reported with the judicious use of words like 'died' and 'killed', depending on which side is being reported on by whom.
Having been exposed to this at times in my own work I can tell you the people who are paid to do this, and the patrons of such services themselves get a psychological boost at seeing their messaging - it reassures them into thinking "there, we're not so bad after all".
Unfortunately, part of the reason for this is the way we the public and media react, eager to see people pay for mistakes. We need to be mature about such things and give folks a chance to put things right. When the crowd is baying for blood, enjoying the spectacle, folks will become defensive.
This is hands down probably the most insightful comment I've ever seen written on a TH-cam video, or any social media post for that matter. Probably why it has so few likes, sadly. Can't have sense and reason here.
And we all know it's BS but managers don't know it's BS so it continues to happen
The CEO was also a cofounder. He hired the team of management below him. I blame the CEO.
The executive class is the reason things get pushed out too early, resulting in issues like this.
"This is not related to null bytes contained within Channel File 291 or any other Channel File."
How are we supposed to believe that? The ENTIRE FILE WAS NULL BYTES!
This is simply not true
@@titan_codes Are you saying the Crowdstrike statement is false or the assertion the whole file was null bytes is false?
@@mikehogan8345 the assertion that the crash was due to an empty or null bytes file.
@@mikehogan8345 this kind of ambiguity is so rampant lately that I'm starting to ask myself if it's AI farming engagement
@@titan_codes "It was all just zeros." --> YT : CrowdStrike IT Outage Explained by a Windows Developer @ 10:20
to be fair their terms of service states to not use this tech in critical infrastructure
to be fair they should have never sold it to those running critical infrastructure
That Statement basically invalidates the usage of their software.
An antivirus that i cannot put on critical things, offers no value, because the critical things are exactly the things i want to protect.
every EULA states that the SW is not suited to do the task and if you loose your data they wont do a sh*t. And still everybody pays and agrees with this. SW companies must be held responsible.
@@fabianletsch1354 people want to protect their little file processing systems, that's a thing you know. They shouldn't put critical infrastructure connected to the internet, period, let alone give some third party root / kernel level access to the system. Crowdstrike offered a product fit for purpose. It was good until it wasn't.
8:53 b4shful: "sir, a second channel file has hit production" lmao. This killed me
"McAfee had some problem as well"
Yes, its existence is a problem.
Their statement does NOT include: Assurance they will add logic to their kernel driver which loads the channel files to verify their contents are valid instead of just blindly trusting the contents. Or assurance they will adopt automated testing of even smoke test level so that modifications like this can be tested internally. Or assurance they will adopt gradual rollout strategies which have been common in the industry for years. Also, how much you want to bet the original draft of this statement included "ChatGPT said it would be fine"?
They should do the parsing entirely in user mode, probably they should even run most of the code from the kernel driver in user mode and just communicate with the driver where needed.
yes, they should solve the halting problem
filesize and checksum checks would even have caught this mistake.
The fact that they had the same issue with a channel update that hit Linux servers a little while back supports your point about how this is a very specific message.
Apparently Amazon logistics were affected. People were paid to not come in on Saturday in the UK because the IT systems were down.
People in the US would have been told to stay home without pay. Kinda sick of republicans.
@@SixOThree You think you should get paid, not to come in on Saturday -- which is a day off? LoL Plus most IT jobs are salary or contractors. That's a weird political statement.
@@Sandy-o4p If they were scheduled to work as part of normal operations, then yes. they need to be paid even tho they couldn't perform the work they were hired for. That's how employment contracts work. Employees get paid for upholding their part of the contract. If there is an issue on the employers end that makes work impossible, the employer is still on the hook for the pay. They could demand employees show up and wipe the floors I guess. But usually they get sent back home with full pay.
Employers have insurance for dealing with any monetary losses for Force majeure events like these. What day of the week it is is irrelevant as long as the employer has scheduled the employee to come work at that time as per the contract.
a Lawyer was in the room when they wrote this !
Correction, a shyster was in the room....
Many lawyers _created_ this document with some technical information from the company.
Legal definitely wrote that post.
These dudes thanos snapped half of all windows machines. Amazing.
Actually it was 8.5 million which is less than 1% of all Windows machines.
@@m4rt_ but if they are ones that run major infrastructure that matters, like airports and hospitals, then the it's a hell of a lot more important than all the gaming rigs in the world going down for a few days.
@@CPFilmMarket I dont know anyones pc that was affected, weirdly
@@m4rt_ most Windows machines _don't_ use Crowdstrike Falcon
@@incriptionbecause it's a business anti-virus. Expensive and customize antivirus
Chernobyl happed due to a similar reason. This was a catastrophic system collapse. For Chernobyl - USSR had failed as a model. Here, their business model had experienced the same and they should, just like Boeing, be freakin’ accountable for what they are doing!
Hmm.... What do you think about Fukushima?
@@neony111that was just stick from uncle
@@neony111 well..Fuku was subject to a major natural disaster after all.. that’s a bit extreme.
@@neony111 When an earthquake kills 20,000 people and causes a nuclear disaster that kills 0 people, Fukushima is a non-issue that has been blown out of proportion.
@@alexholker1309 Those who died were fired before being dead, so the stats stay at 0.
For those who lost their home, apparently, it's not a problem, it is just a minor inconvenience. Just rebuild your life elsewhere, simple as changing your pants (/sarcasm).
You'd think in the age of AI and computer automation they would have some sort of testing environment where they ran their new code thru 100s if not 1000s of vms with varying configs for days or weeks before actually releasing updates to the real world.
that won't help if the update mechanism got borked on release day and sent out a file with some null bytes in it instead of actual data. they actually need to run tests on old code with varying bad inputs to ensure it's robust enough to safely reject invalid updates.
That's what AI bros would like you to think.
Does that increase sales? No way we’re paying for that nerd.
"Can't experience allergies if you're not awake" - Benadryl
"Can't experience viruses if your computer never boots" - CrowdStrike
David Plummer at Dave’s Garage channel on TH-cam gives a very detailed explanation of this issue. The main problem was that CS are bypassing the need to have changes certified by ms for the driver by using these channel files. They’re twits.
Two excellent videos from him on this CrowdStrike. Prime should react to those
I can't believe the government prioritized investigating Delta over CrowdStrike. They should investigate both at least, sheesh
Because Crowdstrike were in bed with the DNC vis-a-vis Russiagate
Crowdstrike is being dishoenst. "Changing the logic of the file" presumably means "adding data to the sys file instead of a bunch of zeros". I have the file on my PC disassembled. It's all NULL.
This file got written to all zeros because of the crash
@@velo1337 True, but on next boot the driver still read that file and proceed to dereference null pointers from it. Their product is still an attack vector.
Latest update gives an outline of the testing procedures. They had a but in the validator within the test suite. They will be improving that as well as giving customer controls over the deployment of channel updates. Can't wait for the next vid, content for MONTHS!
It was probably Tom. That's why they didn't test it. Because Tom is a genius.
They did test it. But there was a bug in the tester (their content validator component). See their latest update.
Can't argue with that
Wait. Did someone leave a comment in the code!!!!
@@SixOThree Those fiends. They must be trying to sabotage Tom.
@@kevinmcfarlane2752that's good that they did testing. But they should've also run it on some of their own dummy computers and see what happens before deploying it.
Or maybe they did that too, and it didn't show up because it only affects some computers not all. In that case there's not much that could've been done.
Null bytes in the file doesn't mean it wasn't a null pointer de reference, or a result of a null pointer.
A lot of executable files have null bytes in the for whatever reason. They are just saying that it wasn't related to the null bytes in the file.
That doesn't mean it was a null deref either, but the crash dumps I saw definitely had a null pointer involved somewhere. It could just be general bad memory access, and the bug could be something that causes the flow of execution to go somewhere it shouldn't, and which could explain why sometimes people had really high memory addresses in their memorydumps.
You're right. I pointed out the same in a comments as well.
This. It still appears to be an null pointer dereference. It just has nothing to do with the null filled files.
The file was full of null bytes instead of actual data. Of course it's related.
@@thewhitefalcon8539 what? No it wasn't.
@@thewhitefalcon8539 the null bytes file appears to be an unrelated issue. Possibly a write to disk problem. Files are validated by Crowdstrike to ensure they have a valid header, so null files are not loaded.
Our account manager sent us a "sorry" email, line 1 was blaming microsoft.
I wouldn’t be surprised if it was Microsoft indeed.
If you just broadcast this today, you were reading an old update from when the incident originally started. There's a new one with more detail that came out yesterday
They're not saying it's not a null pointer dereference. They're saying it's not related to 0's within the data file itself, which is also nonsense if the entire file was 0s, since it's the data that caused the error.
@@joeyjo-jojuniorshabadoo6827 I heard that the file had SOME data in it, but all I saw were screenshots were just null bytes, and all were at the start, which was quite weird.
When they said that, I took it as maybe they forgot to put their flat binary header at the start of the file?
not all customers had file of zeros - there are many reasons it was in this state when the crash occured such as pre-allocated before update, post cleanup wipe for security etc. Valid files have a magic byte signature at the start as code disassembly has shown.
What actually happened is that the 'logic error' (ya' don't say) resulted in an incorrect memory buffer allocation from the kernel non-paged memory pool, either too small in size or mis-aligned. Such memory requests are zeroed out by the kernel before returned to caller (there is a background thread in WIndows whose job is to do just this).
A structure field access into the buffer allocated to handle named pipe objects led to a non-paged memory access violation. What's worse, there is a memory allocator validation routine that failed to detect the faulty buffer allocation.
The real question is how the hell did this get past QC - the first rule of testing is eat your own sh*t.
@@tma2001 I mean, did you get the files and look at them in Ghidra, IDA or binja, or is this just what you think happened?
@tma2001 Stop making things up. You can’t allocate non paged memory
You have no clue what you're talking about. A null bytes channel file will boot and be ignored.
This was a buffer overrun.
On the plus side, I got to meet a ton of people at the airport bars on Friday. Got there for my 4AM flight and then hung out at the bar till 10 AM FTW!
It was legit impossible to find a seat in Dallas Fort Worth International after those flights also got delayed 😂
🍻 🎊 🎉 🤮 🍻 🎊 🎉
Susan what do you think is a good way to hook up with someone like you at an airport bar?
@@XeenimChoorch-nx8wx hey bro, I’m a dude, who works computer security so obviously this username is fake
So part one would be, have a vagina, part two is be ready to talk about opsec until your teeth fall out
@@XeenimChoorch-nx8wx Why don't you go outside and touch some grass
@harleyspeedthrust4013 why don’t you? Lmao you can’t even take a joke
I actually meant to post a response but was already back at the bar. I have a client that needs their analysts trained from zero to hero and the daily 3 hour calls are pushing me to smoke and drink like there is no tomorrow.
Susan Powers is clearly a fake name which refers to the sudo command but I assumed y’all were joking anyway so cheers 🍻
As for the grass, the only way to touch that shit is when you put it in the grinder and then roll that shit it up! 💨
The 4chan greentext HBO Chernobyl parody of this was spot on.
I am so dumb that I now realized that C-suite means the organization level where all the positions start with C's like CEO, CTO, CFO and so on.
Core the problem is they loaded unsigned code. This has nothing to do with C and everything to do with exceptionally poor security practices and trusting unsigned code. Anyone blaming C is excusing incompetency or potentially malice.
And windows doesn't even fucking allow you to swap the kernel with an older version (which linux makes super fucking easy to do btw) and just lacks many other security steps which makes this the disaster it was
They fucked up linux a while back as well. @no_name4796
A lot of the AV companies do this because the signing process takes a while and they need to be competitive. If you need to send out unsigned code you have to be extra diligent tho
@@johnfry5710 They can sign their own code at the application layer.
They are interpreting unsigned p-code at ring zero. It's a security nightmare and a massive security hole.
This has nothing to do with MS signing.
@johnfry5710 how does signing take a while? You mean seconds?
The fact they can update kernel level packages/components without resigning makes them now the "target".
Considering that they now have multiple, multiple different jurisdictions' worth of negligence claims to defend against, I'm surprised they even wrote a statement at all! No amount of shiesty clauses will guarantee that a judge in some part of the world won't rip your ToS into ribbons and treat your shareholder value exactly like the very same pot of tasty, delicious money that their golf buddies are salivating for...
Do we know if any courthouse systems were affected by this?
"negligence"? investigate any ToS (a contract by the way, something judges will typically _never_ overrule) and you will find a little word called "liability" ... and it will not sit at the door of Crowdstrike. That doesn't mean they don't care, but the reason they care is that they've hurt their brand, their future customer base, they will lose customers, and they've already lost stock value.
@@xmurrcattx3498 That's incorrect. Many jurisdictions will throw the entire contract into the garbage if it violates a statute. Look into Blue Pencil laws, state of Virginia is an example. Getting an airtight ToS is not that simple. Many states consider blanket indemnification clauses to be against public policy and they disallow them, nullifying all the fine print. CrowdStrike has likely damaged so many people that they will have a hard time avoiding a judgment no matter how carefully their lawyers worded the ToS.
@@xmurrcattx3498 Actually many courts will not allow contractual indemnification for simple negligence. Some will blue pencil the contract but others will throw the whole agreement into the trash. If CS hurt enough people all around the world they will probably face litigation somewhere and the judgment could be very severe.
There'll be at least one jurisdiction with a judge who had their flight delayed by this.
The thing I'm curious about is, was this a failure of the actual channel file, or a failure of the deployment infrastructure. Specifically was there some flaw in the deploying system that corrupted the file during delivery to all the end machines taking the system file update (be it the software on the client end or the server shipping it out). That would still be a testing failure, but a different sort then the channel file itself being bad.
if the cause was file corruption immediately before/during file delivery, then wouldn't that imply that they weren't using checksums to verify data integrity? granted i don't work for a literal multi-dollar conglomerate like crowdstrike, but that, ah, strikes me as a very noobish mistake to make, no?
@@uzbekistanplaystaion4BIOScrek there is no answer I can think of without being a newbie mistake. Fundamental software habits should have caught any mistake many different ways.
That's why this is so mind boggling.
i don’t think there is an “or” here. there has to be at least three or four separate serious flaws for this to happen. every step of the chain could’ve prevented this, but none did
@@uzbekistanplaystaion4BIOScrek depends of how the “checksum” worked, if they used something dodgy enough then it is possible that it passes the test. However what I think is more likely is that checksum is only added at transmission, so it’s entirely possible that the original file checked out fine, then it was transferred to the public server but was corrupted in the process. Their public server then serve up a corrupted file and add a legitimate checksum to it.
Company name checks out
What people aren't considering is that they could have tested it and tested it, and then have the corruption downstream from there. The real negligence is in the fact that they completely bypass everyone staging schemas. They also don't have their code sanity check the inputs.
How is that the loader did not even CRC check the file with super sensitive code in it?
How is that even possible that it died from trying to load invalid file and taken down the whole computer? Isn't a loader running inside the kernel supposed to be as bulletproof as possible? But it dies from a zeroed file?
No automatic staged rollout? Not even a quick stage, like 15 minutes?
They've said this was a scheduled update. They do checksum evaluations. There's an updated post saying they do gradual rollouts, just took an hour to receive an alert and shut it off.
Caused by a bug in some testing code apparently.
How can it be a logic error when there was no logic?
ClownStrike Faultcon
"Crowd strike is what happens when your technical team are just the sales team again"!" 😂😂
Forget testing, just throw some $#!+ together and deploy it quickly to make management happy ... let the customers test it!
... This looks a lot like Microsoft laying off their testers ... and letting the Windows Home Edition users be the testers!
Did you know: George Kutz, current CEO of CrowdStrike, was the CEO of McAfee in 2010 when it rolled out a failed update that BSOD'ed about ~100,000 Windows machines.
Why on earth would 911 use a freaking WINDOWS SERVER?!
How else would you contact 911??
enterprise intranet usually runs on windows servers. especially government. most of them arent connected to the open internet
Active Directory Domain Controllers, internal DNS, DHCP servers, file sharing servers... Also, a lot of corporate systems runs only on Windows. Many corporate applications are only compatible and certified to run on Oracle Database or on Microsoft SQL Server and many companies opt to use SQL Server because licensing is much cheaper and fair than Oracle Licensing. Although you have MS SQL Server for Linux, this version is much more limited, mainly when we talk about High Availability features.
Broke Bradesco (a Brazil Bank)
"Its not a big deal, it`s just a bank".
DUDE, Banks are a monopoly in Brazil. There`s only 5 banks for 200 million Brazilians. 2 of then are public.
At that point they are just covering themself for potential lawsuit
Still prouder to be part of the UNIX family now
Has anybody read their EULA or TOS to check if there's a big fat: "get out of jail card"?
The right judge in the wrong mood could make that not worth the photons lighting it up on your screen.
It doesn't matter. Terms of service are basically meaningless. A company is always liable for gross negligence.
A family friend of ours is a network engineer for a major payroll tech company, which uses Crowdstrike. He was saying they've been an utter nightmare for years.
In Bangladesh we just got internet back after almost a week....
Damn. Did you read any books in a meantime?
@@GreyDeathVaccine I learned HTML canvas (I have llma 3 on my laptop)and now making a app to decode and encode image as a sound beeps so we can can share image without internet over voice call so we can share image if there is another internet blackout....We are passing a extremely critical time in the history of country, according to official source more than 200(though the actual number is much more than that)people mostly university student is killed by the so called security forces Police, Border Guard, RAB even SWAT and finally Army is deployed to kill, detain and torture civilian who are protesting against the autocratic ruler...you can find all horrible image on the international media may even your local media too...
House builder: Well this wall felt because we forgot some nails, but don't worry, this wall won't fall again!
Every enterprise with a reasonable deployment chain would never have had this issue the way it presented. If Crowdstrike implemented any of several standard QA and deployment practices this couldn't have happened. Full Stop.
You're assuming this update came from WSUS/SCCM. I'm 99.99% sure the striker of crowds updates their mess directly. And you're paying for these fast releases.
@@sirius4kthe hitter of the gathered is the enterprise lacking a reasonable deployment chain... Given they did the deployment of the update....
@@sirius4k I am not assuming it came from WSUS/SCCM. On the contrary:
I am assuming they don't have adequate quality gates in their release process. If they did one almost trivial thing that is standard practice, then this wouldn't have happened. It is the easiest thing in the world: stage the production deployment first to a set of machines you control internally. Then stage it to sponsor users (users who have opted into jank), then to everybody at a rate of a few percent per hour. If anything breaks at any point to stop the deployment. All of that can be done at a pretty accelerated pace within a few hours, is standard practice in most parts of the industry, and would have completely eliminated the problem or reduced it in scope to a handful of their most loyal customers.
They deployed to the world and then tried to control-z it 45 minutes later, and then made a suprised pikachu face that that doesn't help for changes that brick your remote software update.
@@timseguine2 Reading your reply, it looks like there's a communication error.
"Every enterprise with a reasonable deployment chain would never have had this issue the way it presented."
"Every enterprise" (plural) - I assumed you were talking about impacted corporations and such and that they didn't have adequate controls in place to mitigate or avoid this disaster.
You're talking about Crowdstrike 😛
I'm not here to defend Crowdstrike. Fuck 'em and fuck every company that chooses velocity/quantity over quality just to keep deadlines. Companies need to go back releasing working products, not deadlines.
@@sirius4k Agreed. I figured my statement was obvious, but plenty of people expressed similar views to the one you thought I was expressing, so I get where that came from. I have been most frustrated by people using this as an excuse to dunk on Microsoft or on agile, even though both of those things are completely irrelevant in this case.
I added a sentence to my original post to make my intention more clear.
Engineers don’t make the final decision. I was on a team where we could tell the director until we were blue in the face that something was going to be a problem. The director made the decision anyway and threw all the developers under the bus for not being persuasive enough to change her mind. Lots of horror stories from that place. The mantra was “who’s responsible” and “The developers always wrong”
Channel File 291 = Order 66
The sum of 2 + 9 + 1 = 6 + 6
I worked at a company where we had a public statement of "a subet of users were impacted" when we were 100% down.
As a OSCP Red Team i think Its quite simple, Windows should not be used in laboratories, hospitals, or any other place where a high level of security is required. It's not just CrowdStrike's fault, it's also Windows' fault for having the need to require the use of third-party applications for its operation.
Linux has kernel level drivers written by 3rd party sources as well. I'm confused, I don't believe this is different in other operating systems.
So, they either skipped testing or don't test at all, if it's strictly a deployment issue they don't sign or checksum anything, and they don't have any error handling in the kernel driver for what are external modules added via automatic updates. Also, they don't do canary deployments, they just hit everyone at once. Test in prod, deploy on Friday. Oh, and they did the same to Linux users 3 months ago.
And we're supposed to trust these guys to write good, safe, closed-source kernel modules?
quick question: how many CIO/CISO chose CrowdStrike voluntarily .vs. compelled by 3rd party (auditors? consultants? insurers?) to implement CrowdStrike to satisfy external demands?
The main reason this happened was that crowdstrike implemented RTO policy back in july 2023. A bunch of good quality QA engineers, SWE and devOps engineers left the company due to the policy. The ones that stayed were even laid off when they insisted not going back to the office. These companies is now paying back all the money they have lost here from the money they have gained from the commercial real estate investments :D
George Kurtz : Because we're special. It's the BLUE SCREEN OMITTED DEATH.
C-suite exec job 'responsibilities':
-never accept blame
-overpromise and ignore warnings of risk coming from people who actually do the work
-push the immediate bottom line over all else including the long-term integrity of your product(CrowdStrike's decision-makers are likely getting a raise from this somehow!)
-use layoffs as a way to divert blame off your own greedy decision-making
"Sir, how do we do testing after we fired the existing debugging team?"
"Easy, just use our customers, they are our guinea pigs."
Part of the issue with companies like that is that from the C Suite down, they only want business people under them. When you have nothing but MBA's all the way down to engineering in a tech company, that is a disaster waiting to happen. Maybe there was a conversation at the tech level about better testing and rollout and the front office guy, not understanding technology, just said, "We would nee to hire more and if we do that, there goes my bonus, so that's not happening." Or even worse, "we cut down the release time by removing guard rails on the process because an Excel cowboy found out it would save money and look good by being faster to market." Not have enough tech people in tech companies has been a huge issue for years.
Dave’s garage gives the best explanation
Dave is the OG
CrowdStrike exposed themselves with the 291 that "we can crash your system without any MS Certifications because its not the driver but a channel file which we all need to fuck up your entire corporate system."
Also worth noting the same thing happened to Linux distros a few months ago that didn't get quite the same attention. And it was a similar issue.
If it got the same attention the Linux fanboys would blame C and push Rust they're the exact same as the opposite side
It‘s like the Volkswagen disaster all over again…
„We are doing everything by the rules, right guys? Wait what, we’re not? Everybody here knows, we‘re not? I‘m not supposed to be telling anyone? Alright, business as usual…“
„What on earth could possibly make this go wrong all of a sudden?“
I can't understand how anyone in the comments is attempting to justify this.
Discussions surrounding anti-cheat services are not relevant for enterprise managed systems; not to mention the fact that anti-cheat software didn't cause this. This is one of the worst national software issues the US has seen in decades. For a valuable security firm to make a mistake of this magnitude is so completely unacceptable, it could reasonably put them out of business.
As a cyber security professional using CrowdStrike Falcon daily at work; I don't get the big fuss. Yeah, they messed up immensely by having one or more change management controls fail (obviously). But how they are portraying it publicly is a bit weird to get hung up on. Every single business in the whole wide world would communicate this the exact same way: Dispel any fear, be transparent about what actually happened, and communicate the next steps. By saying there is no risk that devices that were not impacted by this, will be impacted by this, is a factually true statement, as they won't push the same faulty update again - that'd be madness. At the same time, it'd be madness if CRWD were to say "There is no risk of future Falcon updates leading to a BSOD," because there's no way anyone can guarantee that.
a robust operating system could easily ensure it never blue screens. for years unix systems could boast that no user land program could cause the operating system to crash. and it was an accurate boast. i've seen sun4 machines running for months. why does a security mitigation tool need to muck around with the kernel? why can't an operating system exist which can allow for layers of sandboxing/rootkits done safely?
@@kennethstauffer9220 That may be a valid point which attacks the OS, not applications running on said OS. Thus, in the discussion of CRWD, not completely relevant.
Clearly there's a bug in the driver itself in that it doesn't do any proper validation of its input from the channel files. Then there's something that corrupted the channel files. Then there's the utter failure to test before deploying to end users. Lawyers have to be rubbing their hands together with dollar signs in their eyes because this is clearly gross negligence which means they can throw out the company's EULA disclaimers.
Any sufficiently advanced incompetence is indistinguishable from malice.
I see you what you did here 🙂
The most charitable read is that there was a problem during the file distribution, maybe a corruption in their CDN upload. But the lack of at least checksum validation on the endpoint software is unacceptable.
8:18 this CEO is also a FOUNDER, he's been there all this time. He was CTO at McAfee, so he should know technology, not just business, so he should know what technical means they could deploy to prevent this and should have been in place soon after starting the company. Whatever they are doing, the process probably stayed the same since the beginning.
That one “c++ expert dei causes it” guy who spread the null pointer bs all over Twitter is doubling down despite the objective fact that it was not a null pointer…….
Since this was not an obscure problem that affected just some customers; it is clear that they did no testing of the update before globally deploying it. With a vendor like this, who needs malware?
To be fair... It's in the name : crowdstrike
I work for a large organization in their testing and maintenance department. I manage thousands upon thousands of bare metal machines (not virtual machines, not containers). If we tested our solutions for our internal customers like Crowdstrike does we'd be in deep trouble. I've said it once but I'll say it again. When I applied to Crowdstrike and got rejected that was my happiest rejection ever.
8:24 that’s why we need a Chief Software Engineer in all C-suites, and a Software Engineering vertical similar to the IT vertical.
There's a good chance that the engineers that said "we're not doing this" were fired before this happened.
Isn't that the CTO?
@@disguysn 100% When I give the solutions that cost more but are more reliable they choose other options lmao. It always bites them back sooner or later.
@@ba8e No. CTO is too broad and likely more IT department focused. Developing and maintaining software systems and all the processes that go along with it requires a specific skill set. It needs to be its own thing.
@@br3nto That's the modern CTO definition. LOL The old school def of CTO was all technology. That was only 10 years ago. Man, how things have changed and got diluted in meaning. A CTO is IT -- that is depressing.
"the issue is not the result of a cyberattack". I disagree, crowstrike IS the cyber attack. You gotta be crazy to install a closed source kernel level driver that gets auto updated into critical infrastructure.
Btw whenever you install triple A multiplayer games, you are installing software which has basically the same level of privilege and can do the fuck it wants with your computer
Just saying
@@no_name4796idk man I dont think im going to be using my gaming computer for running a bank or airport.
So when the next ransomware gang starts attacking critical infrastructure you want your threat protection software provider to release an update, wait for some guy in it to wander in, bother to get the update, check the update on every configuration of hardware in his enterprise, then manually run around to every bit of kit, and install it.
And do this every. Frickin. Day.
It’s what we have at the moment or no security.
@@no_name4796 yep i know. good thing i use linux and i audit my whole system since i use gentoo.
@@angusjohnston7172 yes people should seperate their gaming computer from their office computer. Gaming historicaly always came with nasty stuff attached to them, esepcially in the floppy days. But how many people really know that and they game and work on the same computer?
This - was - CrAzY! I create and maintain CI\CD pipelines and do release management. Rule #1 is you build once, then deploy the SAME build artifact through the testing and validation process. IT DOES NOT GET REBUILT. The same artifact that was tested in the first phase IS THE SAME artifact used in the last testing phase, then it is deployed to Production. No new untested artifact is EVER promoted. Period. Follow this process and your fuck ups are reduced, but not eliminated.
Blue Screen Of Delight
Don’t worry guys this particular error won’t ever happen again cause the numbers increment.
It's time for companies to properly invest in their IT departments and let them handle less intrusive security tools. This approach enables them to test updates before rolling them out system-wide, instead of giving one company access to make kernel-level changes to multiple companies' computers whenever they choose.
Going forward, ALL major companies SHOULD START *THEMSELVES* testing updates FROM Windows and Crowdstrike in DEV environments *BEFORE* PUSHING THESE UPDATES OUT to their own prod environments. What happened on July 19 was Crowdstrike's fault, absolutely. But that's the past. And the lessons should be that we shouldn't put this much faith and trust in Windows or Crowdstrike or other major providers going forward. All IT /network/cybersecurity teams need to test these things out going forward to prevent these future major IT outages.
I believe they did test the channel file internally. Then they put the verified file into a deployment pipeline that does some sort of final checks and deploys everything automatically. During the deployment of this file, the deployment pipeline threw an error resulting in the deployed file being all zeros.
As someone who has had to write these post-mortems, this absolutely SCREAMS "we f***ed up, and it was an embarassingly bad f*** up, and we'd rather piss you off for not telling you what the f*** up was than admit it"
I've had to write these for some very embarassing f*** ups...