It could have been a gaffe if the reboot step was not included as part of the functional testing process. Kernel level system files are only loaded into memory on startup. Would have loved to be a fly on the wall in those post-mortem meetings with MS and Crowdstrike QA teams
@@JayandSarah I highly doubt that as they would be terminated without unemployment benefits and blacklisted. Activities are logged and audited in several places
Does this guy work for Crowdstrike, why does he diminish a totally avoidable mistake, the biggest mistake really any related company can make in this sector - sending out untested code ?!
I heard that the updated .sys file was empty, that is contained only zeros, instead of code and/or data. So the main code would refer to null pointers when addressing this file, immediately leading to clearly uncaught exceptions. So stupid mistake + previous sloppy coding.
It was a 41K data file. The point here is that their software assumed that the internal data was 100% correct. This lack of data integrity checking is wholly unacceptable and the reason why is plain to see. You can't run a ring 0 device driver with input data from a file that has not undergone range/validity checking. This is very much like the buffer overrun issues that has plagued MS for decades. It is a sign that the people writing the code are llazy, or amateurs, no matter how loudly they may protest the obvious.
@@AmericaAndAllieswhere was the testing? Surely any non 8 years old development team would have run their software on a test bench to see if it know actually booted up
@@gnezlukc We can rest assured there was no integrity checking nor was the software even tested. Why no testing? Probably because it required a pittance to pay a few IT guys and that was a pittance too much for management who are raking in millions. IMHO you need your head examined if you think these guys are the solution for your company.
I get that its the third party fault, human error, blah..but my god how is the OS still allowing this kinda thing to occur in 2024. I understand in 1998 a bad driver causing bsod..but 2024?? is there no check the OS can make in the boot sequence to isolate that bad driver and boot anyway..why not force a boot to core systems only until that is rectified?? I get the human just needs to do better but man, so many critical systems are relying on that..in this case there was literally no backup..even for the OS sake..everything just failed.
@@ljgil850 Things could be done but we are talking about MS. That's the problem. They run 80% of the world's computers and have been shuffling the UI for 20 years, creating incompatibilities in Office, and generally being worthless, while demanding millions as an unfettered monopoly.
I am a DevOps Engineer and Cisco CCNP Security certified, A channel file is a configuration file used by security software as a service, aka SAS, to manage, implement and update its instructions and protection mechanisms for specific servers. It's basically a file that tells each computer on a server what to do and not do, when certain things happen on a network. It's also instructions to be implemented against it's users when someone does something they are not supposed to do. This channel file also has a "call home" notification for when someone breaches security, it will notify the company responsible for its network security SAS, so they can remotely take security actions against its intruder and protect the network and servers.
The file causing this problem is C-00000291* dott sys It's located in system32 in the driver's folder. Once this file is deleted, the system can be rebooted properly. This can not be done remotely, this can only be done manually. And if a system has Bitlocker activated, you need the security key to unlock Bitlocker encryption for every single end user computer. And if the company is running virtual machines operating systems, You then have to do this for every single virtual machine an end user is using on the network. Reason is that remote access can not be done due to boot looping of the operating system both on hardware systems, as well as the virtual machines operating systems many corporations have on their network.
-I think it was intentional - CrowdStrike wanted to show the world its impact; the update wasn't tested at their level before distributing it to its customers, then the system owners also failed to test the update at their level. -copy cats (nation states) would want to replicate or exploit the vulnerability or vulnerabilities -of all these years, Microsoft hasn't been able to fix the Windows 95 blue screen of death .:) Vs. Linux
You say you need to put hands on each computer to fix it, but, apparently, no one had to put hands on the computers to install it everywhere. That is the crux of the problem. The IT managers responsible for these systems should not be allowing automatic updates to any module which has the potential to harm the system.
That is a simplistic way of looking at it. Most likely, there was routine testing done, but there is a hole somewhere not merely in the writing and testing of software, but perhaps in the distribution of software. Without knowing in detail how the entire process is supposed to work, and where it went wrong, we don't know what needs to be done to prevent it in the future.
@@GH-oi2jf it is a simplistic way of looking at it, I will give you that. However, updates usually trickle down, not on a global scale. And it’s never done close to the weekend, because that’s usually the time hackers are at work.
Courts are closed Social Security is closed. Massive files are lost rendering the computers useless. This is the beginning of global chaos, and we need to be very cautious.
In reviewing comments here and there, I've seen a couple of different pseudo-technical explanations about how this was unavoidable, but I don't buy it.
@@michaelbacchiocchi8111 I wonder how many companies are going to dump CrowdStrike? Nobody can afford systemwide outages like this one. Some companies have automated and laid off staff to the point where they have almost no IT staff left.
We need to have better layers in the core that don't allow updates to corrupt vital services in the windows OS.... OTOH we could switch to Linux.... ;*=[}
He explained that very well, so that the average person could understand.
I hope crowd isn't,t pait when giving interview.
People DIED because of CrowdStrike's gross negligence and CrowdStrike just says "🤷♂sorry fam," absolutely disgusting
No code review , no regression tests, just plain deploy wow 😮
It was a ninja dev
Given how easy the bug could have been detected, it's hard to believe it's not intentional
It could have been a gaffe if the reboot step was not included as part of the functional testing process. Kernel level system files are only loaded into memory on startup.
Would have loved to be a fly on the wall in those post-mortem meetings with MS and Crowdstrike QA teams
I suspect someone in the company did this on purpose.
@@JayandSarah I highly doubt that as they would be terminated without unemployment benefits and blacklisted.
Activities are logged and audited in several places
@@michaelbacchiocchi8111 you think they would care about that? I highly doubt it.
@@JayandSarah yes of course and there would certainly be criminal charges as well. You’d have to be a complete psychopath
How ironic would it be if the Crowdstrike developers asked Azure AI to update the code?
Does this guy work for Crowdstrike, why does he diminish a totally avoidable mistake, the biggest mistake really any related company can make in this sector - sending out untested code ?!
I wonder how many people are erasing Crowd strike from their Resume 😂😂😂
‘SKYNET Is Self Aware’ now
I heard that the updated .sys file was empty, that is contained only zeros, instead of code and/or data. So the main code would refer to null pointers when addressing this file, immediately leading to clearly uncaught exceptions.
So stupid mistake + previous sloppy coding.
It was a 41K data file. The point here is that their software assumed that the internal data was 100% correct. This lack of data integrity checking is wholly unacceptable and the reason why is plain to see. You can't run a ring 0 device driver with input data from a file that has not undergone range/validity checking. This is very much like the buffer overrun issues that has plagued MS for decades. It is a sign that the people writing the code are llazy, or amateurs, no matter how loudly they may protest the obvious.
@@AmericaAndAllieswhere was the testing? Surely any non 8 years old development team would have run their software on a test bench to see if it know actually booted up
@@gnezlukc We can rest assured there was no integrity checking nor was the software even tested. Why no testing? Probably because it required a pittance to pay a few IT guys and that was a pittance too much for management who are raking in millions. IMHO you need your head examined if you think these guys are the solution for your company.
I get that its the third party fault, human error, blah..but my god how is the OS still allowing this kinda thing to occur in 2024. I understand in 1998 a bad driver causing bsod..but 2024?? is there no check the OS can make in the boot sequence to isolate that bad driver and boot anyway..why not force a boot to core systems only until that is rectified?? I get the human just needs to do better but man, so many critical systems are relying on that..in this case there was literally no backup..even for the OS sake..everything just failed.
@@ljgil850 Things could be done but we are talking about MS. That's the problem. They run 80% of the world's computers and have been shuffling the UI for 20 years, creating incompatibilities in Office, and generally being worthless, while demanding millions as an unfettered monopoly.
I am a DevOps Engineer and Cisco CCNP Security certified,
A channel file is a configuration file used by security software as a service, aka SAS, to manage, implement and update its instructions and protection mechanisms for specific servers.
It's basically a file that tells each computer on a server what to do and not do, when certain things happen on a network. It's also instructions to be implemented against
it's users when someone does something they are not supposed to do. This channel file also has a "call home" notification for when someone breaches security,
it will notify the company responsible for its network security SAS, so they can remotely take security actions against its intruder and protect the network and servers.
The file causing this problem is C-00000291* dott sys
It's located in system32 in the driver's folder. Once this file is deleted, the system can be rebooted properly.
This can not be done remotely, this can only be done manually.
And if a system has Bitlocker activated, you need the security key to unlock Bitlocker encryption for
every single end user computer. And if the company is running virtual machines operating systems,
You then have to do this for every single virtual machine an end user is using on the network.
Reason is that remote access can not be done due to boot looping of the operating system both on hardware systems,
as well as the virtual machines operating systems many corporations have on their network.
No backup plan.
There is no shortage of unqualified IT professionals
-I think it was intentional - CrowdStrike wanted to show the world its impact; the update wasn't tested at their level before distributing it to its customers, then the system owners also failed to test the update at their level.
-copy cats (nation states) would want to replicate or exploit the vulnerability or vulnerabilities
-of all these years, Microsoft hasn't been able to fix the Windows 95 blue screen of death .:) Vs. Linux
Such an obvious error is essentially intentional. It is beyond gross negligence. Basic QA would have prevented the problem.
You say you need to put hands on each computer to fix it, but, apparently, no one had to put hands on the computers to install it everywhere. That is the crux of the problem. The IT managers responsible for these systems should not be allowing automatic updates to any module which has the potential to harm the system.
Excellent explanation, thank you!
Who wrote the bad code? Show them!
@@The_Savage_Wombat How ironic would it be if the Crowdstrike developers asked Azure AI to write the code?👩💻
How can anyone say already CrowdStrike was not manipulated from outside?
Because they willingly created the Steel Dossier at the request of the political opponent.
You would think that the update would have been tested first, before it was deployed.
That is a simplistic way of looking at it. Most likely, there was routine testing done, but there is a hole somewhere not merely in the writing and testing of software, but perhaps in the distribution of software. Without knowing in detail how the entire process is supposed to work, and where it went wrong, we don't know what needs to be done to prevent it in the future.
@@GH-oi2jf it is a simplistic way of looking at it, I will give you that. However, updates usually trickle down, not on a global scale. And it’s never done close to the weekend, because that’s usually the time hackers are at work.
@@GH-oi2jf I know what needs to be done: uninstall and never use it again.
I did restore point one day ago. Now pc fine 😊
the prevention software does a lot more damage than the malware itself.
I wonder what this event covered up.
Maybe not run EVERYTHING on one OS? Probably not a good idea.
Sounds like Crowd CHIT is CLUELESS
They probably use AI and just trust what is given by AI. They are so greedy.
Gorgeous bike, if I could afford another, this would be my choice!
Courts are closed Social Security is closed. Massive files are lost rendering the computers useless. This is the beginning of global chaos, and we need to be very cautious.
Massive files aren't lost, they're just temporarily inaccessible.
The problem is that there is only one chain and not a backup!!!!!!!!!!!!!! Duh, to so "called: educated engineers". Can't see the forest for the tree,
Sales in Apple products skyrocketed today
In reviewing comments here and there, I've seen a couple of different pseudo-technical explanations about how this was unavoidable, but I don't buy it.
It could have been an oversight by Crowdstrike QA if they didn’t include a reboot step in their testing
@@michaelbacchiocchi8111 I wonder how many companies are going to dump CrowdStrike? Nobody can afford systemwide outages like this one. Some companies have automated and laid off staff to the point where they have almost no IT staff left.
IT expert everyone, go go go😁
Planned "outage"check stocks and shares. Strange. Check it out yourself. Someone's made a lot of money.
“Experts”
Computers said Me Too 😁
My first thought - what incrimination data was "lost" ??- Kilarys server? Hunters laptop? Pentagun budgets? LOL ;)
Aspen security forum activities not affected
Simulations reviews Testimonies
crowd strike emergency preparedness training
berns, IT...
Crowdstroke and Microshaft in bed together.
You call this guy an expert? 😂😂😂 He's talking like a reporter.
so we're switching to MACs?
No no no MACs are still controlled by Apple and I don't trust Apple either. Linux is the king and you pay nothing!
Who believe this
End times?.. lol
We need to have better layers in the core that don't allow updates to corrupt vital services in the windows OS....
OTOH we could switch to Linux.... ;*=[}
As long as no third party componentes like crowdsafe gets used on Linux. Otherwise can be also vulnerable as it’s written in c/c++
Cheers for owning a MacBook! Keep yourself away from windows based PC's.
No no no MACs are still controlled by Apple and I don't trust Apple either. Linux is the king and you pay nothing!