I'm half convinced that some conscientious engineer knows deep in their heart that this is a bad model, and sent out that update on purpose. What build / deployment process needs a "dd if=/dev/null of=channel_file.sys" step? 🤣
@@mallninja9805 Regardless of motivations ... I surely would not want to be the guy who released that update. A classic case of how NOT to win friends and influence people.
@@WindTurbineSyndrome Crowdstrike have said it wasn't to do with null bytes. So it seems the null bytes were a side effect of whatever was changed in the channel file.
As well as BitLocker recovery key we also needed to use the local admin account as the folder couldn't be accessed without elevated privileges, further compounded by each PC having a unique password. Ten to fifteen minutes of telephone support to talk the user through this.
Weren't those machines part of a Windows Domain? Microsoft teaches that as soon as you have a third machine in your network, you should strongly consider placing them all in a Windows Domain so a Domain Administrator group can manage them all. Ten servers seem to have far exceeded minimal requirements
The first (really the only) question is why CrowdStrike is not using a staged rollout with telemetry to verify that systems being updated remain functional?
@@henson2k that makes me think whether they also haven't outsourced the development to China, for example. If saving, then saving... I can imagine how some other state agent infiltrated and inserted a breaking code, like that XZ exploit earlier. They are working hard on bringing the free world down, and the greedy ignorant free world opens its door wide and invites them in, as long as they are cheap. Saw how Temu took over the western markets?
It been reported the affected file wasn't miscoded or finger troubled, it was basically all zero's. How this could have passed even a cursory QA is incredible to think about.
I have been using GNU/Linux for the last 30 years and everything works perfectly for me, for free. And my valuable data are at home, not in a data center far far away.
As a user of both Windows and Linux, based on personal experience Linux is not completely immune from its own "red screen of death" and horrendous security vulnerabilities. Whatever OS you use, you should educate yourself on its strengths and weaknesses.
@@tonysu8860Also Crowdstrike is installed on Linux servers too. It just so happens the Windows version was affected this time around. It could just have easily have been the Linux version in the news.
@@jakobole Make copies. That's what I do. Stored in separate physical locations(three houses). I have NEVER used a "cloud"(ie: server farm) service, and I never will. I hate "whataboutisms", but this one begs to be said- What if the server farm("cloud") that your data is stored in burns to the ground? Sorry, no. My data stays where I want it to, and in my direct control. Period.
@@SimonBlandford The Windows update is not compatible with Linux. It is a totally different OS and file structure. Crowdstrike would not have tried to apply it to Linux, as it would not even execute. They construct a separate update for Linux(and Mac) that can be executed by Linux only. Those updates were not corrupt.
Most companies would rather contract a company to provide expertise rather than hire someone in house that knows what they're doing. That's why firms like Crowdstrike exist.
*HERE IN BULGARIA* in our cash-based society - nothing changed, we all just want about life as normal. Imagine if your systems went down for 2 weeks - you would all literally be starving
Yesterday was a big win for CrowdStrike. Finally a virus protection program that disabled the most prolific spyware program on the internet - Microsoft Windows. No Linux/Mac products were harmed.
Did they not test their patch before deploying? It's very hard to believe this error wouldn't have shown up in testing. And it's always a really good idea to deploy patches just before the weekend.....said nobody in IT ever. The fail is strong.
Seems the W1nd0ze version of the file was corrupt and not a bug. They pushed out a blank file, not a functioning file. Who knows how they messed up here.
Why test? Getting rid of QA saved them millions. Just inflict it on customers they can test for them. Even better having customers that allow a 3rd party provider who can update their running production systems whenever they (the 3rd party) wants. What a clown show the "tech" industry is.
I agree with that statement. A few months ago they pushed out an update that broke sound drivers and video drivers. The solution was to delete it. After the delete, everything would resume again. As for the video driver issue, you had a window after a restart to log into the workstation or server to delete the software. But ya. Two for two showing me that they are not testing these updates in a sand box or something else. Truely a great F up.
End user devices are going to be the huge issue - looking up all those Bitlocker keys, walking tech or even end user in to deleting the bad file. Hopefully most of the servers have remote management interfaces (different from remote desktop), whereby preboot environments are accessible over a remote connection via VNC and the like.
There are many organisations rethinking their choices as Windows moons them over bitlocker keys while refusing to boot. If they are lucky the path to the internet was a windows machine that also had CrowdStrike installed on it. Fun times. Makes me glad I'm retired.
Good luck if the recovery key is stored either in AD or SCCM and both of those are down due to being on a jump host with windows server that has crowdstrike installed
This could have all been avoided if Windows had an A/B update system like ChromeOS. Having failed to boot the updated partition it would have failed back.
And it HAS ! But Crowdstrike f it up by not building the restore point. In fact, CS failed at so many levels. This company is finished. Not properly tested at the supply chain level. No staged rollout in place. No zero-checksum in place. A kernel file with only zeroes in it should not be accepted as a valid file and yet, it installed itself without any alarm tripping. MS is also to blame for having A/ insufficient way of checking drivers before loading them, B/ unsufficient ways to boot load a faulty system upon a BSOD.
Interesting show. However, I would like to point out that this only impacted companies using CS, not regular home users unless you are using CS. The issue was CS was also installed on Windows Server OS. Also, there was a CS issue with Linux in the past. The issue was bad or not enough testing by CS. Now the bigger question is, was this a test for something to come?? Thanks for the show
At my workplace only the servers were running Crowdstrike, the clients are using Windows Defender and even though a lot of servers went down it was possible to manage the situation. However as I see it the Windows platform is based on a design that was made in the early 1990's and it has been showing its age for some time now. Unfortunately Microsoft is heading into a direction that will make them basically a single point of failure with worldwide impact if they go down since they are heading for a direction where every login and every bitlocker key is in their pocket. It's now almost impossible to set up a local account on your computer and for all new computers the disks are going to by default be encrypted with bitlocker. With that in mind - Microsoft will basically have every company and personal computer held hostage. If you haven't been touching your files for some time they are removed locally and now only exists in the "cloud" held by Microsoft. I can think of some scenarios: 1. Microsoft decides that your computer is obsolete and prevents you from using it. 2. The cloud service of Microsoft gets hacked or goes down for an extended time. 3. Internet is shut off for some reason preventing your computer from connecting to the cloud.
I had a dead motherboard a few years ago, the drives had Bitlocker enabled. As you cannot unlock in a new machine with your regular key, you need the recovery key. Lost all my data. Nowadays I'm on Linux with Luks, it's far better.
All I can say is test your product to destruction, I suspect Cloudstrike don't test at an acceptable level and believe that shortcuts they made in the past testing will be OK to use today. Cloudstrike claims it's not a security issue but is wrong because that will be judged by their customer base, who'll see it as a security issue due to their systems becoming inoperable. Anything that causes a company to lose money is a security issue.
With Microsoft there is no better alternative , but for CS there are a lot of. After spending 9 hours fixing our computers and servers I hope our company will get rid of this spying software
That's not true at all. Linux is better for critical business functions than Windows, and you *can* do everything on Linux that you can do on Windows. Especially in a world where most things are web-based. The only reason people use Windows is because it's what they already know, so it's convenient. But literally half the internet runs on Linux servers, and when was the last time you heard about millions of Linux systems all crashing at once?
The irony is this issue typically only hit organisations who had the awareness that they needed to run and deploy an EDR/XDR solution. Lesser aware IT shops, who did not have the time or resources to stay on top of things, have been spared this disruption....
The sys driver file was all zeros and could bot have contained a valid signature. Why did a sys file without a valid signature even get loaded by a supposedly top security company? Isn’t that a security risk? This makes me question their whole overhyped approach to security.
Crowdstrike is basically a backdoor into your PC. It is a virus disguised as an antivirus. This incident exposed the vulnerability of CS to the world and now all hackers will be looking to exploit it to hack into companies stupid enough to not change their antivirus software to something else (literally anything else would be better at this point).
Starting Monday (if not already) one or more people at CrowdStrike will be looking for jobs in another field! I wonder if the Walmart in Austin TX is hiring greeters?
Now every computer that's down needs a tech to restart it with high-level verification who is validating that these people are doing their job right and not more concerning information in this update when they remove the file
The definition file was just a chunk of null characters. I wouldn't be surprised if the Azure outage was the original cause of the null file being distributed in the first place.
Why did the update not roll out in a more controlled way with verifications-/validations from both customer's IT-staff and Crowdstrike? Before going in production.
people just need to anticipate this sort of stuff... maybe keep some domain controllers physical with a different AV than CrowdStrike... some paper runbooks in safes etc.
IT people should see this as an opportunity to earn some extra cash. No doubt, many organizations will be hiring temporary workers to crawl from keyboard-to-keyboard.
Hopefully this will force companies to wake up and switch to Linux. You do not need a full Windows PC just to drive a self checkout or a departure board! There are $5 micro controllers that can do this.
This outage is roughly what Y2K could have been like in terms of social impact if we hadn't prepared for it. Mostly different technologies at this point, of course.
It really annoys me when people way it was a big fuss over nothing. NO millions of people did a lot of work to ensure it was a nothing. Ferry Bridge power station in the US shut down near me, they must have missed some sensors of something.
@@piccalillipit9211 I don't see anyone on this thread saying Y2K was a big fuss over nothing. Though I guess this outage will mainly affect critical services through the weekend at the worst, where Y2K, with no action, probably would have taken months to repair badly vs. the solid work we all did over a few years with a lot more analysis and planning.
If you use windows you have to be ready for anything. There was no code that broke crowdstrike. They release an update filled with null pointers. It was effectively empty lol
As a Windows user that doesn't enable useless crap like Bitlocker or force my clients to use these clapped out "security" solutions, relying instead on standard security software and a LOT of education... You know what you can do with your smugness.
now they should look hard into diversifying their security AV software having 2 or 3 different vendors in their infrastructure so that if you have 2 different companies at least it would be 50% affected or with 3 vendors only 33% of your infrasture would be affected....the same can be said of relying on one OS like Windows.
@Krypto121yes it is. If for nothing else because they didn't properly vet a bootstrap driver in CS. It's extremely dangerous and either gross incompetence or it was done on purpose.
I'm already in favor of backing off from the cloud. I worked for a company that first had all their servers in house, then went to leasing servers via Rackspace, then moved fully to the cloud. And then the trouble started. Performance wasn't as promised, prices went up and up and up, they didn't meet their guaranteed uptime commitments (they paid us big penalties according to our contract with them, but we had to pay even bigger penalties to our customers for not meeting their SLAs). For that company the sweet spot was definitely when they leased servers but they were their machines. They ran only the company's applications, the company's IT folk maintained them.
It would not surprise me if Microsoft start to quietly sunset Windows Server. My guess is that behind the scenes 95% of their fabric is running on linux variants anyway.
It's absolutely Microsoft fault and it is Crowdstrike fault, they both created this situation and I don't hear anything about compensation for the mess they've created
how the fuck is this MS fault? The only way I can see you can blame MS for anything, is if you blame them for allowing software to start in the boot-up environment, but that is actually needed for this type of software...
@@MrSmokinDragon Are you serious? OS should check driver before executing, if it failed it should be disabled after reboot. Not a rocket science really
@@henson2k yeah the sys file was literally zeros! no dll signature, certificate etc ... just zip all. At least Windows could have restarted in safe mode instead of ending up in a boot loop.
Well, Crowdstrike also did similar snafu on Linux many moons ago when they didn't test against Debian stable which is part of their supported platform. The botched update also caused a crash.
Its not really an issue of the OS, its an issue of testing. If for example Redhat produced a broken update and everyone just installed it on their system with no testing the exact same coyudl happen. From whats been reported it seems CS just send these updates out and companies just install them without doing any testing. WTF? Wasnt like that in my day in fact it was a massive pain getting new software out into production in many of the companies i looked after. And now it appears code is just chucked into production.
Space X and Tesla were completely unaffected. Elon Musk steadfastly refused to do business with CrowdStrike, even though it is in his preferred city of Austin. So, companies with DEI departments should absolutely love CrowdStrike! After all, they must support their fellow DEI hires in all that they destroy.
Crowd strike implemented a scheme similar to other cyber security software vendors but that doesn't make it right These cybersecurity software applications are overprivileged on the operating system operating at a kernel level... Extremely dangerous and it is malpractice to have allowed them to be installed on all these systems or any similar cybersecurity software; this stuff needs to be isolated
It you get a flat tire, you get to a halt too, people....if you don't have the lug nut key, you can't change the tire , same as not having the BitLocker key😂😂😂😂
If an OTA update can derail the global economy, it's clear how dependent we are on software. It don't matter if people use Windows or Mac. We know China has zero day exploits waiting on Mac and Windows to deploy at their choosing. All of our food and water is dependent on code and software. Everyone should just start thinking about food and water for 30 days at home at the very least.
Software is political. The Linux stock exchange servers are purring. All operating systems are the same right? My custom Linux desktops 20+ years rock solid.
Companies will forget about this in a few weeks. It is by far the best security solution out there. All the others just aren’t as robust with preventing breaches. The same exact issue happened with McAfee a decade ago so it just happens when things move so quickly
@@byrnemeister2008 not really. There was a lot more to it and he worked on the vulnerability management aspect of the business not the endpoint products.
once again we miss the forest for the trees or is that the sky for the clouds ? either way, that darned BSOD is what has been causing much mess. why did it comeback again. guess who created it and why its back on us again. and after this would you still get yourselves stuck on the same ole OS ?
That was a rather low tech talk, you do realize you could do things like not releasing updates to all system as the same time. When the hell did we stop testing on the purchase level of products? I always used testing environments for new updates and then I never updated all 30000 machines at the same time. This was not a critical emergency update! But I guess some lazy ass tech guy though it was better to be sleeping, most likely because greed didn't want to pay, bit chilling to see even The Register just boothlicking, instead of reporting on what really is the problem, because the "problem" is what get you paid.
Thank you for not putting Microsoft images on the headline image implying that Microsoft had any part in causing this mess, which so many other commentators are (still) doing. Did Crowdstrike do any testing of their builds before pushing them out? Beggars belief... Any IT department that does not know the Recovery Key for the Bitlockered systems they support are incompetent. Managers should answer what contingencies they have for catastrophic failures. Alternative methods of getting access to the Windows System should be second nature to any competent IT technician. A bit of a wake up call. What are they going to do next time?
assuming the MS bootloader did basic sanity checks on the driver file (it was all zeros ffs!) it should of least skipped it or booted up in safe mode rather than in an endless boot loop.
CrowdStrike was the cause of the major outage, not Microsoft. I'm sure legal departments will be eyeing up CrowdStrike over the coming weeks I'm certain
@@amyskippy I think otherwise. Since customers have purchased their OS from Microsoft - they can file a case against Microsoft. Then Microsoft in turn can sue their own vendor ( crowdStrike). Since customers don’t have any direct contract with CrowdStrike, they can’t sue it directly.
@@pragatirpatra Wait, you seem to be missing a key piece of information, CrowdStrike is a paid for enterprise security platform. Every system affected has a paid contract with CrowdStrike. So yes, they will be able to sue CrowdStrike, because it's completely CrowdStrike's problem. This isn't an example of Microsoft deploying a broken update, this is a broken update from a third party service provider. There's a reason it's only affecting corporate systems and not home computers, because it's only used for protecting corporate networks!
@@pragatirpatra Not really, that's like fans of Cyberpunk 2077 seeing it not working on PS4, and when CDProjektRed pulled it from PlayStation, deciding to sue Sony for it instead becuase they "purchased their PlayStation from Sony". Someone doesn't seem to understand law, or development and it shows.
@@allangibson8494 Not familiar with Mac but Linux is known for promoting that they don't need antivirus, for how their system works, if we are talking about OS, not a server. You need one when you start running MS software on Linux. If MS software is so good on Windows, why people use third party security packages, paying fortunes?
The Solution is using cloud virtualized desktops that could easily be restored, this has been the standard four servers for a long time, its time to virtualize everything
Until whoever is supplying the cloud virtualized desktops gets arrogant and thinks they can cut costs and omits one or two checks because they are, after all, redundant.
![[DRIP] ‘Love, Maybe’ PREVIEW](http://i.ytimg.com/vi/U7hwHp95m1k/mqdefault.jpg)
Consider the delicious irony of a "security company" bricking half the computers in the world.
A bricked system is invulnerable.
@@foobar476
True ... But also pretty useless.
I'm half convinced that some conscientious engineer knows deep in their heart that this is a bad model, and sent out that update on purpose. What build / deployment process needs a "dd if=/dev/null of=channel_file.sys" step? 🤣
@@mallninja9805
Regardless of motivations ... I surely would not want to be the guy who released that update.
A classic case of how NOT to win friends and influence people.
Mcafee and other antivirus software has done something similar.. Bricked the PC and requires manual fix.
Who could have imagined that installing a 3rd party rootkit on every device is bad? Shocking 🙃
Re examine the Trump and Zelensky phone call transcript. Number 45 asked him to look for the server ad to l99k into Crowdstrike
lmao
The Crowd strike update doesn't fit the definition of a rootkit.
Rootkits are difficult or even impossible to detect by ordinary means.
The rootkit was all zeros! Complete lunacy to release it.
@@WindTurbineSyndrome Crowdstrike have said it wasn't to do with null bytes. So it seems the null bytes were a side effect of whatever was changed in the channel file.
As well as BitLocker recovery key we also needed to use the local admin account as the folder couldn't be accessed without elevated privileges, further compounded by each PC having a unique password. Ten to fifteen minutes of telephone support to talk the user through this.
You can bypass local admin by booting from a Windows boot disk.
Weren't those machines part of a Windows Domain? Microsoft teaches that as soon as you have a third machine in your network, you should strongly consider placing them all in a Windows Domain so a Domain Administrator group can manage them all. Ten servers seem to have far exceeded minimal requirements
@@Cyanide300 If you have one. Corporate users won't have them.
hope the PC having a unique password means you have LAPS and so you could just search for the pwd in AD?
"Hotline, we have a massive problem!!" -
"Ok shoot!" -
"We use Windows 11 servers and..." -
"Well, you said that already."
This patchpocalypse is the best advertisement for Linux EVVAAAARRR
Thank you Crowdstrike, I love you 🥰
The first (really the only) question is why CrowdStrike is not using a staged rollout with telemetry to verify that systems being updated remain functional?
cost saving
@@henson2k that makes me think whether they also haven't outsourced the development to China, for example. If saving, then saving... I can imagine how some other state agent infiltrated and inserted a breaking code, like that XZ exploit earlier. They are working hard on bringing the free world down, and the greedy ignorant free world opens its door wide and invites them in, as long as they are cheap. Saw how Temu took over the western markets?
Exactly, blind updating will leave you in the dark...
It's much cheaper.
It been reported the affected file wasn't miscoded or finger troubled, it was basically all zero's. How this could have passed even a cursory QA is incredible to think about.
I have been using GNU/Linux for the last 30 years and everything works perfectly for me, for free. And my valuable data are at home, not in a data center far far away.
As a user of both Windows and Linux, based on personal experience Linux is not completely immune from its own "red screen of death" and horrendous security vulnerabilities.
Whatever OS you use, you should educate yourself on its strengths and weaknesses.
And it your home burns to the ground?
@@tonysu8860Also Crowdstrike is installed on Linux servers too. It just so happens the Windows version was affected this time around. It could just have easily have been the Linux version in the news.
@@jakobole Make copies. That's what I do. Stored in separate physical locations(three houses). I have NEVER used a "cloud"(ie: server farm) service, and I never will. I hate "whataboutisms", but this one begs to be said- What if the server farm("cloud") that your data is stored in burns to the ground? Sorry, no. My data stays where I want it to, and in my direct control. Period.
@@SimonBlandford The Windows update is not compatible with Linux. It is a totally different OS and file structure. Crowdstrike would not have tried to apply it to Linux, as it would not even execute. They construct a separate update for Linux(and Mac) that can be executed by Linux only. Those updates were not corrupt.
Yes, centralisation of capabilities to one company is dangerous
I'd like to see you discuss whether a modern up to date windows server, configured correctly, actually needs crowdstrike or similar products.
Most companies would rather contract a company to provide expertise rather than hire someone in house that knows what they're doing.
That's why firms like Crowdstrike exist.
Astronaut: Houston, we have a problem.
Base: ah, hold on, we have blue screen on all our computers!
This news channel is the only one being realistic about the recovery time.
Other news channels thinks that the disruption will be fixed in days.
*HERE IN BULGARIA* in our cash-based society - nothing changed, we all just want about life as normal.
Imagine if your systems went down for 2 weeks - you would all literally be starving
In my debit card based sociey in the USA everything was fine, I bought food and the biggest annoyances was outlook ran slow
@@sunnohh Well it wasn't was it - it was OK for you but most of your country crashed. People were unable to pay for medicines they desperately needed.
@@sunnohh You are not the centre of the universe in the US.
Can't think of any place that *won't* take cash. People just choose to use a card because it's convenient.
@@Cyanide300 Where my sister lives in the UK most places wont take cash - which is technically illegal.
Promoted via US Government CISA via events and embedded in training. Also often recommended for federal contractors
Yesterday was a big win for CrowdStrike. Finally a virus protection program that disabled the most prolific spyware program on the internet - Microsoft Windows.
No Linux/Mac products were harmed.
Crowdstrike has software for Macs too. It does an excellent job of forcing frequent reboots.
Android(Linux) wants to know your location.
ChromeOS(also Linux) Left the chat
Did they not test their patch before deploying? It's very hard to believe this error wouldn't have shown up in testing. And it's always a really good idea to deploy patches just before the weekend.....said nobody in IT ever. The fail is strong.
Seems the W1nd0ze version of the file was corrupt and not a bug. They pushed out a blank file, not a functioning file. Who knows how they messed up here.
@@IAT1964 potentially an uncaught build error or a failure to read to transfer to the distribution servers, i.e. errored return code not checked.
@@IAT1964 It may be just damaged compilation of deployment code.
Why test? Getting rid of QA saved them millions. Just inflict it on customers they can test for them. Even better having customers that allow a 3rd party provider who can update their running production systems whenever they (the 3rd party) wants. What a clown show the "tech" industry is.
I agree with that statement. A few months ago they pushed out an update that broke sound drivers and video drivers. The solution was to delete it. After the delete, everything would resume again. As for the video driver issue, you had a window after a restart to log into the workstation or server to delete the software. But ya. Two for two showing me that they are not testing these updates in a sand box or something else. Truely a great F up.
End user devices are going to be the huge issue - looking up all those Bitlocker keys, walking tech or even end user in to deleting the bad file. Hopefully most of the servers have remote management interfaces (different from remote desktop), whereby preboot environments are accessible over a remote connection via VNC and the like.
There are many organisations rethinking their choices as Windows moons them over bitlocker keys while refusing to boot. If they are lucky the path to the internet was a windows machine that also had CrowdStrike installed on it. Fun times. Makes me glad I'm retired.
Good luck if the recovery key is stored either in AD or SCCM and both of those are down due to being on a jump host with windows server that has crowdstrike installed
"Windows 2008 Server not affected". I can wipe the sweat from my brow 🤣🤣
I live in a cabin in the woods like ol' Ted, i was entirely unaffected.
good movie
Watch out for the evil scientists in the cave outside of the force field that has you trapped.
This could have all been avoided if Windows had an A/B update system like ChromeOS. Having failed to boot the updated partition it would have failed back.
And it HAS ! But Crowdstrike f it up by not building the restore point. In fact, CS failed at so many levels. This company is finished.
Not properly tested at the supply chain level. No staged rollout in place. No zero-checksum in place.
A kernel file with only zeroes in it should not be accepted as a valid file and yet, it installed itself without any alarm tripping.
MS is also to blame for having A/ insufficient way of checking drivers before loading them, B/ unsufficient ways to boot load a faulty system upon a BSOD.
Why is bending over for Google any better than bending over for MSFT?
Interesting show. However, I would like to point out that this only impacted companies using CS, not regular home users unless you are using CS. The issue was CS was also installed on Windows Server OS. Also, there was a CS issue with Linux in the past. The issue was bad or not enough testing by CS. Now the bigger question is, was this a test for something to come?? Thanks for the show
At my workplace only the servers were running Crowdstrike, the clients are using Windows Defender and even though a lot of servers went down it was possible to manage the situation.
However as I see it the Windows platform is based on a design that was made in the early 1990's and it has been showing its age for some time now. Unfortunately Microsoft is heading into a direction that will make them basically a single point of failure with worldwide impact if they go down since they are heading for a direction where every login and every bitlocker key is in their pocket. It's now almost impossible to set up a local account on your computer and for all new computers the disks are going to by default be encrypted with bitlocker.
With that in mind - Microsoft will basically have every company and personal computer held hostage. If you haven't been touching your files for some time they are removed locally and now only exists in the "cloud" held by Microsoft.
I can think of some scenarios:
1. Microsoft decides that your computer is obsolete and prevents you from using it.
2. The cloud service of Microsoft gets hacked or goes down for an extended time.
3. Internet is shut off for some reason preventing your computer from connecting to the cloud.
Be still my beating heart El Reg has a YT channel. Instant sub.
Biting the hand that feeds IT!
I wonder if the BOFH is in charge of the channel, or maybe the PFY at least...
I had a dead motherboard a few years ago, the drives had Bitlocker enabled. As you cannot unlock in a new machine with your regular key, you need the recovery key. Lost all my data. Nowadays I'm on Linux with Luks, it's far better.
All I can say is test your product to destruction, I suspect Cloudstrike don't test at an acceptable level and believe that shortcuts they made in the past testing will be OK to use today.
Cloudstrike claims it's not a security issue but is wrong because that will be judged by their customer base, who'll see it as a security issue due to their systems becoming inoperable. Anything that causes a company to lose money is a security issue.
With Microsoft there is no better alternative , but for CS there are a lot of. After spending 9 hours fixing our computers and servers I hope our company will get rid of this spying software
That's not true at all. Linux is better for critical business functions than Windows, and you *can* do everything on Linux that you can do on Windows. Especially in a world where most things are web-based. The only reason people use Windows is because it's what they already know, so it's convenient. But literally half the internet runs on Linux servers, and when was the last time you heard about millions of Linux systems all crashing at once?
@@Cyanide300our next gen firewalls run Linux and remain highly available because of it
So microsoft was bitten by FORCED BLIND UPDATES.. who would have thunk such a thing could go wrong.
The irony is this issue typically only hit organisations who had the awareness that they needed to run and deploy an EDR/XDR solution. Lesser aware IT shops, who did not have the time or resources to stay on top of things, have been spared this disruption....
Irony
The sys driver file was all zeros and could bot have contained a valid signature. Why did a sys file without a valid signature even get loaded by a supposedly top security company? Isn’t that a security risk? This makes me question their whole overhyped approach to security.
Crowdstrike is basically a backdoor into your PC. It is a virus disguised as an antivirus. This incident exposed the vulnerability of CS to the world and now all hackers will be looking to exploit it to hack into companies stupid enough to not change their antivirus software to something else (literally anything else would be better at this point).
Starting Monday (if not already) one or more people at CrowdStrike will be looking for jobs in another field! I wonder if the Walmart in Austin TX is hiring greeters?
You mean receipt checkers
Restore my comment please and thank you
it's not a virus definition. the corrupted file has the extension of .sys and it's loading on a very low level.
CrowdStrike started pushing D.E.I instead of quality.. This is what happens!
Waiting for the law suits to be issued...
I heard Crowdstrike will rebrand as SkyNet.
If you have a server, run a server OS. Which would be Linux. If you run the XBox OS on it, it's not a server, it's a game console.
Now every computer that's down needs a tech to restart it with high-level verification who is validating that these people are doing their job right and not more concerning information in this update when they remove the file
The definition file was just a chunk of null characters. I wouldn't be surprised if the Azure outage was the original cause of the null file being distributed in the first place.
Why did the update not roll out in a more controlled way with verifications-/validations from both customer's IT-staff and Crowdstrike? Before going in production.
Arrogance, they thought they could do no wrong.
people just need to anticipate this sort of stuff... maybe keep some domain controllers physical with a different AV than CrowdStrike... some paper runbooks in safes etc.
IT people should see this as an opportunity to earn some extra cash. No doubt, many organizations will be hiring temporary workers to crawl from keyboard-to-keyboard.
Hopefully this will force companies to wake up and switch to Linux. You do not need a full Windows PC just to drive a self checkout or a departure board! There are $5 micro controllers that can do this.
windows- computer for gamers
This outage is roughly what Y2K could have been like in terms of social impact if we hadn't prepared for it. Mostly different technologies at this point, of course.
It really annoys me when people way it was a big fuss over nothing. NO millions of people did a lot of work to ensure it was a nothing. Ferry Bridge power station in the US shut down near me, they must have missed some sensors of something.
@@piccalillipit9211 we mostly found a lot of pending 2049 bugs...
@@piccalillipit9211 I don't see anyone on this thread saying Y2K was a big fuss over nothing. Though I guess this outage will mainly affect critical services through the weekend at the worst, where Y2K, with no action, probably would have taken months to repair badly vs. the solid work we all did over a few years with a lot more analysis and planning.
and the issue remains, as crowdstrike could press a button and brick all their clients computers again.
Great we should ban it just like Kaspersky based on conjecture and what ifs
If you use windows you have to be ready for anything. There was no code that broke crowdstrike. They release an update filled with null pointers. It was effectively empty lol
Time to get rid of this crapware
*AS A MAC USER* Im in genuine danger of exploding with smugness... 😀
As a Windows user that doesn't enable useless crap like Bitlocker or force my clients to use these clapped out "security" solutions, relying instead on standard security software and a LOT of education...
You know what you can do with your smugness.
@@ondrejsedlak4935 AHHAH that sounds like a LOT of work I do not have to do! I do nothing and I know nothing and it just works.
@@ondrejsedlak4935 Sounds like a lot of work to me - super smug I don't have to do all that
The difference was Mac machines simply can’t do what windows machines do, do. CrowdStrike has a Mac version - it just wasn’t corrupted (this time).
@@allangibson8494MacOS & Linux doesn’t allow apps to have kernel access. MSFT windows does. Who là.😅
The definition file that went out was all nulls. It corrupted at some point in the process.
now they should look hard into diversifying their security AV software having 2 or 3 different vendors in their infrastructure so that if you have 2 different companies at least it would be 50% affected or with 3 vendors only 33% of your infrasture would be affected....the same can be said of relying on one OS like Windows.
Companies will sue Microsoft, their prodict failed to start !
We need universal standards for deploying changes that touch the kernel. And never deploy on a Friday. Internal culture failure.
I agree with them, we should not rely on one OS like Windows or CrowdStrikes.
@Krypto121yes it is. If for nothing else because they didn't properly vet a bootstrap driver in CS. It's extremely dangerous and either gross incompetence or it was done on purpose.
I hope these companies will pay bonus to their it department and buy them cookies every week
Oh, is that what "allow cookies" is for? 😊
The first time the reg showed up on my feed!!!
Makes you question. If we should run everything from the cloud
I'm already in favor of backing off from the cloud. I worked for a company that first had all their servers in house, then went to leasing servers via Rackspace, then moved fully to the cloud. And then the trouble started. Performance wasn't as promised, prices went up and up and up, they didn't meet their guaranteed uptime commitments (they paid us big penalties according to our contract with them, but we had to pay even bigger penalties to our customers for not meeting their SLAs). For that company the sweet spot was definitely when they leased servers but they were their machines. They ran only the company's applications, the company's IT folk maintained them.
For all those saying "switch to linux" just do a quick search and you'll find CS did a similar thing to Debian back in April 2024.
Whitney Webb predicted about a year ago that this would happen.
for people in remote working mode and/or week-end assigment for thei IT dept. might have caused havoc ...
It would not surprise me if Microsoft start to quietly sunset Windows Server. My guess is that behind the scenes 95% of their fabric is running on linux variants anyway.
No bank, no taxi, no coffee, food bank overload, starvation/ the servers are down!😏
i cant take a shower at the truck stop
It's absolutely Microsoft fault and it is Crowdstrike fault, they both created this situation and I don't hear anything about compensation for the mess they've created
Actually it's basically nothing to do with Microsoft.
how the fuck is this MS fault? The only way I can see you can blame MS for anything, is if you blame them for allowing software to start in the boot-up environment, but that is actually needed for this type of software...
@@MrSmokinDragon Are you serious? OS should check driver before executing, if it failed it should be disabled after reboot. Not a rocket science really
@@henson2k yeah the sys file was literally zeros! no dll signature, certificate etc ... just zip all. At least Windows could have restarted in safe mode instead of ending up in a boot loop.
@@lashlarue7924it’s Microsoft fault for not securing their own OS in order to encourage a 3rd party market of anti-virus add-ons.
I think this is more on Microsoft than Crowdstrike. A single definitions file shouldn't bring down the whole OS.
Ummm there is a problem if the driver doesn't validate the definition.
😼 All the not smirking I'm seeing here 😼
That's the trouble when people rely on tech
As a plus two decades Linux user ... all I can do is shake my head. People just doesn't want to learn.
Crowdstrike Falcon Agent can also be installed on Linux and Mac
Well, Crowdstrike also did similar snafu on Linux many moons ago when they didn't test against Debian stable which is part of their supported platform. The botched update also caused a crash.
Its not really an issue of the OS, its an issue of testing. If for example Redhat produced a broken update and everyone just installed it on their system with no testing the exact same coyudl happen. From whats been reported it seems CS just send these updates out and companies just install them without doing any testing.
WTF? Wasnt like that in my day in fact it was a massive pain getting new software out into production in many of the companies i looked after. And now it appears code is just chucked into production.
@Krypto121 it's not even the program it's the "of course I test, I test all my apps in production" mindset.
@Krypto121 The model of "Delegating all critical security to a single centralized root kit" sure doesn't help either
Space X and Tesla were completely unaffected. Elon Musk steadfastly refused to do business with CrowdStrike, even though it is in his preferred city of Austin. So, companies with DEI departments should absolutely love CrowdStrike! After all, they must support their fellow DEI hires in all that they destroy.
It’s more likely a culture problem. Move fast, break things, the tech department gets shorted, or they’re just lazy or arrogant. 😂
Crowd strike implemented a scheme similar to other cyber security software vendors but that doesn't make it right
These cybersecurity software applications are overprivileged on the operating system operating at a kernel level... Extremely dangerous and it is malpractice to have allowed them to be installed on all these systems or any similar cybersecurity software; this stuff needs to be isolated
What are u guys doing st my house taking my pitur showing to world ?
Has no-one ever heard of CHEQUES ?
It you get a flat tire, you get to a halt too, people....if you don't have the lug nut key, you can't change the tire , same as not having the BitLocker key😂😂😂😂
If an OTA update can derail the global economy, it's clear how dependent we are on software. It don't matter if people use Windows or Mac. We know China has zero day exploits waiting on Mac and Windows to deploy at their choosing. All of our food and water is dependent on code and software. Everyone should just start thinking about food and water for 30 days at home at the very least.
wish john mcafee was here to see this. RIP
Software is political. The Linux stock exchange servers are purring. All operating systems are the same right? My custom Linux desktops 20+ years rock solid.
Companies will forget about this in a few weeks. It is by far the best security solution out there. All the others just aren’t as robust with preventing breaches. The same exact issue happened with McAfee a decade ago so it just happens when things move so quickly
Funny that the crowd strike CEO used to work for McAfee and left just after that issue. Strange coincidence.
@@byrnemeister2008 not really. There was a lot more to it and he worked on the vulnerability management aspect of the business not the endpoint products.
How can it be the best yet make a rookie mistake
Real time updates to production systems...hmmmm
So thankful we don't user CS.... We are not bothered by this shit show.
Unbelievable.
Most servers are Linux now
Move to Sophos.
Look at the Root! This is the problem with Software as a Services. Everything is Catastrophic when it's reliant on One system.
Who said it is not a kind of cyber attack!? Or type of hack against cs.. Stop trusting everything the ceo's say
once again we miss the forest for the trees or is that the sky for the clouds ? either way, that darned BSOD is what has been causing much mess. why did it comeback again. guess who created it and why its back on us again. and after this would you still get yourselves stuck on the same ole OS ?
Public Infrastructure = Public Code. Period.
try to delete some antivirus from windows, it’s impossible
That was a rather low tech talk, you do realize you could do things like not releasing updates to all system as the same time. When the hell did we stop testing on the purchase level of products? I always used testing environments for new updates and then I never updated all 30000 machines at the same time. This was not a critical emergency update!
But I guess some lazy ass tech guy though it was better to be sleeping, most likely because greed didn't want to pay, bit chilling to see even The Register just boothlicking, instead of reporting on what really is the problem, because the "problem" is what get you paid.
Should've lasted longer
Thank you for not putting Microsoft images on the headline image implying that Microsoft had any part in causing this mess, which so many other commentators are (still) doing.
Did Crowdstrike do any testing of their builds before pushing them out? Beggars belief...
Any IT department that does not know the Recovery Key for the Bitlockered systems they support are incompetent. Managers should answer what contingencies they have for catastrophic failures. Alternative methods of getting access to the Windows System should be second nature to any competent IT technician.
A bit of a wake up call. What are they going to do next time?
assuming the MS bootloader did basic sanity checks on the driver file (it was all zeros ffs!) it should of least skipped it or booted up in safe mode rather than in an endless boot loop.
have you tried switching off and on again .... 15 times .... 🤣🤣🤣🤣
I wonder if anyone knows how to program?
Probably not
I know someone who can, but ive got to take a shit
Why TF anyone is still using Windows to run critical infrastructure??
Its not a Windows problem. There is no replacement for an Active Directory environment
Have some cash at home.
lol the backgrounds in this video
Why people are not suing Microsoft ? If their software has caused loss to my business, they are responsible for it.
CrowdStrike was the cause of the major outage, not Microsoft. I'm sure legal departments will be eyeing up CrowdStrike over the coming weeks I'm certain
@@amyskippy I think otherwise. Since customers have purchased their OS from Microsoft - they can file a case against Microsoft. Then Microsoft in turn can sue their own vendor ( crowdStrike). Since customers don’t have any direct contract with CrowdStrike, they can’t sue it directly.
@@pragatirpatra Good luck suing Ford if your aftermarket stereo burns your car down. Microsoft aren't responsible for third party software!
@@pragatirpatra Wait, you seem to be missing a key piece of information, CrowdStrike is a paid for enterprise security platform. Every system affected has a paid contract with CrowdStrike. So yes, they will be able to sue CrowdStrike, because it's completely CrowdStrike's problem. This isn't an example of Microsoft deploying a broken update, this is a broken update from a third party service provider. There's a reason it's only affecting corporate systems and not home computers, because it's only used for protecting corporate networks!
@@pragatirpatra Not really, that's like fans of Cyberpunk 2077 seeing it not working on PS4, and when CDProjektRed pulled it from PlayStation, deciding to sue Sony for it instead becuase they "purchased their PlayStation from Sony".
Someone doesn't seem to understand law, or development and it shows.
You can't just go on pretending windows is serious thing. It is only adequate for gaming.
Nobody says it aloud thet modern Windows still needs the 3rt party security packagew with such high privileges. Failure of Microsoft, too.
Linux and Mac have CrowdStrike applications…
Microsoft didn’t. They have their own security app built in.
@@allangibson8494 Not familiar with Mac but Linux is known for promoting that they don't need antivirus, for how their system works, if we are talking about OS, not a server. You need one when you start running MS software on Linux. If MS software is so good on Windows, why people use third party security packages, paying fortunes?
thAt*
The Solution is using cloud virtualized desktops that could easily be restored, this has been the standard four servers for a long time, its time to virtualize everything
Until whoever is supplying the cloud virtualized desktops gets arrogant and thinks they can cut costs and omits one or two checks because they are, after all, redundant.
Crowdstrike should be renamed Clownstrike. Cause everyone who uses this software and produced this software are clowns 😂
Yeah, let's go to digital currency! 🤣
Snowflake will be happy
The COVID 19 of computers
Alex jones was right