Part of the problem is how the updates are automatically deployed to corporations, IT people should be testing these first before letting them be rolled out across entire fleets. I imagine there would be settings on the Crowdstrike client software to instead have the updates pushed out via a schedule which allows time for the corporate IT to test, identify if there are any issues like what we saw. There is a trade off between getting updates out as soon as possible due to newly discovered vulnerabilities and the risks that the updates themselves could introduce.
I had been saying it for the past few days, some of the machines that went down like check-in kiosks and flight information displays has no reason to go down because they shouldn't have needed the security software in the first place if proper system infrastructure was in place, everything today is setup lazily where everything is online and exposed to the internet
Lack of canary releasing, insufficient e2e testing in their deployment pipeline, (if they have one), not checking for self inflicted DOS as part of release procedure. Lack of rollback to old binary images...the list goes on....
Hi Nate, thanks for your video. I think any significant solution to this issue requires first and foremost a culture in which quality of the sytems / software matters. Only after that technical solutions start to get implemented and make sense.
You would think that Windows should be resilient enough to understand that there is something amiss and quarantine or rollback the defective driver in order for it to continue to operate. It generated a log of exactly which file was causing the problem so it should be able to do this.
There's a professor also on TH-cam whose talks I like. Name is Dave Ackley, his concepts of robust-first design come to mind as a solution to the problem of a single fail point affecting the global system of technology.
Tronix is still good for at least another 10 years. We don't have these issues over here. Even the craziest vulnerabilities are found and fixed within two weeks max, so they don't even reach the slow and stable distros. Meanwhile you have no control over Windows - neither over what it does or what data it collects and sends out not just to Microsoft but even third party data collection agencies. So yeah, do the math.
if you look at the price of shares of crowdstrike it does not make any sense. Crowdstrike stock started to drop off a cliff the 1st of July, and fell dramatically for 3 straight weeks *before* the outage of 19th july. The outage itself barely affected their stock price. In fact, 1 week after the outage the price of crowdstrike went back up by quite a bit. uh?
I would (as an application security person) like us to reflect on the idea of minimizing impact (and hence the need for such products as the one in question) by building better software in general for that reason as well.
1. The world didn't break. That is gross exaggeration. 2. It didn't 'brick' computers. That is NOT what that word means. 3. I wouldn't call deployment strategies "system architecture". I also wouldn't call a sound security strategy in a company "system architecture". 4. Doing security 'properly' is way too expensive so yes, most companies must and will outsource. To suggest otherwise would be a bad financial decision. 5. The 'mistake' was NOT the bad update that got pushed out, it was not doing canary deployments. 6. You seem to be arguing against local IT fixing things whilst simultaneously arguing against outsourcing your security. Seems to be a contradiction. 7. Fixing this in the future? Easy, we know how to do it, Crowdstrike chose not to. Many many companies in the future will choose not to, because new features are always going to be more attractive than prevention of possible issues. 8. You don't seem to have heard about canary deployements.
Part of the problem is how the updates are automatically deployed to corporations, IT people should be testing these first before letting them be rolled out across entire fleets.
I imagine there would be settings on the Crowdstrike client software to instead have the updates pushed out via a schedule which allows time for the corporate IT to test, identify if there are any issues like what we saw.
There is a trade off between getting updates out as soon as possible due to newly discovered vulnerabilities and the risks that the updates themselves could introduce.
I had been saying it for the past few days, some of the machines that went down like check-in kiosks and flight information displays has no reason to go down because they shouldn't have needed the security software in the first place if proper system infrastructure was in place, everything today is setup lazily where everything is online and exposed to the internet
Lack of canary releasing, insufficient e2e testing in their deployment pipeline, (if they have one), not checking for self inflicted DOS as part of release procedure. Lack of rollback to old binary images...the list goes on....
Hi Nate, thanks for your video. I think any significant solution to this issue requires first and foremost a culture in which quality of the sytems / software matters. Only after that technical solutions start to get implemented and make sense.
Would love for your future videos to include technical details on security system architecture!
You would think that Windows should be resilient enough to understand that there is something amiss and quarantine or rollback the defective driver in order for it to continue to operate. It generated a log of exactly which file was causing the problem so it should be able to do this.
There's a professor also on TH-cam whose talks I like. Name is Dave Ackley, his concepts of robust-first design come to mind as a solution to the problem of a single fail point affecting the global system of technology.
Tronix is still good for at least another 10 years. We don't have these issues over here. Even the craziest vulnerabilities are found and fixed within two weeks max, so they don't even reach the slow and stable distros. Meanwhile you have no control over Windows - neither over what it does or what data it collects and sends out not just to Microsoft but even third party data collection agencies. So yeah, do the math.
if you look at the price of shares of crowdstrike it does not make any sense. Crowdstrike stock started to drop off a cliff the 1st of July, and fell dramatically for 3 straight weeks *before* the outage of 19th july. The outage itself barely affected their stock price. In fact, 1 week after the outage the price of crowdstrike went back up by quite a bit. uh?
IMO the concept of allowing a 3rd party to get anywhere near your OS kernel is flawed from the get go.
I would (as an application security person) like us to reflect on the idea of minimizing impact (and hence the need for such products as the one in question) by building better software in general for that reason as well.
yep
...you can 'transfer risk" but you can't transfer 'uncertainty!' ...lol, tic, toc,
It didn't break the world , it only affected companies who were foolish enough to use windows for mission critical applications ...
1. The world didn't break. That is gross exaggeration.
2. It didn't 'brick' computers. That is NOT what that word means.
3. I wouldn't call deployment strategies "system architecture". I also wouldn't call a sound security strategy in a company "system architecture".
4. Doing security 'properly' is way too expensive so yes, most companies must and will outsource. To suggest otherwise would be a bad financial decision.
5. The 'mistake' was NOT the bad update that got pushed out, it was not doing canary deployments.
6. You seem to be arguing against local IT fixing things whilst simultaneously arguing against outsourcing your security. Seems to be a contradiction.
7. Fixing this in the future? Easy, we know how to do it, Crowdstrike chose not to. Many many companies in the future will choose not to, because new features are always going to be more attractive than prevention of possible issues.
8. You don't seem to have heard about canary deployements.