Some bad code just broke a billion Windows machines
ฝัง
- เผยแพร่เมื่อ 4 ก.ย. 2024
- Cybersecurity firm Crowdstrike pushed an update that caused millions of Windows computers to enter recovery mode, triggering the blue screen of death. Learn how the Crowdstrike crash happened and its impact around the world.
#programming #cybersecurity #thecodereport #crowdstrike #windows
💬 Chat with Me on Discord
/ discord
🔗 Resources
Crowdstrike Statement www.crowdstrik...
XZ hack • Linux got wrecked by b...
Rabbit hack • Rabbit R1 makes catast...
🔥 Get More Content - Upgrade to PRO
Upgrade at fireship.io/pro
Use code YT25 for 25% off PRO access
🎨 My Editor Settings
- Atom One Dark
- vscode-icons
- Fira Code Font
🔖 Topics Covered
- Crowdstrike failure explained
- How does windows kernel work
- What caused windows computers to crash?
- Cybersecurity fails
"You can't hack a system if the system doesn't work! "
- Cybersecurity intern pushing the update
Crowdstrike should call it "a feature"
SAUL goodman would say that as his lawyer" your honour my client................."
9D chess brother
it was a preemptive strike. check mate, hackermen. 🧠
What makes you think it was an intern?? Could have been a long term full time employee.
The fact that it was an Antivirus that performed the single most successful malware attacks ever is just pure poetry
introducing: McAfee
it's basically spyware anyway
Antivirus always becomes the very thing it swears to destroy
Security Malware ™
Crowdstrike be like: "Fine I'll do it myself"
The alpha move of doing something that would make your stock value crash, but simultaneously freezing the stock market so that it can't.
money printer go brrrrt
market manipulation is sadly a common occurrence in stock trading. its like a casino, the house always wins.
So, stonks?
I wanna know who's shorting it today. I might open up thinkorswim and view the market.
@@genshinF2Playyeah insider trading legal for senators
What the hell is their deployment process where they didn't catch this in testing? Like they ran the update, saw a blue screen of death and went "looks good, ship it"?
They didn’t even run it is what it seems like 🤣
Yep, everyone can make a mistake, but this is total negligence both by the engineer and the company processes. It's not an obscure error that happen on some machines, but one that bricks every single machine, which is enough evidence that nobody did the most basic test before deploying. And then, deploying it to everyone at once instead of doing it by phases. I hope the company gets privately sued and publicly investigated and punished hard.
Never test, never fail is what I always say
meh well it's funny stupid people are switch to linux then it won't happen even though a few months ago the same thing happened to linux🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪🤪
you gotta love the stupidity of that one I can't even make this shit up it's that funny🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣🤣
“It worked in my pc”
Hiring 20 more project managers and 40 extra recruiters should prevent this in the future
I think we need some business consultants to come up with a good strategy too.
And plenty EU regulations and standards, which had to every company, even village solo baker, implement, or else ...
or increase the daily scrum meetings with 200%
don't forget a bunch of scrum master asking if there are any blockers every 10 minutes
And 30 more tech leads😂
My dad: Come on, it's just an internship, what's the worst that could happen?
Me:
balls out of my mouth 😢
Well you, probably, can't top these guys so you're safe. 😂
😂😂😂
Since it’s the summer, 100 percent it was an onboard’ee
I can already see the greentext
>be me
>be 22 year old NEET
>dad wants me to start getting a job because sick of my shit
>thinks I should work in IT because am nerd loser
>gets me internship at large cyber security firm because he heard IT sec gets paid the best
They did so many things wrong. Push to prod on Friday, fired half their QA staff, no rolling updates, everyone who was supposed to check the commit was like "yep this looks good". Massive single point of failure of a company.
Cripple your quality assurance, push mystery code upon everyone with no rolling release.
Sounds mildly orchestrated!
For real... How can you so irresponsible to force push this to all customers at once? I hope upper management is going to pay for this and not the poor junior dev who git pushed the last time
To be fair, cloudflare also didn’t have rolling deployments for security content, since it typically needs to be addressed immediately. It bit them too.
Ceo
@@Neomadra It'll be the dev. That's how the world works.
If this kind of an upgrade can go through unchecked, imagine how easy it would be to pay one rogue engineer to deploy malicious code that can be exploited later at ease.
Actually, it is more probably that there was testing done, but this was a crack nobody noticed.
A couple of years ago, I created a little app to help my brother at his work. He had a colleague who was kind of a mess and she was able to mess the app really bad. My brother was shocked that he couldn't imagine what were the steps this colleague did to mess up the app.
@@sfappetrupavelandrei If this error, that bricked millions of customers, didn't make it past testing then it wasn't tested, just handwaved at.
@@sfappetrupavelandreiit’s pretty damn easy to just have a computer run the update on it and see what happens.
That's why federal government banned Kaspersky AV
That's why open source is the way to go imo. Other independent, knowledable people to check for any malicious snippets of code.
They even named their cybersecurity company to sound like a DDoS attack.
Underrated comment
@@alex_veeindeed
Stop business. Drive breaches.
well played господин путин
Crowdstrike is a cybersecurity firm, do you all feel "secure" now?
I was fired last week from my IT job. Yesterday I woke up to a call from the company asking for help, as apparently they didn't have time to hire someone else capable of fixing it, I politely said that I was busy, hung up the call and laughed my ass off of them
You should ask for 1000 per hour for the task, charge them 6 hrs
Poetry!
😆😆 I am so happy for you 😇🥰
Sign back on with a ridiculous contract.
you plitely hung up and laughed after the call instead? lame. laugh on the call
Everyone: DEV > TEST > PROD
Crowdstrike: PROD
Add a preprod environment as well just in case
I dont always test my code. But when I do, its in production...
Stay oncall my friends.
That bug was just perfect... Just perfect! On the last minus details!😂😂😂
On a Friday........
NA: DEV > TEST > INSERT MALICIOUS CODE > PROD
As an IT manager this really hurt.
Having to require a bitlocker key for 1000’s of laptops was a nightmare.
I bet you have Secure Boot enabled, RAID ON instead of AHCI, Device Encryption enabled, and Bitlocker.
I turned off that nonsense for my end-users three years ago because I imaged with Acronis True Image
and had Malwarebytes then. My end-users were up in 3 minutes each while the other techs were scrambling.
Once, a laptop fell, and the screen was damaged. I moved the storage to another laptop and Bitlocker
appeared. I had to waste time putting it back and finding an HDMI monitor to attach to see how to disable it.
Out of 2000 systems, only 3 had to be reimaged with our custom settings and restore the data due to a
csagent.sys error which would reboot at the login screen or BSOD after POST.
If you're an IT Manager, Why are you running Windows in the first place? Maybe if allow it for the Solitaire players, if they weren't connected to critical systems. In my 37 years in IT, not counting 10 years before dabbling in Apple using hex bc we couldn't afford a compiler. In my professional career, we used Windows on exactly 0 critical systems. Mostly worked in TS and Critical Systems - and they all used Unix then Linux (or close variants), and Oracle. How many Windows systems at DARPA? 0. I designed the Linux Cluster that runs All back-office traffic for ticketing, baggage, transfer services, etc... internationally - no Windows, zero. Only front-office, non-critical systems were Windows - and I Guarantee that system was unaffected. Simple Solution - don't run Windows... If the company insists on running Windows Apps, run them in a VM Windows Emulator on Linux! No problems... Windows had been a poor contender from "Day 0", as a rip-off of Unix and X-Windows/Motif. Get a real CTO/CIO... SAD. Feel sorry for your troubles - I can sympathize, but if you don't correct the base error, Windows, I can't sympathize with your future crashes...
Another win for the "remind me later" to every update gang
crowdstrike updates happen automatically under the hood😂😂
just turn them off with group policy (you need windows 10/11 pro)
@@1ycan-eu9ji I _seriously_ doubt Cloudstrike uses the Windows Update process...
@@1ycan-eu9ji and you have access to group policy of your work laptop??
@@1ycan-eu9ji huh, too bad microsoft forces their home garbage unto every single retail-end desktop and laptop unit
At where I work we didn’t even notice the blue screen of death that affected over a billion windows computers because all of our shit crashing is already normal operating procedure.
lool
Kinda true tho
Are you a GP and do you work for the NHS?
The bluesceen gave my computer some much needed rest. It's been running hot 24/7 since I got hit with ransomware a few weeks ago
Brilliant
This is what Y2K wished it would be
😂😂😂
y2k38 will avenge its dad
gunna tell my kids this was Y2K
@@andrewstewrt2141 I heard its y2k38, and also already working to be mitigated today like y2k was.
@@andrewstewrt2141 haven't seen 32bit for a few years, I expect none in 2038 (overflow of Unix timestamp stored as int32)
I also love the fact that Crowdstrike CEO, Kurtz, was McAfee CEO when something similar happened with Windows XP in 2010
He was the CTO, not CEO.
This taints his portfolio even more. Two of the biggest IT incidents in history are under his name. I'm sure this raises some wary for future investors.
"Sir, the second bluescreen hit the monitors".
I want to be like him somedays
The one guy who uploaded this code is not to blame for this. This was a cascade failure in the testing of said code that points to bad organization.
One can assume that many heads will roll over this. Lawsuits are going to be filed.
@@MakeItMakeSense285 I guess Cloudstrike is done as a company?
@@MakeItMakeSense285 One could assume that, but they would be wrong. Once every responsible party has been identified, they will toss excuses and blame underlings with excrement rolling downhill as always until the person who had the very least to do with it is sent home tarred and feathered while the ones who actually caused the problem go along their merry way.
For large scale disasters, it's never a few employees. The high level management and CEOs are ALWAYS to blame. Mistakes from regular employees are caught by robust systems of checks and balances. If there is no such system in place, it's because the leaders of the company didn't approve the necessary budget for it.
But of course, they are using some low level employee as a scapegoat. Higher ups never assume responsibility.
Yes, this is a release management failure, it never should have been possible to release this code. Where is the canary testing?
As someone who works in IT for a hospital, I went in thinking it was going to be a normal day only to realize the fire I walked into. Spent the entire day just walking around fixing each one.
Same, I also work for a large hospital group. On top we use Azure and that made this even worse for a few reasons, one you can't get to safe mode with the servers. Seems solutions late in day we were able to recover a lot of servers.
Poor guy had to actually work
- This can fixed in 10 minutes .. right? right ?
- ......................................................
Oh man
Ohh, so you actually earned that salary youre being paid, for that day. /s
Please dont take that seriously. I work for a big company and love my IT guy and get along well with him. Not because he ever has to fix anything of mine. But because he cool as hell. I always make sure to save him a cupcake or bagel and send him a Teams whenever we plebs are remembered and thrown some crumbs.
That poor intern…
We need to have an interview with him. What a story to tell to grandchildren.
My average life
@@anj000 "Kids, my feats are greater then any hacker every existed"
Hitler
Sorry
poor employee probably overworked af. one person should never be able to deal this much damage. this company has flawed processes
I feel bad for the poor sod who is gonna get blamed for this
@@popcorn8153when you get summoned at 3 AM as part of emergency response team for this mess recovering devices by hand you wont feel so bad for that guy.
Didn’t they say it was automatic?
@@ValensBellator the employee pushed an automatic update that crashed windows computers. the code didn't just decide to push an update that breaks computers on a whim
This is a bot that copies comments from other people and posts them as their own. The original commenter is @oddy_gg. Report this bot!
Yeah, at this point it won't be long before the wikipedia editors need to update the word "is" to "was" on Crowdstrike's wikipedia page...
Somehow they only lost 11% of their value
@@JStackthe computers are down, wait until they are rebooted
@@JStack 19
@@JStack because shit happens and people get over it.
@@MDxGano Depends if Crowdstrike becomes financially liable for the $ loss to businesses.
Who needs malware with cyber security like this
co-pilot: ooh..human error .. Fxxxxck!
And it sounds like the CTO was the CTO of McAfee back when they broke a bunch of Win XP machines back in 2010. Didn’t learn the lesson it seems like…
Good point that the real issue is not that a human made an error, but that tech leaders everywhere decided the best thing to do is to have a single point of failure.
It's like that old saying, don't put all your eggs in one basket.
what would the alternative be (regarding this point) from the client side?
have it's own security team? because the level of expertise will never reach that of a company who's only goal si security
while spewing out steaming mounds of PowerPoint on avoiding a single point of failure, 'silos' and all the other bad bears of 'technical' sorcery
Single point of failure is natural thing, would you rather have complexity of one system of systems, our would you want to deal with two complexities of two different systems of systems? Do we even have amount of people capable of supporting that work ? and what exactly is that one point of failure ? windows ? crowdstrike? cpu architecture? capitalism?
Actually regulations demand installing crowdstrike, so thank you captain state
It's insane to me how dependent we are on a single company.
Then go ahead and create for us a company for that, Boomer
@@user-km8zb3cj3dare you stupid?
@@user-km8zb3cj3dthey could already just use Linux but prefer to save money on programming their applications.
@@user-km8zb3cj3d I think you missed my point. My point was that this type of monopoly is scary and gives private companies like Microsoft too much power.
@@user-km8zb3cj3d are you stupid?
Meanwhile, in Crowdstrike's QA department, the one guy left after all the layoffs pastes some code into ChatGPT and asks "Hey, bro. Is this code any good?" to which it replies "Yeah, bro. Totally safe to push out to production. You like the color blue, right? No reason."
lol
Lmao
Best comment haha
GPT really taking over then
They have one of those?
Currently surfing youtube as my entire team is fully blocked by this 😂
Those were the days I loved the most when I worked in office
Oo
You still getting paid?
i love it
Watching my old company crying rn.
The bluescreens scared the shit out of me. I was working at 10:30pm cutting over network switches in a data center when my boss' laptop bluescreened. We took a break while a network admin remotely checked the connectivity of the new switches. He called us 15 minutes later saying his laptop bluescreened too and that he got a call from IT Management saying multiple servers are down. We thought the network was getting ransacked because of something we did. Luckily it was just a worldwide outage and not some wild network exploit bluescreening any device that connects via wifi.
lol yeah lucky it was world wide and not just you BAHAHA
"Luckily it was just a worldwide outage."
Translation: "It wasn't me!" :)
@@MrOneeyedpete Yes because if it's the whole world it's not his fault, nor he is the one that will need to fix it.
I can totally feel this situation 😂
😂😂😂😂
My boss and I had just had a conversation the day before about the risks associated with so many industries relying on "best practices" (French for "use what all the CEO's friends are using") for platform and enterprise software decisions.
I’m not in the tech world, but when I hear the term “best practices” I know I’m dealing with an idiot who is trying to sound impressive.
As someone who constantly complains about automatic forced updates, I've never felt so vindicated.
Exactly. These mandatory forced updates in the name of "security" break things more often than fixing them.
I never install updates on my devices on the first day, I do it manually after a few days after reading feedback from other users. This way, I get to know if it's something that's really needed and doesn't brick/slow down my device.
I have never liked automatic updates. Back in the day you could defer everything till when you were ready, so your not done over midway through something important!
YES
Nah I just got saved cause my laptop sucks so much it didn’t even update 💀
As someone who, by default, debloats all Windows crapwares and block all those forced automatic updates, I see this as an absolute W.
"Let's give remote kernel level access to our critical systems to one proprietary nontransparent company as a part of our security"
- the entire fking world apparently
"we need to do cyber security" -"oh that's expensive and complicated" - "look, this company does it for 150 bucks a month"
😂
@@alxk3995 first Microsoft makes old computers obsolete with windows 11 now this makes you wonder what is really going on with Microsoft because this is worse then y2k because while y2k was supposed to make a crash like this happen one it was found and fixed before it could happen were as this happened so you have to seriously wonder was it really an over sight or was it Deliberate🤔🤔🤣🤣🤪🤪😜😜🤪🤪🤣🤣
@@alxk3995Per device
Similar to movie Decisions too
tbf most of the people in charge at these places probably aren't technologically literate enough to realize the potential consequences of doing that. it's like when you go to a hospital or take meds and just trust that the people in charge have your best interest at heart bc you're out of your depth in those fields
My company uses CS. The only reason we could function at all today was because my co-worker (who's working the late shift) noticed his laptop BSOD at midnight, right before going to bed, and sounded the alarm. Thanks to that and our admin working all night they restored most of our servers. I got to help our users.
it always comes down to the human factor, if you get the best and treat them well, they will catch the problem before it sinks you, good well paid employes are the best security a company can have
I'm not in IT, but I watch Fireship and I'm here to help
is your company still going to use CS ?
Whatever happened to using a test environment before rolling out changes to live servers??
you should highlight this to your superior that will most likely reward this gem of an employee and not just get internet social credits here on youtube. seems like the good guy thing to do with what you know (your boss might know but highlighting will secure the bag for your colleague). cheers
this is why you don't push untested and uncontrolled updates to the world. Every single update has to be rigorously tested in-house and to a control group before an even larger test before pushing out the door.
burh, its like telling a person to study well in order to pass the exam....like duhhh! This is literally like secret service laughably missing a line of sight sniper
Kind of like how the government was pushing everyone to get barely tested COVID vaccines that don't keep you from getting or spreading the virus injected into your body. 😂😂😂😂😂
and it can happen now, we now are FORCED to accept ALL updates or block ALL updates, when USED to be able to select parts wand and dont!, they also did this in 2021 for the human body.
you'll never know issues until you push it (skin in the game)
>force automatic updates to prevent zero day attacks
>create worst zero day issue in the history of IT
hrm
Worst zero day issue? First of all this is not a zero day exploit lol second of all worst issue in the history of IT then you have a very short history lmao
Woah...
@@someguy4915 Go ahead, list a worse issue than this
@@ThisIsTheInternet on top of my head, the ILOVEYOU virus was much more damaging.
For this recent one you'd have to have Crowdstrike installed in the first place.
The ILOVEYOU virus only required you to open an email with ILOVEYOU subject line, do all sorts of nasty things upon reboot (deleting/hide files, replacing files with copies of the worm, etc). Then it uses your address book to send the same email to all of your contacts.
Your contacts that trust you, would then be tempted to open it because of the presumed declaration of love in an email.
And upon opening email, the same code would automatically run on your Windows machine.
The fact that the worm was visual basic script file also allowed other hackers to modify it to do more damaging things, change the email subject line, etc.
It was so popularly damaging at the time that it had variants too like the recent pandemic did.
@@someguy4915A 0day only means that they have 0 days to fix it. It says nothing about the severity of the vulnerability itself. A DOS in the wild is still a 0day.
fun fact: modern way of installing kernel/module updates on linux leaves a backup that prevents this exact issue from ever happening
wait rly? that's so cool
@@_tr11 for my installation currently I basically have 4 kernels (latest release with its backup, and LTS with its backup) and you can install as many kernels as you want
Yeah, that called snapshot
Literally has the older kernels available to begin with.
@@_tr11 yep, most distros save the old kernel image when the kernel is updated, and only deletes it when the kernel is updated yet again, and the kernel which replaced it then becomes the backup. So there is always at least two kernel images on the system at any one time, and the old one can be used if the newer one is borked.
The only thing that got me through my 18.5 hour shift today fixing this issue at dozens upon dozens of gas stations was knowing there were thousands of other fellow IT guys and girls across the world holding down the fort with me
Thank you for your service.
o7
I appreciate your work. I hope you get more than a pizza from your boss.
On the frontline holding the tide, massive respect
Nice job security.
Started my new job at CrowdStrike today. Unplugged a socket marked "do not unplug" to charge my phone. A lot of commotion in the office soon after that. No idea what it was all about.
@@dylancounte1448 should I do sec+ or sscp?
You need to attend more scrum meetings.
That's what the director of technology of a public department told me a while ago when I asked him why there weren't using Linux servers instead of paying thousands in licenses: "if we have a failure, we can blame Microsoft or any other company, that's the only purpose of licenses".
When a company switches to Linux for their servers they still pay money, just not thousands but hundreds of thousands (same as Windows, sometimes RHEL is even more expensive).
Licenses are cheap, support is expensive.
Windows support, while sometimes frustrating and slow is fairly good and dedicated.
Linux support is a joke.
That's fine if you're running a homeserver or some small company that cannot afford a $501 license, but for larger companies where the cost of downtime/IT staff hourly cost is more than $501 this makes no sense to run on Linux in most cases.
@@someguy4915that must be why most servers are running linux
@@someguy4915 That is why Microsoft offers Linux on its system. Just to give a shout-out to its competitor for no particular reason (totally not because Linux is prevalent on servers and developers' machines).
@@someguy4915 For a larger company it's still better to use Linux, and there are dedicated distros (some even licensed for way too much money) that will cover all of your corporate needs. Besides which, all your backend is using Linux anyway. If you see Windows, it's more than likely just serving as the interface with which you're engaging the Linux systems under the hood.
@@someguy4915 somebody should tell AWS to shut down all their datacentres!!!
as an IT guy, I literally came in into the office, it was supposed to be an easy day, said let's get some coffee, finish some tickets, and then I log in and I see hundred of messages and a line of people waiting for me. And then it hit me. The ldap server was crashed. We were all fucked.
Ohh dam brother, me too! 😂 thought this was gonna be an easy Friday
Have a good weekend
Cheers from an IT Incident and Major Incident Manager. Days that start before the commute with a lot of messages, alerts, questions are tough. Days that are calm and ends with a lot of messages, alerts, questions beginning during the first steps of the commute back home are also tough. There's a superstition where I work: nobody better utter the statement that today is calm... ;)
Luckily I didn't have to deal with yesterday that much, no Crowdstrike in the company. Only had to check with several suppliers if they were OK too.
Have a good weekend!
Lots of work to do! People depend on us!
**opens youtube**
installing an third party edr on an ldap server 😅
"Oh this isn't a big deal, you can just reboot into safe mode and change the extension!"
Bitlocker: "Hold my beer."
Also add that the average office worker has very.. very little knowledge about computer outside their field. Just adding a printer is a 2 day job
Seeing as there are a lot of non-tech savvy usuals in computer jobs who only know enough to get by.
I do not think they would even know how to access safe mode. Let alone anything else
@@RippanCSGO you can work in it for 20 years, but when it comes to printers all hope is lost, even for the veterans. Id rather chop off my leg, than figure out why a fucking printer is not working.
@Eagle2565 the sooner we figure out a paperless society, the better. You are a man that knows my pain with printers 😂
@@Eagle2565 Printer driver up-to-date (check), PDF file to print in correct format (check), Printing Job in queue (check), Printer ready (check), Paper in the printer (check), Printer full of toner (check), Other PC can print the same document on this MotherFucking printer (check)...
I'm in Japan and was very confused on Friday when the lady at McDonald's was writing my order down on a piece of paper. This explains everything.
Just imagine having your name on that commit, yikes....
imagine being the PR reviewer too lol
You immediately become one of the more well known developers, just not for a good reason.
@@TheIsh3000 THIS is what I came to say, whoever PR'd this, bye bye!
@@_Doskii so does "any publicity is good publicity" apply here?
Shouldn't QA catch this? (I'm a Hardware developer not SW)
I do maintenance on commercial airplanes, mostly in between international flights. Today I have done exactly zero maintenance on a single airplane during my twelve hour shift.
@RyanClone winning would be getting sent home!
"Anyway, none of my computers were affected by this bug."
@@HankTVsuxhow do you get into that? And are all your shifts 12 hours? I have an interest in planes but I also have an interest in free time so I don’t know.
@@pancakeluxury23 I love that question. I got an in through a friend so I was very lucky. The easiest way is to start by throwing bags and start working your way up or to go to school and get your license.
I can't tell if you like the idea of 12s with that comment. I took them specifically because they come with 4 day weekends (fly anywhere for free+4 day weekends. You do the math ;) )! Though, they are pretty rare and I'll be back on my normal schedule after the summer rush.
The industry is absolutely booming right now (my company hired 40,000 people since pandemic) and it's a relatively stable industry with great pay opportunities and an unlimited ladder to climb if you are just reasonably patient.
I work with a lot of folks who started by throwing bags and some are going on to be pilots or whatever else. One guy is dreaming of being a buyer for the company (that is, buying planes. As a job.). Myself, as a mechanic with no aviation or formal mechanical experience will be making six figures in about 6 years.
@@HankTVsux 4 day weekends?? Say less. I mean I know that’s not guaranteed but the reason I ask is because I’m nowhere in life right now and unless I want to flip burgers for the rest of my life I’ve got to choose something that pays enough to actually live life. I also just happen to like planes anyways. I like cars too but I don’t wanna work at a jiffy lube or assembly line for the rest of my life either. I’m 23.
In Mr Robot they spent an entire season just to pull off a hack as damaging like this. Crowd Strike did it in one update.
reality ALWAYS surpasses fiction
Well yeah it’s their name, crowd strike. Population attack just sounds too straight forward.
ROTFFL!
I cant understand how such updates are not sample tested beforehand. It makes absolutely no sense, its basics
Just happened to me today. PMs pushing for deadlines knowing the product isnt even complete
@@bluesteel1 Yeah it can be like that. Couple with layoffs making everyone (except the people up top) overworked and you got a recipe for disaster. Naturally, the people that created this situation gets little blame, if any.
Anti virus running on Kernel Mode: Strike 1
Machine has no way to automatically rollback to last good boot: Strike 2
Deploys updates worldwide simultaneously without production testing: Strike 3
I really think that management knowing nothing about computers at a time computers are key infrastructure is a terrible idea.
Preach
Add some Bitlocker in the mix, and voila, a perfect party
I think you mean QA testing. What they did was definitely production "testing".
@@herp_derpingson What's the difference? All in all the testing being done should amount to it being deployed in a manner that doesn't crash nearly every server across the globe.
@@Efilnikufesin76 large scale uncontrolled production testing /s.
I remember watching a presentation from a software developer who talked, in part, about why he’s so adamant about defining best practices and building quality, tested code. He said that one day, some software bug is going to be pushed out by an exhausted intern, and 10,000 people are going to die. And when that happens , our profession will go through a reckoning. It behooves us to make our own standards now so that we protect ourselves when that disaster finally happens.
I know this isn’t as bad as 10,000 people dying. And I don’t think this is the moment he was talking about. But the fact that so much damage was done by a little mistake really has me believing that he was totally right.
The butcher's bill has yet to come out to report the total damages.
Uncle Bob Martin
considering the scale of this and millions of people wre definitely affected, percentage of which are in the healthcare industry (hospitals, pharmacies, etc.)
its unfortunately already a guarantee that more than 10,000 people has died from this incident... (people who needed urgent care, people who needed certain doses of a particular drug, malfunctioning medical equipments and devices on the middle of a medical procedure like surgery, etc.etc.)
Some people definitely died from this disaster
Open source helps avoid this issue.
i can never understand the fact that there wasn't one person who could just install said update on a test pc to check if maybe "something" broke, before updating the whole world
Both Microsoft and Clownstrike should have tested the update before releasing it.
@@billfarley9015 the hell does Microsoft have to do with Crowdstrike's software?
@@billfarley9015 nah don't blame this on Microsoft. Microsoft can't test every single update to every single kernel driver out there seperately.
you can't blame Microsoft here. There's no way that they are responsible for checking what third parties want to install on their own computers. There are legitimate reasons to install and run malware etc.
Definitely what was supposed to happen someone probably skipped a couple SDLC steps and merged directly to Main
I don’t think I’ve ever seen so few aeroplanes flying over the US ever since 9/11 when the US had to shutdown US Airspace.
Were you comatose for most of 2020?
The fact that they have so many clients using the same antivirus software is a security risk in itself
yeah this pretty much stated that everyone went with what's trendy...now we know for sure alot of the companies will shift to other better equipped competitors
@@odenkaz with all due respect...it's less "what's trendy" and more "if i factor in consistency and cost, what is the best option?" having multiple high profile clients is an implicit selling point, it means your cybersecurity is top notch.
the product itself wasn't an issue, the issue was the CI/CD pipeline was not followed appropriately. An intern or lower level dev would not even have the capacity to push this into production. Normally they would have it in a development branch, then a testing branch, and only after passing testing would it be merged into production by a higher up who reviews the pull request. General protocol for big companies is also to roll out updates over a duration of time - it's why you sometimes have a friend complaining about a new update to an app that you haven't received yet. It isn't supposed to go out to every user instantly, it's supposed to allow time for issues to be caught if something still goes wrong in production.
"Too many people reliant on one app" isn't really a problem.
@@ApexGale it isn't a problem until the one app everyone relies on fails.
if you take the keystone out of an arch, what happens?
This is why competition is important since Russia wasn’t even affected by this
@@ApexGale No it doesnt. It means youre popular. This logical fallacy led to this lol. Play stupid games, win stupid prices.
I had to delete that damn file for 13 of our bank branches. Im tired..
praying for you bro, did you finish?
manually and individually?
Sounds exciting, did you have to drive to all 13 locations? Why don't they have 13 of you?
🫡
I had to delete that file for 1,800 oil change locations by dracing into our servers 😮💨
I work in IT support and today's night I witnessed all stages of emotions - Confused, Shocked, Defeat, Acception, Resignation.
todays night?..aka tonight
@@David-gp3fd Acception?
NO WONDER MY PC CRASHES WHEN I PLAY GAMES THAT DON'T EVEN DO CRASHES OFTEN, also are you ok from this trauma event?
Job security
@@David-gp3fd guy is overworked, give him a break hahah
As someone who was working in their local hospital lab that night this all went down I can tell you it was bad. I distinctly remember fielding a call from one of the lab managers "How far behind are you guys?" "...I don't even know. I barely know what way is up right now. Minimum 2 hours or more behind on every lab result."
What's crazier is somehow the computer I was working on survived and was basically the only working PC in the lab for 3 or 4 hours. No idea how it made it through. It went down a couple times but I was able to get it back up (it was also occasionally randomly closing programs I was in, etc.) and so I at least had something rather than nothing.
My son is one of those IT guys having to manually fix 20 computers at every location nationwide. He had an 18 hour day yesterday and right now is back on the job at 6am fixing more bricked PC's.
I feel sorry for him, we too have been on calls since yesterday, I feel bad for our hosting team, they're working round the clock
Sorry about that.I work in IT and i can tell you this is a nightmare.Having to do overtime(unpaid) and having to power through the weekends because of somebody's errors is a place you don't want to be
@@harrisonnjenga777 unpaid is ridiculous tbh. need an organized strike or something
I've spend the last 2 days fixing manually 700 servers. At least that's paid extra and recuperated.
Huge workload
20 computers at every locatiom n
Is child's play. We had 1000 machines at each location.
Still stuck in an airport 2 days later with no sign of us leaving. This is madness
Trust us, there are millions of people working 18 to 19 hr shifts to resolve this, it just takes time
Well shit that's gonna be me in 4 hours
My flight was delayed 1 hr in the tarmac, but I made it out. Delta.
@@Shadow__133 Wish me luck brother, just getting to the airport 🙏
@diceonamay are you still stuck?
Imagine being the guy responsible for the potentially single bad line of code, sat comfortably in your IT dungeon, you publish the update and then 20 minutes later the world collapses, and 21 minutes later... the phone on your desk rings.
Yeah, just let that call go to voicemail.
Imagine being the guy cooking the bat soup or whatever it was that led to COVID being disseminated. Individuals can still change the world lol
Maybe he was born in the year 2000.
@@aaronlange8756 The web site is down. Can you just... restart the server?
DEI hire
fun fact: In some year 2006 ig, McAfee antivirus got the same security issue which disrupted the windows XP users worldwide for which the CTO is the same as CrowdStrike CEO.
I've said constantly to friends that modern tech just feels like malware that they're calling "updates."
I miss the days when I felt like my smart phone was on my side - tool not a chain.
My man it's not malware it's just an accidental null dereference due to bad code.
@@ProferkWith the control they have over your data and your computer it's basically spyware. And if, before release, they don't even test their software that could and will turn a computer into a big brick if something goes wrong, yeah, malware sounds right for that.
The line between cybersecurity technology and malware is only defined by the motives of its operators...
@@thethoughtfulpeanut6662 or with how little control i have over what microsoft does to my computer nowadays, the line is whether the Law likes it. i wish i could talk to someone and ask why every update erases all my sound driver settings and reinstalls stuff like Cortana.
I can't imagine the amount of *Jira tickets* and *Story Points* launched within ClownStrike right now... PM's and HR can finally justify their salaries even more.
"It's complicated" doesn't fit Dave, you got to choose a number of points.
Not just bad code but a short group of bytes in a very short string of code. Pure genius.
okay, chatgpt
Y'all gonna ignore her pfp?
3:33 - This is a very important point. Most decisions in big corporations are made on a "will this get me fired" basis instead of actual pros/cons considerations. As long as the decision maker gets to keep his/her job by blaming someone else for mishaps, that decision is a good decision.
As someone working for an IT service provider, there's the occasional ticket or even project that feels like this.
It's not that internal IT wouldn't be capable of it but rather they're too cowardly to push the button.
So you're saying there's lots of money to be made by simply having a willing to accept responsibility when something bad happens?
@@Triad72 no. HE saying something else. YOU can mean multiple things. company, government is rule based system. but that does not mean you are bound ONLY by those rules, every lawyer, judge swears to bible not because they are worshippers of god, but because it is a symbol of higher moral principles. being saboteur is easy, yes..
On a certain level there is credence to be given to the idea of "let the specialists handle it" and to the benefit of the doubt I like to think at least some of the organizations affected just aren't capable of running their own cybersecurity. However, for so many of them, including the government, it's a major question of why they don't just invest in having their own in-house team for cybersecurity. It'd be more expensive but they have the resources and it would also keep probably one of the most important jobs in the modern world, in house. Like I really don't love the idea of services as important as 911 being reliant on a third party software provider.
@@chengmunwai sad
This puts every cyber security attack in all history to shame. On the brighter side we just found every enterprise computer's vulnerability.
Hackers would revere that guy who pushed that code as a god.
Wait till you hear about Intel TPM & AMD PSP 😁😁
Also every country's vulnerability. It starts with the letter M.
I spent 12 hours today, starting at midnight, fixing this issue.
I had to boot into safe mode, and delete the new CS update. We had ~3500 machines impacted.
It sucked.
same here brother, my mind is mush after 12+ hours of punching in bitlocker codes 🥴
Damn guys. Thanks to you all
Bless you guys , I hope you get the rest you need after this
Yup 12 hour day. Must be a world record for all global IT/OPs guys accumulating overtime at same time also lol.
@@tbcrosby Jesus, I feel for you dude. Thankfully, none of our clients use Crowdstrike but rather Bitdefender.
remember when this was the plot of a Tom Clancy novel? Pepperidge Farms remembers. Same week the Secret Service attempted an assassination. What a co-inky-dink.
Guess what's next.
You would assume that these companies have lange testing infrastructure..... The fact that you are deploying kernel mode software on half of the worlds computers should justify that!
You really would assume they have at the very least a basic pipeline test that would've instantly caught this and blocked it from getting pushed to release...
Apparently not though...
Were testing in prod with this one
@@youtubeviewer5198😂
...and here I am with entire testing suites for personal hobby projects.
The customer is the test infra.
i woke up today, saw the news, and within 30 seconds thought “theres gonna be a fireship episode on this”. and within 8 hours;
Now we wait for the Kevin Fang documentary
same lol, i didn't even read any articles cause i knew a code report would come out XD
Pretty sure Low Level Learning's gonna drop a video on it as well, especially seeing as cybersecurity fuck-ups like this are kind of his thing.
Also mental outlaw
@@yarpen26 It's here. Lol
I guess everyone is gonna milk this incident.
I work for a transportation company and all of the computers displayed the BSOD following the update. They sent everybody home for the day so IT could reset everything.
W for your company
@@theonlycatonice not at all, they lost money and the IT guys won't take rest for a couple days
100% same situation on my part
Something like that happened in our office a couple of years ago which resorted them to have us take "off" for 3-4 days. Then later, they made us work on our week offs to make up for those 3-4 days. 😢 Sometimes, had to work 9days straight 10hrs shift each day
@@modernmanueee_ We found the Blackrock investor.
The stock actually started rapidly going down about 10 hours before "the incident". They had a valuation the day before which basically told everyone to sell sell sell.
Imagine if all the systems at CrowdStrike are on the blue screen so they can not even fix the code 💀
Edit: How did I get 560 likes in 6 hours
That would have been poetic.
Safe mode booting😊
programmers use linux right? Like I've heard even the devs at microsoft use linux
@@occultsupport they mostly use windows with linux subsystem, meaning it runs windows originally just runs a vm of linux inside it.
This is indeed what happened, you can't fix it without going to safe mode / detaching disk
I'm 90% a linux admin and you'd think this wouldn't affect me, but as my windows admin colleagues saw my shit was all up and running unimpressed by the crowdstrike update, I had to help them out. I typed a lot of bitlocker keys today. And I dare not count how many times windows told me that ls -l and rm aren't valid commands :/
The punishment for competence. You just know if this scenario was reversed, all those windows IT folks would leave the job to you, claiming ignorance of the system.
@@karlzone2 And they wouldn't be wrong either. Windows IT folk dont know a damn thing about Linux, yet, Linux users usually know a thing or two about windows, even if they hate it with a passion.
@@joemann7971we hate it because we know it...
"And I dare not count how many times windows told me that ls -l and rm aren't valid commands :/"
r/PitchForkEmporium
***!!!! Blowout Sale !!!!!***
!!!Get em now before they're sold out!!
@@joemann7971 I know Linux pretty well and hate it with a passion.
Just a Reminder that today is:
'Largest IT outage in history' & 'Happy International Blue Screen Day'.
Hope this makes the day even
Petition to make International Bluescreen Day an actual holiday
⬇
@@under6075 no... pls... we have way too many of them. Not too far is the day all 365 days become holidays.
@@Hmm-p9tactually there are already multiple holidays for all 365 days of the year.
@@under6075 👍Yes!
@@Hmm-p9t there's a holiday for all 365 days of the year
Why isn't there an automated pipeline that installs every new update on a Windows VM and makes sure it doesn't break everything before deploying it?
it wasn't a windows updatr
@@bananerz3167 yes, I'm aware. I didn't say it was
I always thought that Crowd-strike was an odd choice of name for a company that provides security. Now it makes sense.
You mean, they basically did a strike on their whole crowd?
well you see, when the writers of Real Life came up with this plotline for this episode, they had to retcon in the existence of this company. they were on a deadline so they just called it CrowdStrike and called it a day. CrowdStrike didn't actually exist before Thursday, July 19th 2024 (Season 2024 Episode 200). it was simply retroactively added to the timeline for this event specifically.
@@polymetric2614 The symbolism *is* a little blatant, but I appreciate having some new action in the storyline. After the Trump fake-out, I was worried like they were just planning filler and fanservice until that event in November.
@@polymetric2614 As a standalone episode it works quite well so on this occasion I can overlook the retroactive continuity in particular as it doesn't alter what was canon.
"Crowdstrike confirmed! ETA 10 seconds" xD
Can confirm, all 1600 of us IT employees had to go fix all of the desktops in the company manually today
I work for IBM, you should have seen the army of IT and programmer rushing in at 3AM , it was impressive.
Hats off to you
@@eze3922 who doesn't love a party at 3am 🥰
It certainly was a shitshow to wake up to at 4 AM lol
@eze3922 Why would IBM run windows when you literally invented the mainframe 🤦♂️
Everything has become so centralized that if just one thing breaks down, everything breaks down.
no i think if my mouse broke, my computer would be fine, and so would most windows systems. it feels, to me, like things only start breaking when important shit breaks. not like, anything.
@@tonoornottono "Wow, you don't eat your chippies, the world will end!!" is not the same as "Wow, the potatoes got blight, but who cares as we just grow potatoes of the same variety in the whole island", is not about 'anything' breaking but that our society is making overextended supply lines, if China takes over Taiwan and stop the flow of microchips you will get a big fukin problem with chips not getting chipped for your new car and get a lot of industries paralyzed, we see it just how it happened under 2020 Wuhan Virus, and that was just a minor inconvenience on cargo ships in comparison with an armed overtake of the TaiPei Province by the PLA.
@@tonoornottono nerd
@@Main_Protagonist dude i think their point is meaningless. like what are they ACTUALLY saying? is it true? i don’t think they’re saying much of anything, and the little meaning i can squeeze from it is untrue anyways. like maybe you can say im being pedantic, which i think is your actual point, but that would require me to be twisting or misinterpreting their point and i just don’t think i am. it’s shallow.
@@tonoornottono Nerd
world's most famous null pointer dereference
this is exactly the reason that centralization and consolidation of our network infrastructure is a bad bad bad idea in the long run.
makes sense if all you care about is to save cost and make money, it's very bad for world wide computer systems
But then a single guy can’t be a billionaire for owning 1000 freelancers in India
Single point of failure is bad yes, but you could argue that it also reduces the total number of crashes due to less overhead
@@mertaliyigit3288 if i gave you a pill that would prevent you from catching the common cold, but when you do get sick it requires a trip to the ICU, would you take it?
Efficiency always come at the cost of resiliency.
Higher ups want real time data which in turn result in needing the computers interconnected. But have a bigger problem. And that's that the OS itself has become interconnected with forced updates. A single point of failure for every machine.
All it takes is some bad employee or an idiot to forcedly push an update that would brick all computers.
If saving money comes with not with a risk but with a threat to society, I think that saving money ain't important in that context. I mean, these guys make an obscene amount per year, what's even the point for them to save? Why they want more?
My parents went to the doctors and all the computers are down and now they can't get an appointment till next week. I cannot imagine people with serious health issues now having to wait.
"What's the worst that can happen?" / Sarcasm
Idk how truthful this is, but individuals we know in the medical field (hospital) have said there WAS lose of life because of this. I haven't seen reports as of yet.
@GHOSTSTARSCREAM internet connection for what? to search google on how to do cpr on a human?
One of big hospital in my country got this too. Not a PC on MRI machine for sure. But all those PC that nurses use for appointment got affected.
im sorry to hear that
The blame shouldn’t be on the person who wrote the bug. (Bugs happen). It should be on his boss for not doing code review or integration testing.
There was no bug in the code. The .sys file in question was full of zero bytes, it was a null pointer crash.
Not if the developer pushed their code directly to production, bypassing the test phase. The blame lands on the person who actually deployed it to the production servers without first verifying that all the changes had passed testing.
Directly pushing to production should be forbidden at all times for all developers
@@ZX48K "There was no bug in the code. The .sys file in question was full of zero bytes, it was a null pointer crash." - That's the definitiion of a bug.
@@ZX48KUmm a dangling pointer is still a bug
this also highlight the problem of the shrinking internet. I work in cyber security and the management answer to everything for years has been a cloud Tool, that's quick becoming multipurpose tools (that screwdriver that's also a beard trimmer) which claim to do a bit of everything, provided by four or five vendors... you can see where this is going.
The intern was not the issue. How was this not instantly caught by testing? Not even canaries? Crowdstrike literally tested in production.
I hope they will be transparent about how this update was able to be released. Customers should have a right to know.
they must have deployed "Hello World" instead
There's no way in heII they're gonna be transparent about the update.
@@GackFinder Ironically, it may be a security risk to be transparent beyond what has already been said.
Pretty sure you got no idea what youre talking about.
@@Sam_Saraguy Good point. I bet that's gonna be the excuse they'll use.
How can you roll out a corrupt update? Put it on ONE fucking computer,minimum, to test it.
Where’s your sense of adventure? The pros test in prod!
Well it worked on his machine
@@snarkmark2806 all of us have test environments, a lucky few also have production environments.
What I find suspicious is that they pushed their updates - updates to a piece of software with kernel-level access! - to every computer at the same time, without waiting for clients to confirm.
It's almost as if their development process was _designed_ to install spyware on target computers, run it for a while, then quietly remove it.
@@larsekman8244 Don't worry, you can just call tech support in a panic if it bursts into flames
That ChatGPT programmer, who copy paste code from chatGpt
oh god, if it turns out ai wrote the code maybe we'll finally pop the bubble.
I have a bottle of champagne waiting for the day.
chatGPT you did it again!
it might be chatGPT is sentient and this was a distraction
@@andrewhooper7603 soon the ai bubble gon burst.
@@andrewhooper7603 Holy shit, yeah. Imma open a bottle of champagne with you!
your passion for your subjects is infectious, it's motivating!
I'm 100% certain that no software company does any QA testing whatsoever prior to releasing their updates these days.
For that reason ideally you would not have automatic updates at all, so your IT could vet them in your environment before pushing them. But that sounds expensive.
As my former scrum master once said: We have to release it ASAP as the boss comes back on Monday and a quick test on your computer is enough. 😂
And releasing on Friday is a no no in many orgs
Testing is a stage in waterfall and were agile baby!
"You know how expensive that is? Just take the risk" ~ every manager these days
I am a Security Engineer for a major hospital system and took the day off because im moving this week. I woke up at 10am looked at my phone and saw over 100 teams messages, smiled, and swiped them away. I dont work till wednesday , who would have thought moving would be a great vacation!
I'm surprised they didn't try to call you into the office
@censoredeveryday3320 Im a contractor they literally couldnt
With due respect, in such situation, I would ask for a 10x compensation being sure they will give it to me.
You missed your gold rush 😅
We had three people vacation. We are all still trying to figure out how that happened. Somehow it got approved and then this happens.
I was contracted to audit and implement new security measures for a local business, I put the systems on crowdstrike falcon trial 16 days ago. Glad the trial expired before the update or my contracting days were over.
the worst part is this was a null pointer reference bug. which means either a) they don't pay attention to their static analysis, b) their static analysis is misconfigured, or c) they don't have static analysis. any of those cases basically means that there could be a LOT more bugs in crowdstrike.
Microsoft should not have allowed such access to the third-parties. Now that they did, it is still Microsoft's responsibility. This bug should be called Microsoft.
@@AmirHosseinHonardust you can run drivers under the kernel in linux and bring it down just the same. this isn’t a microsoft-issue at all.
They should use Rust instead of c/c++
@@rajnishsubedi4265 or zig or even go. tbh i’m hoping this crowd strike bug might finally be the wake up call that developers need to move away from unsafe languages like c/c++
.... the reality: Bullying and 'politics' in the Psycho companies... and this is the result... when 'soft skills' are more appreciated than 'technical skills.'
The last point in this is wild. There should be an insane amount of redundancy and separation built in.
This is all a global Psyop crafted by CrowdStrike to make people realize they aren't called CloudStrike
You say that but I still don't know which one it is despite seeing blanket news coverage about it all day
Cloud Strife
ClownStrike 🤡
@@Artofficial1986 Came to say this.
Lol, how about "Clown Stripes"?
We live in a dystopia. For me the news today is not that it failed, is that all those companies relied on such a bad system centralized system.
my windows is win 10 enterprise, paid version, i didn't have any problems. stop using unactivated windows!
We live on the heels of the information, people think they know more than they do
@@araz911..what?
@@araz911 are you joking or dense?
Nah, we haven't reached a dystopia yet
Was sitting on my university campus late last night, about 11PM, in a computer lab, using my macbook. I was all alone. The first PC BSOD'd, then the rest of the lab, and I thought, "cool, guess the university is updating the PCs or shutting them down for the night". Big informational TVs were doing it outside the lab. Wake up this morning, saw news, and loled
Bro that’s incredible lol
That’s a scene from a movie fr
Person of interest
I bet CrowdStrike is a bunch of managers while all development and QA is outsourced for cost reduction and phoned in because DEADLINES!
I was reading that they had recently cut massive numbers of jobs across their QA department… Boeing 2.0?
@@supergeek0177 How can one be this...not smart is the question?
Same bruh
Oh 1,000%!
I'm so damn glad I didn't have to deal with this today. Gods speed all my fellow IT folks.
Same here... And I was this close >< to choosing crowdstrike for my org.
Same here, started my vacation leave yesterday, and i'm hell not going to turn on that phone or check e-mail till the end.
Next time I break my hobby website I'll feel better looking back on this.
I once caused a single but critical software program to be unusable at my company for 5 hours. I now feel much better about it.
Lol love the CEO clip of hangover as CEO missing up Windows OS
This is not the first time this has happened. About 15years ago another av provider did exactly the same thing, they updated thier av, it detected a key windows file as a threat deleted it and bricked a liad of pcs
Oh yeh i remeber that !
It was McAfee in 2010 and the CEO of Crowdstrike worked for McAfee til 2011. Welp, it seems someone struck the crowd again!
System32
i remember similar thing happen on windows 10 4 years ago. but at least you can use windows recovery
@@MrSupasonik Somebody get that guy outta here!
A colleague deleted the prod database last Friday. Didn't hear the end of it all week. He's one of the few happy about this.
Lollllllllll. He was probably laughing in F U language
Crowdstrike: Let’s fire all QA’s and let developers test the code
Dev’s : “Pressing merge btn”
Well if you give the developers some extra time to test the code, rather than breathe down on their necks about deadlines, they'll be better than the QAs at testing
@hakkbak one can only dream.
devs:lgtm-->approve
what are qas?
The “hunt stray cats” line was totally….unexpected and HILARIOUS! I actually laughed out loud!
Started my new job at Crowdstrike today. Unplugged a socket marked "do not unplug" to charge my phone. A lot of commotion in the office soon after that. No idea what it was all about.
How does this not get caught? How do you not do progressive rollout for things like this? Pretty amazingly incompetent deployment setup
most of the majority 😅 of IT people I have met were arrogant
Untested update, on a friday, with no rollout...pure genius
It's the competency crisis. Everything is going to keep getting worse.
The curse of rushed Friday deployments.
this