CrowdStrike Destroyed The Internet
ฝัง
- เผยแพร่เมื่อ 15 ก.ย. 2024
- Recorded live on twitch, GET IN
Big thank you to John Hammond!
/ @_johnhammond
/ _johnhammond
My Stream
/ theprimeagen
Best Way To Support Me
Become a backend engineer. Its my favorite site
boot.dev/?prom...
This is also the best way to support me is to support yourself becoming a better backend engineer.
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-K...
Get production ready SQLite with Turso: turso.tech/dee...
Man it was such a treat to finally join you on a stream -- thanks for having me, and looking forward to more!
Will prep some ciphers for DEFCON 😎
good opportunity for Huntress ads "Hey, Remember this? Wasn't us ;) "
@_JohnHammond is the man!
Dream team!
@@mechanicalfluff it's the same terrible shit made to get money from corporate budgets that are tax exempt, later the same people that decide to buy the software start to work for the companies that sell this crap isn't a bad system.
Hey I'm on the windows 11 dev build and I recently had an update and got the same error blue screen booting up my PC and it won't allow me to get in safe mode or even on my windows password lock screen and I've restarted many times and even did a fresh install of Windows 11 with the automatic repair and did nothing to fix the issue but I kept my files but what it tells me
what failed: rcbottom.sys on the blue screen I can get in my bios settings and that's it I need help please 🥺 I don't want a full clean install and lose all my files with pictures this is my home gaming PC and I hope my storage and hope my RTX 4090 won't get bricked from this.
Edit I don't even know how crowd strike is affecting my PC I never heard of it till now
"I don't always test code, but when I do, I do it in Production." - CrowdStrike
and on a friday
My company does deployments on friday too ☠️
I did a deployment today as well. But it was a fix to spaces in a free text search and I tested it in dev, staging and prod. Very minimal and good testing. But a rollout to millions of machines? Why not?
@@b_twoif fast food and essential workers can operate on the weekends white collar folks can too.
That happened with a tester team that decided to not test the cases and It was showed on production
They found the ultimate protection against malware. No working machine = no malware
Can't get hacked if the blue screen of death is in the way
Think of all the data not being leaked!
@@NODGD You can still get hacked in the half-second before CS hangs you - that's the window CS expects you to use to get the fix.
Can't get Infected nor Hacked. 😂
The Skynet solution
"Hey, boss, I removed this useless regression test to save time. It was called 'boot just one single machine"
This is what i was saying all day on repeat. Love when they didn’t test on a SINGULAR machine
My bet is, the problem happened on the push, or on the build, after the tests.
That's why you only build once, and test that. You never build after the fact
Let's rollout a kernel level patch globally on a Friday Yolo 😂
😂😂😂
Let’s also not test it at all before deploying it to all our computers. All these companies just outed themselves as vulnerable to supply chain risk. They could have prevented this by simply testing it in a limited environment first. At some point both parties are culpable.
4:50pm Friday in New Zealand.
"Yo, Mike, did you test this sht?"
"I tested the minor before. This one is basically the same."
"full send it?"
"FULL FRlCKlNG SEND IT!"
@@skunkwerx9674No you cannot test it in a limited environment, they just push the kernel patch in the background to ALL devices with no admin or user action. I work in one of the biggest bank of the world and it is like all of a sudden all APAC Windows machines got BSOD and no one can do the work anymore. They actually thought that was some sort of cyber attack.
Kudos to the one that came up with the name Crowdstrike; spot on!
I thought CrowdStrike is an infamous hacker group like Anon International
@@kahnfatmanThey are now!
*foreshadowing*
"striking the crowd since 2004"
Worldstrike would be a better name
They should win a Guinness World Record for blue screening the whole world 😂
Add to this a "Carrington -Event style solar flare at the same time..... ", ...... Exit Light , Enter Night .... Take My Hand., .... We're off to Never Never Land .......
3:00 Not knowing Ryanair and being confused about it might be the most American thing I have heard in a while.
It is not the Irish Spirit, it is more like Spirit is the American Ryanair.
Spirit carries around 20 million passengers per year. Ryanair carries 180 million.
Yeah as an American, Ryanair was way better than any of our cheap airlines tbh
Ryanair are the true trailblazers of treating customers and staff like absolute dirt.
Lmao I thought the exact same thing! Wild how all of us over here probably know about a decent amount of American airline companies but they don’t know about Ryanair (not even from the countless memes)
@@Hooverdreng they're better than american budget airlines at least
@@Hooverdrengif you want to get from airport a to airport b in the cheapest way possible, you choose ryanair
0:32 "Security Expert John Hammond"
Something ain't Jurassicing in my park
wdym?
@@petaflop3606 Book Hammond was really lenient on security.
@@petaflop3606(Jurassic Park reference.)
😂 nice one
Spared no expen[SYS_FAULT]
Just so you know, Prime says "server" but this affects clients too. That is, hundreds of thousands of SCCM deployed laptops and workstations...if not millions. Everywhere. If you are doing remote work and your work issued laptop is running this trash, then it's hosed. But so is your whole organization.
You don't have this by default if you have sccm. It is still a paid service. Plenty of companies running AAD were unaffected.
Yep, I am surprised John didn't push back against this. Having an antivirus on a full on server is one thing that maybe could be criticized, as Prime did, however, antivirus and kernel-level monitoring on enduser devices is quite a bit more reasonable.
@@megaing1322 Windows Server don't even need Windows Defender.
But I know why people would do it. A lot of Windows Server is running desktop software and serving it via RDP.
So users are actually using the desktop .
Ideally everyone would use Windows Server Core which is reduced and more like a proper server.
@@megaing1322 having antivirus on servers is unfortunately a PCI DSS and ISO27001 requirement, believe it or not. If not running AV, then you'd have to show adequate vulnerability/threat scanning capability via other means.
The businesses hit by this also show their immaturity. They blindly trust pushed updates, without backing up or snapshotting their crap... IDIOTS! They had no manual backup processes in place to keep business going -- albeit slower but going. This dependency on computer systems is eerie. What if power goes down for 48-72 hours? Which also means no diesel trucks could replenish your diesel generators (oh year which as of 2035 are not allowed by the eco-police in the EU either :/)
What's funny is that 2 days ago the company I work in (a bank) released a post on it's internal network celebrating the acquisition of the "Falcon" tool to make the work computers more secure. I guess it was a really bad timing
"So ironic"
That Falcon driver took down our whole Falcon company the Falcon day we installed it.
I'm also working for a bank. My deepest regrets mate! :(
sorry for your loss
@@arvetemecha I mean it was not really big deal, just a part of the office PCs and laptops were affected, but the suport team quickly released a note explaining the recovery procedure. We are not crazy enough to try to use Windows for servers or critical parts of the business.
CrowdStrike needs kernel space to override syscalls like reading files, mmap, etc. Rootkits and other malware will rewrite syscalls as well. There is no way to intercept calls/access memory for other processes in userspace, and AV is perpetually trying to be "on top", hence the kernel-mode drivers. All AV works like this - once it's hooked in, processes that e.g. read files will be accessing it through a rewritten fopen() syscall that goes through CrowdStrike's driver.
"Channel update" means CrowdStrike's updates - they pushed a new DLL to their release channel, machines downloaded and applied it. There was some kind of error where the file that was pushed (to CDN?) was corrupted, and CrowdStrike's "channel updates" don't employ checksums, so machines just downloaded, applied it, and BSOD'ed cause the driver was invalid. Very hard to imagine how their process possibly could have done an immediate rollout of a corrupt file to everybody... Clearly not a great test engineering culture...
Why is kernelmode AV needed? If I get RCE on Windows or Linux, I can install background software. It doesn't make a difference if it's Windows or Linux, but there's much more money in mass-targeting Windows machines with e.g. ransomware whereas Linux is usually more specifically targeted with 0day exploits. With AV you have a shot at preventing this without patching the software (CrowdStrike is essentially patching it without relying on the vendor); on Linux you're definitely vulnerable until you patch, but Linux also has a much better patching culture ¯\_(ツ)_/¯
Basically it's not exactly clear whether it's good to have something like this or not, but shitty software is the problem in both cases (rewrite it in Rust lmao)
That's the best explanation ive read . Now it make total sense . Thank you
I was curious what the specifics were, that makes sense
worth noting that in kernel space, not even rust can save you from everything. it can make memory easier to wrangle, but it doesn't protect you from other critical faults. a rust panic in kernel space can only sensibly be mapped to an os crash - you REALLY don't want that. and if i'm understanding the issue at play, the os would've crashed either way here, either due to a memory access error or due to a failed bounds check assertion.
I think the real problem is lack of a solid strategy of machine imaging and relying too much of "pet installations"
Ideally you should be able to redeploy your entire infrastructure on clean slate hardware remotely.
I was doing that in 2012 on my in premises.
I could remote Ina server reboot it, feed it's PXE and the entire OS would reimage to a known image.
On top of that I was using virtualization so I could move the VM to another hardware.
@@momofomomofo their update process is an utter crap and those who designed.it.are criminally negligent. And Microsoft is equally liable here for not making overlay updates.a default. They don't even have an overlay fs to start with!
My wife works for an insurance company as a software engineer. She and her team has been asked to report to work today (Saturday) to help the IT guys fix the PCs affected. The number of machines affected is too many for just one team to fix.
You as a team were too stupid to completely rely om auto updating the whole company in one go, you go and work on sunday! Will make you feel the error of your ways! NEVER TRUST AN UPDATE! NEVER AUTO UPDATE! At the very least click, okay start update
I’m a software engineer and my response would be “nope”. CrowdStrike is garbage, I’ve warned it was garbage and haven’t been in IT support or even production support for over a decade (almost 2). The CIO and CSO that thought a garbage startup on the conference circuit can handle IT security can go help… but I’m enjoying my weekend.
It’s bad enough CrowdStrike and Threatlocker DoS my C compiler against a “Hello World app”. Those that made the decision to install that trash can fix it themselves.
Praying for her 🙏
That’s why enterprise desktops need IPMI. And that exists. All this can be scripted via IPMI and BMC on the servers. Same goes for Bitlocker. All scriptable.
I'm loving this. All the times I had to explain to management why we should wait a few days before implementing an update, only to be met with blank stares. lol
Finally you have a good example they should be able to agree with.
Sadly it will only work for 3-5 years, then they will counter it with "that was so long ago, this shouldn't happen with today's technology"
Yeah it’s to common. Everyone is nervous when you push an upgrade…
I feel so bad for the engineer who made this mistake. He's probably going to lose his job even though there were a 100 different failure points from management, procedures, redundancy, and QA testing point of view. I would never want to work for a company like this where one mistake could literally lead to someone dead in a hospital.
If they fire some developer over this, then he dodges future bullets. I'm 100% sure this was some manager's fault who thought QA, staging and safe roll out is dragging away from his annual bonus. Fairly sure the engineers at CS already saw something like this coming. Everyone in the business knows how this works.
issues like this in large software companies don't normally result in the dev's individual termination unless their corporate is chronically micromanagey, and i've never gotten that impression of cloudstrike. more likely, the dev's boss will be in serious hot water, if not their boss's boss (or both).
also cloudstrike isn't known to be a garbage fire of instability - that's part of what made this so shocking to everyone. many IT people _liked_ the software, and that's an honor few apps and services get to enjoy.
@@Asto508This. The developer is too often a scapegoat for bad management and bad processes.
When I hear "What is Ryanair?" I know internet has brought nothing together, Americans still live in their own little bubble and literally and figuratively there's still an ocean between us.
I'm American and I immediately knew what Ryanair was, even though I've never flown on them and only been to Europe once. Primagen was one time talking about worthless courses that people are required to take in college, and his first example was geography.
I knew what Ryan air was. It’s the cheap flights airline in Europe. Not everyone leaves the country enough. 😂
I flew American home from a commissioning trip today. Luckily my flight was only delayed an hour, but there was a like 250+ft line from almost the end of the terminal up the customer service desk, and I shit you not, most of the monitors in the terminal were blue screened lol
First time I heard of Crowdstrike, I was on call on a Saturday night. I happened to be on the computer at the time just checking our systems.
Suddenly all of our ETL jobs were failing, databases down. Turns out they installed CrowdStrike and it blocked network communications and shut down a bunch of our containers. Yeah, that was a fun (not) overnight work session.
In that case, it was doing what it was suppose to. Just nobody told us they installed it.
Unfortunately critical infrastructure like hospitals and government running Windows doesn't surprise me one bit.
What *did* surprise me with this whole thing is how many billboards, signs, etc. runs Windows... You could EASILY power those with probably even a Raspberry Pi Zero, yet they licensed Windows for that...
It’s a minor miracle billboards, signs, and bigass screen arrays work at all. Ever. Even under best case circumstances. The last time I worked with them I had to translate menus from Chinese to English on my phone just to do the most basic tasks and I still almost flung myself off the roof in protest, I can’t imagine trying to manually patch something like this. What an actual nightmare. Just, pain. Legacy broadcast and media standards truly make all that equipment almost unusable when it’s designed well, maintained well, and working. And I’ve yet to see any one of those three things in person. Honestly, I’m more shocked that any of those systems were secured and updated now that I’m thinking about it.
i feel like i see public displays showing some silly windows thing all the time. usually a dialog from some useless software showing up on top of the full screen thing they're running or that it updated itself to become unbootable. either way i'd be more surprised when something is done well than when something is done poorly, because the latter is the standard
IDK about murica but in some countries there's a state law saying that all the software any company uses (esp affiliated with the state) must have a license purchased (for each machine they have and are using). This includes OS, text and tables editing programs, etc.. If an inspection arrives and finds out there is at least one system without license or with an expired one - the company gets a huge fine to pay. Probably that's why they are using win.
Because nobody likes Linux.
@@Hirokuro_Asura you can purchase the right to use linux under the general public license. you can also get it for free, but I'm sure someone will take your money if you really need to get rid of it. regardless i don't doubt your claims about corruption existing.
As someone in the financial services industries, I'm too well aware of this type of software. It's essentially required to run this stuff to pass audit.
parasite industries
I’m an old unix /linux guy currently working at a windows managed services company. You have no idea how little knowledge, especially basic engineering knowledge, 98% of windows administrators have. Including basic engineers street-wise knowledge. And they are working with an OS which is an order of magnitude bigger and more complex than Linux. They have zero mental image of how stuff works. THIS is why this happens.
It shut down the airlines... except for Southwest who were spared because they are apparently still on Windows 3.1
Rollout for NT 4 server scheduled for 2027
no ways ! 🤣
"Isn't EU all about privacy and security?"
Privacy? Yes. Security? Not really. Expecially not enforced to a foreign country entity. That's more of a USA thing. 😅
My mom is the head nurse of a department at a big hospital in my city. She went in at 5 when she usually does at 8. She said today was an absolute nightmare. Like 4-5 usable computers in the whole hospital that were being shared by every department. Nurses writing everything down by hand. She said shes never seen anything like it before.
wth does a hospital use windows software?? thats your problem there.
@@alexd7466 the usual line or argument is "but everyone uses Word and Excel. We need Microsoft products to stay in touch with everyone so we can collaborate" something something
@@alexd7466 I mean yeah they don’t have the time to teach 500 people how to use Linux. You gotta keep in mind some of them are 50ish years old and barely know how to operate a computer in the first place.
@@alexd7466hospital staff are usually between 30 to 50 years old, windows is the easiest to operate even for boomers, why shouldnt they use it?
My wife is a nurse and has been using Ubuntu for over 10 years . I had to switch to windows recently because some of the systems setup require that she login from windows and there are too many for me to help her. Linux is hard to teach is just not true.
So, to me, it appears that CrowdStrike seemingly did not test this on any actual machines before deploying it globally, think about the negligence of that move.
@@ricardodelacrvz1400it's a Microsoft problem only insofar as much as Microsoft's products are so garbage that this kind of 3rd party crap is necessary, and they are responsible only so far as they have made this kind of thing a race to the bottom.
@@ricardodelacrvz1400Why would Microsoft test another company's software?
Writing drivers in Kernel space so any driver issue crashes the system. That's not a Microsoft problem.
Microsoft don't test their own products. Why test another companies ?
@@ricardodelacrvz1400Dude it's not an update that was pushed by Microsoft, it's not their responsibility to test everything and anything a user can install on top of the OS. Do you think car companies like Nissan test every aftermarket part that could be installed? This was a problem of CrowdStrike not testing the update before pushing it and the consumer not testing it on an isolated environment that mirrors their production one to ensure it plays nice with all the other software they may be running. If anything, Windows did exactly what it should've and crashed immediately.
@@RmAndrei93 "Microsoft doesn't teat their own products." That's quite a nuanced view you have there. Yes, no test ever done at Microsoft. I'm sure that's true... not.
2:58 "who is Ryanair and why does he have his own company; why should I trust Ryan?" I can't stop laughing 😂😂😂
CrowdStrike destroyed the best Rootkit ever made*
You mean deployed
@@orbatosUnless Windows is the rootkit 😂
😂😂
It's still in Windows\System32\Drivers\C-000*.SYS heh
@@BillAnt 💀
5,000 isn’t that bad. My company has 7,000 workstations that will need to be manually recovered in addition to a few thousand servers. Gotta feel bad for the IT guys
So my Company's ISP uses Fiber, With Linux based servers, and yet all 40,000 + workstations on my Company's Intranet use W-10 ..... How stupid is that ???.... Anywho ...I managed to get about 10+ PCs active in critical areas, before IT showed up and took over at my site this morning.
Kudos to that ISP, which is the same ISP that I have at home ..... And I've been using Linux since ~2003.... So needless to say, I was unaware of the " Take-Down by Crowd Strike" until I read Google news this morning.
(FORCED) Pusheed to Prod at Fridaaayyy -- Burned by its sins.
In all seriousness, forced remote updates are horrible. And it was pushed to millions of users without proper testing...
This is the poster child for untested changes, and it's unbelievable how much risk companies are assuming by allowing forced pushes from this vendor.
It’s insane, push based autodeployments with no vetting period…wtf!?!?
It’s not really forced at all, every organization that uses crowdstrike has the option to review the updates before using them, everyone that was affected didn’t even test the updates crowdstrike provided. The fact they just went with the force update workflow was a disaster waiting to happen, and here we are. Source: Crowdstrike documentation.
it may have been a security update that worked on the absolute latest version and not any prior version
which i've heard of happening many times before
@@skunkwerx9674apparently this came out of an automatic update, not a new program binary
Crowdstrike: Security so good, it attacks itself.
☠️☠️☠️☠️☠️☠️☠️
The computer has autoamune desiese 😂
_It hurt itself in its confusion_
Apple had a similar problem with a content update for their XProtect a few months ago. It falsely identified iOS simulators as containing a virus and would remove them. It only affected developers working in Xcode for about a day. It does show how automatic security updates can create big problems. I unchecked the “auto install security updates” box after that.
I think the best scenario is to have delayed auto-updates to avoid quickfire rubbish. I think being behind like 2 weeks should be fine.
Back in late 90's/early 2000's a lot of European airlines were using Linux but complained after a few years that it stopped working correctly and abandoned it. Root cause of their problems lie in that they never ran updates on their systems. they somehow thought that they never had to run updates and that their systems would just continue working fine forever.
That's what *I've* always heard about Linux 😄
That's because before SystemD updating linux in the massive IT systems corps use was indeed a pain in the ass.
@@SimonBuchanNzIt's true if you're completely offline. But if you're connected to the internet, standards are going to change and vulnerabilities are going to be discovered. That requires updates to prevent things from breaking.
@@Tetus7 more just a joke about all the weenies back in the day crowing about all the windows security updates... as if Linux didn't need them too.
@@Tetus7 Not just offline, embedded systems too if exposed to public
Longtime security professional here and I must say that I am shocked by the lack of awareness around how all of this stuff works.
It should be noted that enterprises run EDR/XDR agents such as Crowdstrike on Linux, Mac, and Windows machines.
To be able to detect modern, sophisticated malware, you need low-level/kernel access to the machines. Enterprises manage a ton of machines and to protect our environment from endpoints (servers/laptops/etc.), we need to monitor them as users are traditionally the riskiest thing in an environment.
It is obscenely sad that I had to scroll down this far to find this. Well past morons somehow watching this only to decide that this is somehow the fault of SCCM. Much less all the misinformation and just general weird opinions being pushed around by people that are supposedly tech savvy in TH-cam videos like this. I mean Jesus. This dude does not apparently understand why AAD servers exist in 2024 when Linux still runs as jank as it does and is in no way friendly to learn for general users?
What you said makes sense. What doesn't make sense is how this F#%K UP happened. Do companies test in production now?
@@laughingalienYes
@@laughingalien Always have, always will.
@@laughingalien No. AV companies have infrastuctures in place to test product updates before pushing them into real-world. I''m guessing it is either a QA engineer fucking up when checking the testing results or an issue with their CDN serving a corrupt file.
CrowdStrike and Kernel Panic on Linux happened like a month or a few months ago. So...this isn't a Windows VS Linux thing. I work at a Cloud Provider and I've seen these security solutions tear up Linux environments too.
Yeah, Prime failed pretty much on Windows part...
Worst I’ve seen is falcon “storms”.. where they use an obscene amount of CPU.
This is a partly valid point but with Linux you generally don't need to have this sort of anti-malware client in the first place!
@@JimAllen-PersonaI've also noticed weird spikes on CPU by an unnamed EDR solution. The oddest thing is that they aren't triggered by any obvious system call or daemon. It's non deterministic as far as I know.
@JimAllen-Persona yeah seen the same happen on 2016 iMac model too.
Thank fully they replaced it with 2018 edition iMac lol.
31 seconds ago is wild, it's neat to be in here at the same time as the scam bots for the first time in a while.
"is wild" like what does that even mean
A bot would for sure say this!
@@youtubepooppismo5284"is wild" means "is crazy".
@@hdbrot No shit sherlock
@hdbrot this man either isn't chronically online and/or does not know any black people
Why does a BILLBOARD need to be linked up to a computer with windows installed? What a waste.
It's pretty standard. Reason is that there are drivers for weird resolutions, industry standard "digital signage" software for Windows, and it mostly "just works" with zero effort. Not sure why they'd install a virus scanner like this on one though - generally they don't have internet access (or only access a specific server once per day to download media).
That said, many of the latest "slab" type screens you see in malls etc use Raspberry Pi compute modules internally.
@@Stabby666 they'd have crowdstrike to be in compliance for cyber security insurance. If it connects to the internet at any time for any amount of time, it has to have an EDR solution for most cyber insurance AFAIK.
At this point it's clear they didn't perform integrity checks on the update when sending it on the client end and there is no rollback mechanism for an update failure. The bug causing a null payload is severe, but nothing compared to a total lack of sanity checking, rollout testing and staging.
Yup, we can talk all this trash about CrowdStrike, but at the end of the day the client should have a vetting period in stage
@@tc2241Turns out this was just a massive security audit.
@@tc2241Yeah but mind you, some companies have been using it for years. Eventually you just trust the “experts” and focus on other important things. Not excusing anyone, just saying, it’s understandable. Crowdstrike takes all the blame because they were the ones promising security and reliability.
@@tc2241 No? That is misunderstanding what CrowdStrike provides as a feature. CrowdStrike should have had some kind of actual, real-world testing infrastructure to check that the updates get applied correctly.
The point is more that shit happens, and CS is no exception. But they appear to have had zero safe guards, or whatever safe guards they had were terrible.
@@tc2241No? That's literally what Cloudstrike is paid for? Why would you pay a company for managing your security infrastructure if you need to test their changes every time?
I’m surprised how much infra uses Windows.
That was my immediate thought as well. Holy hell. You would think that infrastructure like transportation would have their own OS's with a ton of redundancy.
shocking really!
Near all cooperation and governments use windows because of Office & Teams + mentioned group policies(which also ended up that a lot of industrie hardware has .Net APIs) . Linux is only on the webserver side big. This also the reason windows is so big its as much feature complete as possible
@Fiercesoulking you're right. Many products from MS are very useful for business, when similar programss are not that good or untrusted for managment. It's sad but there is no many good alternatives.
second reason is that windows in the peak had around 90% of market share, so many industry specific programs was written for windows and they still in use.
The hospital staff didn't know which medication my dad was scheduled to receive today.. This is absolutely embarrassing for the hospital in my opinion. They should've never setup their infrastructure like that.
My dad went the doctor today, the computer system was showing incorrect insurance information.
Cuts mean no funding for paper backups or fallback systems.
With security like this who needs ransomware
@@Wahiniesi mean crowdstrike all but zero-day'd itself. the bug in question was one of the classic vulnerability culprits.
How should those hospitals have set up their infrastructure if you don't mind me asking?
Effected my department. I had to go around recovering my coworkers' conputers.
Ditto, I got about 10 going here this morning before I.T. showed up and took over.
I had too as well. What a headache.
It did not turn off any Internet, it turned off machines that use Internet. There were no internet outages.
Mine was perfectly fine...
Prime doesn’t have an IT ops background. To him servers are ephemeral but that’s not how traditional IT systems work.
I drank a shot every time you said CLOUD STRIKE as the words CROWD STRIKE were on the screen right in front of you. Now I'm being rushed to ER
The ER is closed due to bsod
@@thewhitefalcon8539 🤣
✅️Confidentially
✅️Integrity
❌️Availability
It's 100 % confidential is it's 100% reliabiably useless
CIA
Exactly!
Considering they didn’t use checksums to verify the update files, I think we’ve only got the C here
In the US, folks woke up to this, but in Australia, this all happened at 3 pm, peak hours
Yup - same in Japan. All I could think of was the people in the US about to wake up to this. :)
@@zoeherriot oh yeah, forgot how similar our time zones are
list of people who asked:
(it's empty)
@@harleyspeedthrust4013 like your head.
in Asia, its in the middle of the day, 12nn - 1pm. Many of us came back from lunch with our workstations stuck in the BSOD loop.
You said the right thing: "Why are you using windows for a serious thing, in first place? "
Two questions: 1) Why didn’t they see this bug in testings????
2) Why didn’t they push this update incrementally to a smaller amount of customers?
Skilled people are expensive and less inclined to kiss ass.
1) Because the testing was insufficient.
2) Because that insufficient testing was believed to be sufficient by them.
@@takeuchi5760I'd rather think cost reduction by management. CS has become big enough that some cowboy managers entered the company and wanted to increase their share.
@@takeuchi5760every. single. time.
Because nowadays you get in higher position by kissing your boss ass... that's why this happens and it would get even worse in the future. Non thinking "yes men" get better salaries and are placed in higher positions.
My team was in the middle of a production go live when our systems started getting struck down one by one. thankfully, my own machine would only bsod intermittently and not on boot-up. When googling the issue, I found that this wasnt even the first time crowdstrike has caused these issues (my company adopted crowdstrike late last year). There were forum posts from july 2023, and march 2023 of the exact same issue.
>thank the day off
>I'm an IT Tech
MORE LIKE ENJOY THE HELL ON.
Thank you for your service
🫡
Jokes aside, imagine your life or the life of a loved one depending on systems like these (for travel, insurance, or healthcare) and getting stuck without any immediate resolution. Hope no one died because of this.
Imagine missing final moments with your dying loved one bc of this
Oh wait till the damage law suits start piling up. This could very well wipe out cloud strike
Sadly, in the UK, one hospital did report a critical incident as a result of a third-party IT system being impacted by this. Who knows how many more will be reported?
I get that "shit happens," but this incident needs to be independently investigated. Was this update properly tested before being rolled out, or did they skip best-practice safety measures to save time and/or money? If they skipped safety measures, then they could potentially be liable for involuntary manslaughter.
That's why you don't do every system the same as the other one, even if it's the most secure system in the world, you have to have a back up system made in a completely different way. But I thought the geniuses who secure banks and whatever and get a lot of money knew that... it's seems all are just like parrots, everybody does the same thing as everybody else... and then when one mistakes happens, everybody "burns together".... what a stupid thing to do. Again why everybody was using the same security software?!... and why critical systems were not using Linux?!... By the way I didn't even knew that company existed... until now.... why banks and other organizations use products from some unknown company is beyond me. Friends were asking me if we have problem with our Windows systems at my work... I just told them nah, we don't use Windows in critical systems. I thought that was common especially in banks... but it seems they use Windows with some antivirus/security software.... *which I didn't know exists, until this whole thing happened.* it's unfathomable to me.
Also best impression of Seth Rogen in Cybersecurity
Right?🤣🤣
I actually love that "anti cheat" is like a point on the scale of how intrusive something is
Y2K finally came but it was 24 years late
* Ryan and John push a global kernal update *
"Wait Ryan, are you seeing what I'm seeing?"
"Shit."
The short pause to slander United Airlines was cathartic. I’ve been saying the same thing for the last few years and I finally feel heard
"i'll never use linux, it doesn't have antivirus!"
meanwhile, antivirus:
This turned my normally pretty dead Friday morning into a hellscape. My organization has Falcon on all endpoints, and many of our customers are on Windows, and we had a LOT of tickets come in.
As for servers, unfortunately there are a good bit of windows only application servers, it does suck
Yup, I work in fintech and even the working PC's were struggling to load apps and software systems, half the time nothing worked. It was a long day.
A lot of companies run crowdstrike or generally cybersecurity suits on linux/unix too, this is not a windows problem. And generally enterprise runs on windows because of active directory and office. Also .net and c# is quite common for monolith applications
In grocery: my beverage company had an issue in sales, some system went down.
Another beverage company, their warehouse picker system for beer went down.
A grocery store(singular to my knowledge) clicklist system went down, no online shopping allowed.
Starbucks mobile ordering went down (nation wide I heard).
Mr. Hammond, I think we're back in business
EDR is for Linux and MacOS too. Not only Windows. EDR for linux server is the first cell to detect a security breach - as long as it works 🤣.
This happened to some Debian servers in April, just the blast radius wasn’t big enough to make news
because crowdstrike is not required to make linux secure, now on windows....
@@vilian9185 my company used to run crowdstrike on all machines, Linux as well. SecOps policy :)
@@vilian9185 GLibC binary incompatability is enough.
@@vilian9185Falcon is still available for Linux, so some people must be installing it
@@vilian9185 you don't need crowstrike on Windows either.
my phone started ringing at 12:49 am - "we are down, have BSD on many machines, can't reach the server screens", fun way to wake up. Long night,
If you had BSD on your screen you would've been fine :D Think about an OS that's called BSD ;)
Most companies and governments in the EU are lobbied into oblivion to use Microsoft, antivirus and such. There is somehow a strong urge to be dependent on US big tech. Open source efforts are usually belittled and soon de-funded. It is quite frustrating.
Of course they are, there was a regulation in my country to use only licensed software, by companies... i.e. Windows... this stupidity stayed for a few years until the regulation was changed. I think some US politician just came to my country and then that "regulation" was invented. Later the regulation was changed, but for a few years that nonsense regulation was enforced.
Dunno why everyone's complaining about Microsoft and Windows here. Crowdstrike isn't their product, and it has Linux and Mac versions, it's just they happened to not get hit by this one.
It's not like there isn't plenty of *good* reasons to point and laugh at Microsoft security and reliability: they recently took like half a year to squash all the print spooler vulnerabilities, for example.
Agreed, we run it on Linux.
You missing the point even while you have it in front of you: Windows is SO SHIT that it is the only OS who got affected by this thing. Maybe if Windows would'n be such a shit, Crowdstrike wouldn't kill their systems.
@@alulim4968 CrowdStrike had a very similar problem causing kernel panics on Linux systems just this april. This does not say anything about Windows.
imo it's just a good excuse to keep laughing at microsoft, bonus points for the fact that forced updates is something they have been pushing very hard for (not that this was a forced update anyway). also in a lot of these cases windows was just used for things it should never even have been considered for (public displays, etc.)
@@alulim4968 when u make kernel level software we no longer talking about pure OS, u missing with the OS if it fail its on u.
not microsoft fault, honestly not even the kernel level software fault cause these shit are t be expected, the idea that this sort of solution where used in very important areas that should never have kernel panic is very stupid.
This is possibly the best named company in history. This is exactly the same result if the entire crowd goes on strike.
On updating old systems to new ones: 6 years ago, when I was working at Walmart, we had someone updating our Self Check-Out machines with newer software. They updated the computers from XP to Vista. Yikes! And people wonder why our security is such a big issue.
Spent the day fixing computers at the office and went to buy cat food at the pet store and most of the registers were blue screened. "Not my circus, not my monkeys."
01:00 The answer is that probably 95% of all business and B2B related software (in offices) runs on Windows and Windows only even on the Server side.
Try to teach your average office worker who struggles to tell if the PC is turned on or not (when the screen is black because its turned off) to install some random Linux Software with 7 dependencies that you need to install via shell (Good luck with that one) or in other words: "Why no one cares about Linux in offices and no one ever will". And no: Ubuntu is not a good example of "easy to use" by MacOS or Windows standards that are already considered to be "hard to use" by average people.
Average people dont't even know the difference between "user" and "password" when they get prompted to login. Any more questions?
Server class hardware has out of band management (HP has ILO, Dell has iDrac) which can be simply described as KVM over the network. The machine does not even need to be switched on. Many client machines in enterprise environments have similar functionality, such as Intel AMT.
So no, if configured correctly, no one needs to physically visit each machine.
I'm in IT. Our servers came back up pretty quickly. The bigger issue was the endpoint client. We couldn't just write a PowerShell script and push it for a fix because none of then endpoint had internet access. We had to access the Recovery option, get into CMD, remove the bad update file and reboot manually; on. every. single. machine. (sometimes guiding our user over the phone). The reason for the shut down, from what I can tell, was not the severs being down, but the endpoints.
My personal laptop was affected. Like three days ago it started tô get slow. Then even slower. And Then yesterday the blue screen showed up forcing a reinitialization
Hey Bill. I hear ya. I had to brute force make about 10+ workstations disable the csagent.sys update in some rather critical departments here.
Fortunately our ISP, Fibre - based, was unaffected and once I got my PC going, ( all of maybe 2 minutes of finagling without a sweat), I went after some others, all done via GUI and some brute force resets, no CLI, ( CMD in your case), needed. What I did see though, is the update affected PC's differently, which was odd. If the PC was inactive during the update, ( User logged in with screen locked but still on the intranet, as in my case..) , a GUI reset took about 2 minutes to force the use of the older csagent.sys file,(?), and log on, as the update had failed, but it gave the same BSD diagnostic. But,... If the PC was active and the forced reboot was attended by staff logging out to allow the update, then the reset was almost impossible without the intervention of IT to go Root and do a reinstall of CrowdStrike , or possibly re-map the drive of the PC.
Nuff said on this , ... lol
@@Yamahog is it possible tô fix mine?
IT management: we need to restrict employee's permissions for security.
Employee: please approve I'll need some permissions to do my work.
Security team: wtf you need that permissions.
Crowdstrike: I need your super admin to install patch on your keneral.
IT management and security team: go for it. Thanks so much.
Given: that CS CEO was McAfee CTO who created a big disaster crashed tens of thousands of computers.
That guy is much more reliable than your loyal employee.:)
This would've never happened if they did a internal test before they push out a update.
Helpful insight
I'm amazed they don't do that.
MacAfee come back! We have cocaine here too!
It crippled a bunch of Windows machines,...it never "destroyed the internet". The "internet" is a network, not a "computer". A network is like the roads and computers are like the cars,...crippling a bunch of cars doesn't "destroy the roads".
Ryanair is the other cheapest flight travel provider but in EU , They might fly from airports nobody else makes money from and shure dont expect much in term of service but its cheap for students
How can you not make a million Jurassic Park jokes?
I feel left out, my IT infrastructure didn't get taken out today, all our stuff (including some windows boxes) are on-prem and don't have it installed. I still had to work.
Mostly, crowdstrike is enterprise cloud security for fortune 500 companies. Everyone's freaking about PC level, and I get it's a lot of machines, but if your enterprise servers are busted who cares if machines turn on, you legit can't access anything required to do your job at least not consistently, even if some servers are ok the load is too much.
What's really crappy about machine level at enterprise with these types of securities is that there's master admin sso keys that are typically necessary to even be able to boot into recovery to apply fixes. Hopefully orgs with hundreds of employees per site have those all organized by workstation (hahaha everyone knows that's unlikely!)
thank god i was left out and hopefully won't have to touch a windows machine for the rest of my life. you know, except for when i check in on a flight or some other thing where they decided an angle grinder was a good screw driver
0:43 "So sit back and enjoy the day off. Thx crowdstrike" Just savage xD
The hassle was global. The company must be held responsible.
1:02 IMHO it is companies, not people / employees, who choose to use Windows because of the features provided by Active Directory.
Also, a lot of creative software does not natively deploy to Linux. Not to mention the driver headache that non technical people don't want to hear about.
BSD doesn't seem to be exactly made for desktops.
And justifying the pricey mac to the financial department doesn't seem to work.
Hey, great to see John Hammond here!
My employer was unaffected because we don't use CrowdStrike.
I also wouldn't have been affected because I use Linux. GG EZ
Literally happened to linux back in April. Crowdstrike has a linux version and it also caused a kernel panic.
The only difference is that not a lot of public infra runs on linux so it didn't make the "normie" news.
@@AmonAsgaroth”not a lot of public infra runs on Linux” - the vast majority of the Internet runs on Linux distros. I would imagine almost none of them uses Crowdstrike, however, hence no public outcry.
Better is to have a stage env and not allow automated push deployments to prod
@@tc2241this
@@Ubben1999and yet the vast majority of companies use AAD or AD or azure hybrid.
You'd be surprised what an amateur hour the airport, medical and banking world is sometimes, so many "server" applications which are just a GUI running on some desktop machine. If they're lucky they get a dedicated machine, but often it's just running under someone's desk, being also used as a normal client computer.
CrowdStrike really made the dream of wannabe hackers come true
There’s some misconceptions in here. My company got hit pretty bad but we were a microsoft house so a lot of our servers including domain controllers, local DNS, build servers, web and api servers are all windows based. But a lot of the impact was from customer representative machine, people that just about know how to use a computer. Even if a lot of the infra is linux, a lot of the front end stuff is still windows as that’s what people know how to use.
Also Crowdstrike isn’t just for Windows, its installed on all our machines, i.e. all linux installs get it too, it was only the Windows machines that were affected this time.
My cousin works there and said he was on call but it wasn’t his team. Wild shit
R.I.P. your cousin's Employee Stock Investment Program.
reminds me of the time while i was on call at aws where cloudwatch's log ingest kicked the bucket for hours in us-east-1. fortunately we weren't impacted much beyond just flying mostly blind, and the justifiably spooked backend teams in my department also survived. but the execution plane for one of the other departments almost went down with cloudwatch bc their logs weren't rotating (bc the uploads kept failing) and their disks were filling up so fast it was threatening to down some of their hosts in mere minutes. (they ended up manually deleting logs regularly across these many thousands of hosts, just to keep availability.)
Good job getting this video out editor. Well done Prime and all.
They went the Soviet route: "no computer - no problem, comrade!"
I called in sick on Friday, at like 6:30 in the morning.
Went back to sleep.
Woke up 5 hours later: "Oh wow, IT is globally on fire, the Internet is imploding, Windows is dying..."
Went back to sleep.
If there is a God, homie's got my back lol.
How did it pass through QA checks, do they really have bad deployment setup, like they are the best in the business, how can that slip
Non tech CEO tells management "Cut costs. Do we REALLY need QA? Get rid of them"
"We haven't had a mistake in years, fire the qa team, we don't need them"
@@aisle_of_viewTo be honest you don’t need QA if you have a proper staging environment that mirrors the Live environment and if you have an adequate deployment process.
Agile Teams that deploy and test their features themselves work far quicker and more efficiently without QA - the idea is to fail quickly and deploy quickly in small increments. Obviously you still need safeguards such as a proper deployment process - but that isn’t an argument against agile development.
Beancounters want to save 1 cent, that's how these things happen.
2:29 tell my you never flown Ryanair without telling me you've never flown Ryanair.
Ryanair is the largest airline on earth. Known for wanting to sell stand up "seats" in the aircraft, basically they tie you down to a vertical pipe.
The cheapest
Aint no way they would allow these stand seats, because the aircraft has to evacuated under 90 seconds or they wont be certified to be commercially used. And if you have higher seat density with these stand seats you will to not pass the evacuation limits.
LOL
@@petrsebik Maybe they wanted to do that on smaller aircrafts that can be evacuated faster
@@araarathisyomama787 maybe. But not by ryanair, as ryanair's fleet is by 95% boeing 737 with around 190 passangers capacity. And the other 5% is 28 leased airbuses A320 with same capacity.
John Hammond f’d up when he hired Dennis Nedry to run his theme park technology. 😂
Somebody's getting fired for releasing this to Production on a Friday.
And sadly it's going to be the nervous Jr guy who were pressured into "just push to prod" on his first day.
In the UK, the NHS (National Health Service) is heavily reliant on Windows and Microsoft products (Office, Exchange, ActiveDirectory, Azure, etc...) for they get the licenses for cheap. RyanAir is a budget airline that flies mostly to European countries.
Is this why my fucking internet has been crashing like 2-3 times everyday for the past few days???
No, this wouldn't be related. Who's your ISP, and what are your speeds?
Medical - I have a friend who maintains medical equipment. He has to dispose of old, (fully depreciated) PCs(!) that they use to run MRI, CAT scan, and other imaging equipment. He tells me that he's continuing to swap out WINDOWS SEVEN machines that are still being used to run the machines.
You have to realize that operators know what they're doing, the techs know what they're doing, but anybody above there is completely clueless, so they have to make 'executive decisions' that will cover their butts in case something doesn't work. So windows, Microsoft, and other big names.
ChatGPT please can you fix it ? 🤣🤣🤣🤣🤣🤣🇧🇫🇲🇱🇳🇨🇳🇪
Dev: no updates on Fridays
Hackers: release malware/attack on Friday :evil laugh:
Sky-Net went Online.