$135 Billion Accidentally Deleted By Google

แชร์
ฝัง
  • เผยแพร่เมื่อ 22 ธ.ค. 2024

ความคิดเห็น • 871

  • @AwesomeAlec1
    @AwesomeAlec1 4 หลายเดือนก่อน +1171

    Friendly reminder that "The Cloud" is just someone else's computer.

    • @mallninja9805
      @mallninja9805 4 หลายเดือนก่อน +112

      And it's someone who does not care about you or your data.

    • @styleisaweapon
      @styleisaweapon 4 หลายเดือนก่อน +1

      @@mallninja9805 Google cares about every bit of data that they can get their hands on -- Google surely had another "backup"

    • @2rx_bni
      @2rx_bni 4 หลายเดือนก่อน +7

      YUP

    • @marcosdly
      @marcosdly 4 หลายเดือนก่อน +3

      but... but...

    • @Live-ug6zb
      @Live-ug6zb 4 หลายเดือนก่อน +6

      @@mallninja9805 say that again but with Big Data companies and their means in mind.

  • @tc2241
    @tc2241 4 หลายเดือนก่อน +365

    As someone who’s worked a decade in the financial sector. You could spend 200k on back ups and it still would be a cost save.

    • @danielthecritic88
      @danielthecritic88 4 หลายเดือนก่อน +11

      But also a decent DR plan helps. 😂

    • @jackpowell9276
      @jackpowell9276 4 หลายเดือนก่อน +9

      These guys will be wishing they spent a billion on backups right about now.

    • @SomeUserNameBlahBlah
      @SomeUserNameBlahBlah 4 หลายเดือนก่อน +5

      Yeah, but some new manager or finance guy decided to increase their bonus by saving money...aaaaaannnd here we are today.

    • @Volvith
      @Volvith 4 หลายเดือนก่อน +19

      200K a year.
      ... _I worked for a private bank that will not be named, handling absolutely stupid amounts of money._
      Even just the fines on record keeping obligations being broken could bankrupt medium sized businesses in those cases.
      Say what you will about the financial sector building with dinosaur bones, they sure as f*ck make sure your compass of consequences is aligned dead straight. A colleague of mine accidentally threw away the phone records of 5 traders.
      The legal shitstorm that ensued was honestly terrifying to behold.

    • @1130MarsV
      @1130MarsV 4 หลายเดือนก่อน +1

      I always figured anything financial is regulated to the point where stuff like this can't happen but I guess this is completely wrong.

  • @matt755394
    @matt755394 4 หลายเดือนก่อน +107

    I work for an Australian Super Company (As a Software Engineer). It is a legal requirement, enforced by the regulator (APRA), that we have back-ups of back-ups (Stored with different providers, or different data centres (Not in the same State) if servers are in-house). We are audited each, and every year to prove we can recover from a disaster, and data can be recovered. Massive fines, and restrictions can be placed on the companies if not met. Superannuation is a heavily controlled/regulated industry.

    • @interstella0
      @interstella0 3 หลายเดือนก่อน +9

      a W for Australia on that one

    • @NickTaylorRickPowers
      @NickTaylorRickPowers 3 หลายเดือนก่อน +5

      ​@@interstella0yeah we have good regulations from government at the moment
      We are actively losing them thanks to government ineptitude or corruption
      Victoria just lost the union for the building industry because of allegations of links to crime

    • @Foxstab
      @Foxstab 3 หลายเดือนก่อน +1

      Conglomerates and megacorporations, yes. But regular SMBs and such, even some enterprises, no.
      There is just not enough staff and so they don't bother.

    • @BabyDingo
      @BabyDingo 3 หลายเดือนก่อน

      Its the same in singapore. Of course its gonna be heavily regulated- this is everyone (including the rich people’s retirement money)

    • @MrSonny6155
      @MrSonny6155 3 หลายเดือนก่อน

      ​@@NickTaylorRickPowers Oh shit, I remember seeing something about the sheer corruption of the building industry in Australia close to a decade ago now. Given how this was treated as a "well-known fact" in some circles even then, it was probably a long time coming.

  • @size_t
    @size_t 4 หลายเดือนก่อน +746

    At my last work place, I wrote a tool, that sets up a test environment of our product.
    It was very hacky, because I had to mock some external dependencies (ex. SAP). And I included a very prominent disclaimer, that this tool is only for testing purposes and should not be used for specific cases.
    Guess what, they're still using my tool, to "install" our product on customer environments ...

    • @-Jakob-
      @-Jakob- 4 หลายเดือนก่อน +171

      it's your fault, you didn't add functionality that detects the scope of operation and activates restrictions if necessary - like e.g. silently activating automatic expiration.

    • @lynx-titan
      @lynx-titan 4 หลายเดือนก่อน +94

      @@-Jakob- truth. Nobody reads the docs. Build them into your code.

    • @mattymattffs
      @mattymattffs 4 หลายเดือนก่อน +32

      That's normal. A hack I wrote for a customer is in production. It really only fits that customer's use case...

    • @David-ty6my
      @David-ty6my 4 หลายเดือนก่อน

      Thanks I'm gonna do that. I'm new in the IT game​@@lynx-titan

    • @pluto8404
      @pluto8404 4 หลายเดือนก่อน +20

      this is why I build in auto delete into these cases as default. Then if they do use it in prob, it will auto delete itself do they cant continue using the test software.

  • @ratiod-nz
    @ratiod-nz 4 หลายเดือนก่อน +335

    THE OUTRO GOES SO HARD

    • @sirtra
      @sirtra 4 หลายเดือนก่อน +8

      Best. Outro. Ever. 😂
      (But which one am i talking about? The answer: yes)

    • @peterson6824
      @peterson6824 3 หลายเดือนก่อน

      Thanks! Aaaand, it's now the 3rd most watched part of this vid.

  • @Kane0123
    @Kane0123 4 หลายเดือนก่อน +331

    17:25 the commenter saying “time for the garbage collector to run” just made me spit my drink

  • @thewayfaringshadow
    @thewayfaringshadow 4 หลายเดือนก่อน +156

    worked for a tax company that had quarterly physical backups, in a vault in a different region that was not the home region not connected to the internet. It was wild.

    • @IreizD
      @IreizD 4 หลายเดือนก่อน +17

      that's badass

    • @adissentingopinion848
      @adissentingopinion848 4 หลายเดือนก่อน +36

      This is the correct solution

    • @ci6516
      @ci6516 4 หลายเดือนก่อน +10

      Networking guys weep

    • @alilhard
      @alilhard 4 หลายเดือนก่อน +28

      I was always taught "you keep at least 2 backups, one on site for if your computer explodes, one off site as far possible for if your building or your city explodes"

    • @0x1EGEN
      @0x1EGEN 4 หลายเดือนก่อน +14

      A lot of companies do physical backups. It's called tape rotations and these physical media often get sent to Iron Mountain.

  • @raskalthefirst
    @raskalthefirst 4 หลายเดือนก่อน +72

    Re: backups of backups.
    It all really depends on your fault tolerance, here in the Netherlands we had a giant flood in the 50s, it killed over a thousand people, the country got together and decided let's make sure that "never" happens again.
    And we build the delta works, a flood prevention system designed to withstand up till once in 10 thousand years surges.
    The same goes for IT, you have to calculate the value of your data, the cost of loss and the cost of proper prevention...

    • @chupasaurus
      @chupasaurus 4 หลายเดือนก่อน +9

      Old IT saying goes like this: "If you don't have backups - you don't have your data. If you don't have 2nd backups - you don't have backups".
      As for last sentence, it is a formula which applies to any risk/security period.

    • @Foxstab
      @Foxstab 3 หลายเดือนก่อน

      @@chupasaurus If you never do complete SUCCESSFUL recovery, you don't really have backups.

  • @Volvith
    @Volvith 4 หลายเดือนก่อน +39

    _"Has there been a breach? Has data been compromise?"_
    UniSuper: "Well, in terms of data, *_we've got no data."_*

    • @DE-sf9sr
      @DE-sf9sr 4 หลายเดือนก่อน +1

      "What Data?"

  • @GreedoShot
    @GreedoShot 4 หลายเดือนก่อน +89

    "what do the logs say?"
    "...what logs?"

    • @MyWillBeDone
      @MyWillBeDone 4 หลายเดือนก่อน +3

      NANI?!

  • @arkemiffo
    @arkemiffo 4 หลายเดือนก่อน +210

    I image the UniSuper team had a crash before but it was insignificant to outsiders, but hair was on fire inside the company. When they moved over to Google, the dev-team said, imagine if that would happen, but with a billion dollars! That's how they leadership aboard on such expansive backup-plans.
    And I'm guessing that dev-team can write "Literally saved the company a billion dollars in loss" on their annual performance review now and have a black space to fill their own payrise. Well, they should if there is any justice in the world, but this is the corporate world, so 3% extra was probably the max.

    • @theman47145
      @theman47145 4 หลายเดือนก่อน +6

      Or 0 bonus as this is their job.

    • @TheOneMadHawk
      @TheOneMadHawk 4 หลายเดือนก่อน +17

      How about a bottle of wine and a bar of chocolate as the thank you gift? Or my favorite: Why didn't you show up on time this morning after fixing this issue all night? => The entire team quit that day.

    • @sanderd17
      @sanderd17 4 หลายเดือนก่อน +10

      They'd be lucky if none of the blame stuck to them.
      If you have to give bad news to your manager, so bad that it reaches the top of your company, the blame game is likely to start before figuring out what actually happened.
      If management marks you as the guilty one, it's pretty hard to get rid of that stamp as they often don't want to admit they were wrong.
      Even if Google is at fault, like in this case, they'd still blame you for choosing Google, for not providing a hot backup on a different provider, for not detecting this could happen,...

    • @conceptrat
      @conceptrat 4 หลายเดือนก่อน +2

      C-suites bonuses 😭

    • @MrLordLowbob
      @MrLordLowbob 4 หลายเดือนก่อน +1

      non-tech CEO be like: "but the service was down for multiple days and we got flak for it, you're shit"

  • @MattMcQueen1
    @MattMcQueen1 4 หลายเดือนก่อน +66

    If you are dealing with $135 Billion, it has to be backups all the way down.

    • @OFDM-network
      @OFDM-network 4 หลายเดือนก่อน +1

      No if they dealth with $135 Billion, so to speak.

  • @fromduskuntodawn
    @fromduskuntodawn 4 หลายเดือนก่อน +125

    Aaaannnd, it’s gone. It’s all gone. This line is only for customers who have money!

    • @mg4mg281
      @mg4mg281 4 หลายเดือนก่อน +1

      S.P. xD

    • @Littlefighter1911
      @Littlefighter1911 4 หลายเดือนก่อน +1

      Lessons learned, don't trust Randy when it comes to finances.

  • @Exilum
    @Exilum 4 หลายเดือนก่อน +9

    5:25 Still expensive, but apparently the "ve1-standard-72" is 2 CPU, 768GB of memory, 20TB of NVMe storage with a 3TB cache. That's a solid server. The price is probably very different when negociated, even the pricing page mentions sales discount.

  • @KucheKlizma
    @KucheKlizma 4 หลายเดือนก่อน +18

    Triple backups is the gold standard since the dawn of computers though. What's more modern is testing backups, similar to end-to-end testing in programming, but it's called disaster recovery excercises.

    • @MeriaDuck
      @MeriaDuck 4 หลายเดือนก่อน +3

      I literally asked about backup restore testing in our company in a meeting today 😂. Glad to conclude it is done regularly.

    • @brandyballoon
      @brandyballoon 4 หลายเดือนก่อน +3

      Well worth going through those exercises. It's amazing how often you end up in a chicken and egg situation when testing a bare metal restore.

    • @manicmarauder
      @manicmarauder 4 หลายเดือนก่อน +1

      Absolutely worth testing before you need it. I just had an "unscheduled" data recovery exercise last night. Thankfully it passed.

  • @mouaztabboush8000
    @mouaztabboush8000 4 หลายเดือนก่อน +27

    Prime: Flip, Flip Zoom in.
    Flip: **Zooms out**

  • @shodanhideo8286
    @shodanhideo8286 4 หลายเดือนก่อน +294

    blank logs really sound dreadfull

    • @scottdwln
      @scottdwln 4 หลายเดือนก่อน +21

      Not enough fiber in the diet...

    • @SomeUserNameBlahBlah
      @SomeUserNameBlahBlah 4 หลายเดือนก่อน +12

      No logs = no recorded problems
      I don't see an issue here.

    • @bravosix6738
      @bravosix6738 4 หลายเดือนก่อน +2

      @@SomeUserNameBlahBlah
      Not an expert here, but couldn't a info level log have averted this?
      Just a line like, "Hey boys, i'm gonna do a bit of a cleanup just like you asked me an year ago... just so you know..."

    • @AntiAtheismIsUnstoppable
      @AntiAtheismIsUnstoppable 4 หลายเดือนก่อน

      @@bravosix6738 I think it's because it was a test tool. It would be logical for a production tool to implement email warning say 3 months before deletion and one month before deletion, mabe a couple of days before deletion, but to set such warning, that's probably another parameter. There's no real need for a (default) warning in a test tool. The idea though that a test tool can be used at all for production, and then at google, I don't know if it's scary or comedic.

    • @Tahgtahv
      @Tahgtahv 3 หลายเดือนก่อน +1

      @@bravosix6738 You mean like the one shown at 8:23? Logs only tell you what happened, they don't stop the thing from happening.

  • @akuma2124
    @akuma2124 4 หลายเดือนก่อน +24

    The mandatory 11.5% Superannuation ("super" for short) is a requirement set by the government to employers, however the money itself (usually) isnt managed by the government. They're private companies that employees nominate to manage their fund for them (even banks have divisions to do it for you). Usually these companies also try to "grow" your money by investing it for you, and most of them make a profit from general account fees and some of that return on investment. People can also nominate to cut a larger percent of their wages to go into the super fund, and is usually pre-tax, hence you can even get away with paying less tax by doing this. The government usually does watch over other things involved though, like under what circumstances you can withdraw from your retirement fund (when you are under the retirement age)

  • @siljrath
    @siljrath 4 หลายเดือนก่อน +38

    "I wouldn't like to be caught without a second backup" -- The Miles O'Brian Principle

  • @PetrGladkikh
    @PetrGladkikh 4 หลายเดือนก่อน +57

    Google does this all the time. Just, well, google it. This case only seems to be the loudest so far.

    • @thewhitefalcon8539
      @thewhitefalcon8539 4 หลายเดือนก่อน +2

      Google uses the CADT development model for their commercial products which is insane

  • @daphenomenalz4100
    @daphenomenalz4100 4 หลายเดือนก่อน +369

    It handles the entire country, so them having that many backups literally saved Australia lmao

    • @connorhsm
      @connorhsm 4 หลายเดือนก่อน +38

      This is factually incorrect.

    • @jackschofield7308
      @jackschofield7308 4 หลายเดือนก่อน +57

      ✨ How it feels to spread misinformation ✨
      There are many Super funds in Australia. UniSuper is among the largest, however.

    • @LeetHaxington
      @LeetHaxington 4 หลายเดือนก่อน +3

      Continent *

    • @NeilHaskins
      @NeilHaskins 4 หลายเดือนก่อน +2

      @@LeetHaxington So no Tasmania?

    • @retardo-qo4uj
      @retardo-qo4uj 4 หลายเดือนก่อน

      Indonesian gov got ransomwared and has no backup. So yeah, complaining about too many backup officially a first world problem

  • @WinterMute99
    @WinterMute99 3 หลายเดือนก่อน +4

    as a 2021/2022 ex googler, internal communications, not any dev team, Google does praise itself a lot as a company, and does a lot of things right for employees, but man do they value numbers and statistics more then people when it comes to any issues

  • @0cky
    @0cky 4 หลายเดือนก่อน +44

    5:18 Something to keep in mind is that not all of that bill is going into the server capacity.
    You are paying for reliability, failover, automation, etc that ensures your services never go down.
    We've had data collection systems that ran for multiple years without any interruptions.
    Not something that is easy to pull off with on-premise solutions, Exhibit A: 6:00

    • @estranhokonsta
      @estranhokonsta 4 หลายเดือนก่อน +7

      Yep. Totally right. It is a shame that in the end such a fortune only got them the garanty of deleted data after 1 year.

    • @mattymattffs
      @mattymattffs 4 หลายเดือนก่อน +2

      And now Google should host them for free for at least a decade

    • @Spencer-wc6ew
      @Spencer-wc6ew 4 หลายเดือนก่อน +1

      It doesn't sound like they were paying for any of that stuff, lol

    • @_owen.c
      @_owen.c 4 หลายเดือนก่อน +1

      Also it’s only the cost of 1~2 employees

  • @themuslimview
    @themuslimview 4 หลายเดือนก่อน +4

    Hilarious End. Does that video really end that way!! Amazing!!

  • @MeriaDuck
    @MeriaDuck 4 หลายเดือนก่อน +3

    10:45 off site backups seem quite the norm to me. The three two one rule: three versions, two media types, one off site still more or less applies.

  • @privatesocialhandle
    @privatesocialhandle 4 หลายเดือนก่อน +7

    It's just a sloppy work by Google. If the task was "Write your name" and it involved $88 billion dollars, I would of spent an hour verifying my OWN name before I commit.
    So, the Google team who ran the tool didn't bother verify it's full parameter set? A tool that is never used manually for production and the risk $88 billions, and they didn't do that?
    Yeah, sloppy reckless work by Google.

  • @4bSix86f61
    @4bSix86f61 4 หลายเดือนก่อน +4

    Google: Collects alot of user data
    Also Google: Deletes accounts to make space for the 'extra' data

  • @CodingWithCompassion
    @CodingWithCompassion 4 หลายเดือนก่อน +12

    Oh my fuck empty logs is the scariest two words you can hear

  • @__NikolaTesla__
    @__NikolaTesla__ 4 หลายเดือนก่อน +2

    the off-side backup is the first thing I cut when I'm too busy, like the previous two weeks. It's a gamble, but my assumption is, if the place burns down we're out of business anyway.

  • @masamaeda9217
    @masamaeda9217 4 หลายเดือนก่อน +2

    The fact that this all happened with no notification and it wasn't "soft-deleted" at least for a little while is very troubling.

  • @Markadown
    @Markadown 4 หลายเดือนก่อน +1

    10:27 - two decades ago I worked with a guy who was rightly obsessive about backups. People in IT would joke about him but he was always right. His stance was that your backup should have backups and you should test those backups to make sure the backups were working. Two decades later, I deal with people who run production environments with zero backups.

  • @User-actSpacing
    @User-actSpacing 4 หลายเดือนก่อน +22

    Backup is a religious talk in Japanese companies.

  • @JGComments
    @JGComments 4 หลายเดือนก่อน +3

    Good rule of thumb: expect people to do boneheaded things and plan accordingly.

  • @theman36951
    @theman36951 4 หลายเดือนก่อน +1

    Who are you to produce such great content. Regular, quality, and educational. You made me want to learn vim.

  • @Tactician00
    @Tactician00 12 วันที่ผ่านมา

    "please dont just say something stupid" while hovering over "stay sexy" lmaoo 5:13

  • @JGComments
    @JGComments 4 หลายเดือนก่อน +2

    What most people don’t understand about insurance is that your premium covers the average risk of the group over the life of your policy, not your individual risk.

  • @tttm99
    @tttm99 4 หลายเดือนก่อน +1

    10:46 We've always have backups for backups. Only now we're looking after our precious newborn babies by wheeling them into an abandoned carpark at night in the bad part of town and asking whoever happens by to mind them ... Because it saves a buck. Insane.
    Absolutely insane. We're right to be anxious about companies relying like this on cloud services. And they should not be rewarded.

  • @TheRealBigYang
    @TheRealBigYang 4 หลายเดือนก่อน +2

    This reminds me of the day I deleted the bin on the production server. Feels like only 3 months ago, because it was. Check the paths in your makefiles before doing a clean build, kids

  • @Volvith
    @Volvith 4 หลายเดือนก่อน +5

    "Oh don't worry, Bob has a script."
    >1 year later<
    "Ooops..."

    • @mhavock
      @mhavock 3 หลายเดือนก่อน

      "Bob left the building"

  • @Consequator
    @Consequator 4 หลายเดือนก่อน +12

    23:00 i laughed with how accurate that is.

  • @Elfcheg
    @Elfcheg 4 หลายเดือนก่อน +5

    As a DBA I often answer questions like "Why do we need backups when we have replica?" It was funny first 100 times.

    • @phillipsusi1791
      @phillipsusi1791 4 หลายเดือนก่อน +2

      This sounds like RAID IS NOT A BACKUP!

    • @smooth1x
      @smooth1x 3 หลายเดือนก่อน +1

      Snapshots are not backups!

    • @Elfcheg
      @Elfcheg 3 หลายเดือนก่อน

      @@phillipsusi1791 yep. High availability is not a disaster recovery.

  • @stidi9649
    @stidi9649 4 หลายเดือนก่อน +14

    From 25:00 till the end is the best primeagen moment of all time. Period.

  • @sambrien1020
    @sambrien1020 4 หลายเดือนก่อน +27

    “Australia is a lucky country run mainly by second rate people who share its luck. It lives on other people's ideas, and, although its ordinary people are adaptable, most of its leaders (in all fields) so lack curiosity about the events that surround them that they are often taken by surprise.” -The Lucky Country, 1964. So no, they won’t sue Google, they plod along into the next foreseeable problem they’ve been ignoring and everyone will forget about this one.

    • @ymx7947
      @ymx7947 4 หลายเดือนก่อน +1

      hahaha

    • @TinBane
      @TinBane 4 หลายเดือนก่อน

      Dude that wrote that died in the 60s. Australia change fucking radically in the 80s. In the 60s, our currency was tied to British pounds, our institutions still tied to British ones on the whole, and we had fucking morons. In the early 70s, we had a massive change that brought in huge reforms and universal healthcare. In the 80s we went all in on trade and technology. You are literally posting to youtube using standards that were invented by Australians. We often didn't invent the "first" thing, but made them good. Whether it's contacts that don't make you fucking sick and hurt your eyes, or wifi that actually works in a house.

  • @pickledparsleyparty
    @pickledparsleyparty 4 หลายเดือนก่อน +18

    "As a cloud engineer myself, I can confidently say this is complete bullshit, it is 100% UniSuper's fault" is the funniest thing I'll read all week.
    My workplace is full of that kind of hubris.
    "As a guy who does something like a part of that for another company, I know EXACTLY what's happening right now."

    • @EnderElohim
      @EnderElohim 2 หลายเดือนก่อน

      why they even use google in the first place when they can own their own reliable server

  • @mito._
    @mito._ 3 หลายเดือนก่อน +5

    how is it default behavior when you can simply set a flag for "--testing" that avoids the situ all together?

  • @cameramaker
    @cameramaker 4 หลายเดือนก่อน +1

    There are likely hard requirements for such funds put together by ICT experts for actors that are in the role of government suppliers. In a non-corrupt world, they do serve their primary purpose - to make the solutions safer, as opposed to a corrupt act of speccing something in favour of a pre-selected supplier.

  • @insainbassist
    @insainbassist 4 หลายเดือนก่อน +1

    I work for the government and, at least at my agency, we don't have that many managers but we do often get directives from on high that are totally out of touch with what's possible. Like, the governor decides to do some program and now we have to scramble to try and get six months of work done in two with only 75% of the business logic nailed down lol

  • @KwincksIT
    @KwincksIT 4 หลายเดือนก่อน +2

    I'm always telling ppl to backup their cloud data. And i get 'but why, its backed up by X(not actual X)' and i laugh.

  • @yintercept4612
    @yintercept4612 4 หลายเดือนก่อน +1

    One needs multiple backups. One of the backup needs to be in a facility in a different town ... or maybe a different state in case a regional catastrophe occurs.
    A great way to do this is to have a local back up then make a back up of the back up.

  • @amitkarmacharya4493
    @amitkarmacharya4493 4 หลายเดือนก่อน +2

    Politician working @ google: Yes, another bail-out. We are going to be rich.
    Uni: We have backups.
    Politician working @ google: We need an executive order to get rid of that.

    • @Cadambank
      @Cadambank 4 หลายเดือนก่อน

      😂😂😂😂

  • @MrSomethingred
    @MrSomethingred 4 หลายเดือนก่อน +1

    Super in Australia is great. Unironically it holds up a massive chunk of our economy, because every single Australian is a "investor" putting 15% of their income into stonks and infrastructure

  • @gsgregory2022
    @gsgregory2022 4 หลายเดือนก่อน +18

    ...No American remembers when America had pensions. No wonder the worker class is getting fucked.

    • @_sola1re_
      @_sola1re_ 4 หลายเดือนก่อน +2

      my dad is 49 and just retired after 30 years at Ford Motor Company
      I may not have a pension but we're sure as hell lucky he does. My sisters are well taken care of thanks to this.

    • @lashlarue7924
      @lashlarue7924 4 หลายเดือนก่อน +3

      what's a pension?

    • @_sola1re_
      @_sola1re_ 4 หลายเดือนก่อน

      @@lashlarue7924 essentially it's a really great retirement plan that's pretty much been destroyed in america.

    • @colinstu
      @colinstu 4 หลายเดือนก่อน +2

      ​@@lashlarue7924something between SSI and a 401k/IRA.

    • @davidk.8434
      @davidk.8434 วันที่ผ่านมา

      I work as a SWE for an insurance company, everyone gets a pension after 5 years. My seniors have been there for 10-30 years. Real pros.

  • @suyashsngh250
    @suyashsngh250 4 หลายเดือนก่อน +1

    I work for the company which helped them recover this data. These things happen all the time, so probably a good idea to have a solid backup/recovery system in place.

  • @stefanjud6345
    @stefanjud6345 4 หลายเดือนก่อน

    10:55 you need Backups not just if the provider fails - but its good to know if there is a cyberattack. They usually target backups. Having additional, not linked ones is often a blessing.

  • @Denominus
    @Denominus 4 หลายเดือนก่อน +2

    This reeks of some team at Google provisioning a customer account with an internal tool designed and intended for development teams. The default expiry is obviously there so stuff gets cleaned up and not left to rot.

  • @MikkoRantalainen
    @MikkoRantalainen 4 หลายเดือนก่อน +1

    I see archived backups of backups mandatory for any important service. However, the thing that most companies fail to understand is to estimate the recovery time in worst case. If the recovery time is too long compared to the losses of income during the recovery process, the company may still die for it and then the theoretical possibility of recovering data perfectly no longer matters in reality.
    The big question is should to do regular test recoveries from archives? With my current work, we only regularly test recovery from main backup, not backup of backup.

  • @nuclearmedicineman6270
    @nuclearmedicineman6270 4 หลายเดือนก่อน +92

    Having backups for your cloud backups is like trailering another car behind your car everywhere you go just in case.. and everyone around you thinking it's perfectly normal behavior.

    • @alan5506
      @alan5506 4 หลายเดือนก่อน +28

      Having another car behind you, for individuals, is expensive.
      So using that as metaphor is a bad metaphor.
      Backups are a cheap kind of insurance. It's normal and needed since the world is chaotic.

    • @fyfaenihelvete
      @fyfaenihelvete 4 หลายเดือนก่อน +15

      I don't think you know much about this topic OP.

    • @lynx-titan
      @lynx-titan 4 หลายเดือนก่อน +1

      no it isn't 😕

    • @TJackson736
      @TJackson736 4 หลายเดือนก่อน +3

      It's the cost of not providing your own infrastructure.
      Even then, you will want some backup processes.

    • @adammiller9029
      @adammiller9029 4 หลายเดือนก่อน +3

      Incorrect. Most businesses are going to opt for hybrid cloud for 2 reasons. 1 Backup/failover services. 2 Broader service offering. its normal for anything mid size up to have backups with another cloud provider for anything that is truly mission critical. Large businesses should have a disaster recovery plan that considers the possibility of 1 cloud provider getting blast off the face of the planet. Mid size business should have DR strategies that move between different geographic regions, even if they opt to use the same provider for DR.

  • @T1Oracle
    @T1Oracle 4 หลายเดือนก่อน +1

    Imagine trying to catch up on decades of Google docs in the 2 year time-span between when you start, and when you go somewhere else? It's absurd! Just reading the docs isn't enough, you need to put it into a meaningful context. That will always take time.

  • @jasoncox7244
    @jasoncox7244 4 หลายเดือนก่อน +1

    in a small business operation, my boss has anxiety about data loss... so I run our local file server with local replication in Hyper-V in separate physical locations on campus, Backed up to a Local NAS, and cloud backup with iDrive. ... My server capacity per end user is obscene. it's f&^%& hillarious... especially when I get to take home the hardware that depreciates to zero and gets replaced.
    (edit) P.S. I also have 2 managers in my 4 person dept.

  • @lunalect
    @lunalect 4 หลายเดือนก่อน +1

    Best prime video of all time, thanks bruh

  • @Alpha725_
    @Alpha725_ 4 หลายเดือนก่อน +1

    As someone who works with super funds and with financial institutions 3 backups is very very normal and required. We also have regular simulated disaster recovery tests were we fall back to the recovery environment. Some will even switch every year between the two

  • @TheDaviddoodles
    @TheDaviddoodles 4 หลายเดือนก่อน

    Guys. I love this show, I always enjoyed the humour and the serious conversations that are properly had to set us all to a better future... Bro is definitely funny... Awesome stuff man... Hope all the best !!!

  • @UnfiItered
    @UnfiItered 4 หลายเดือนก่อน +1

    Always flag your test code for deletion.
    Function:
    -check status of project
    -if project is production then
    -alert, flag these functions for deletion

  • @Aberusugi
    @Aberusugi 4 หลายเดือนก่อน +3

    Im glad you reacted appropriately to the procetag. Its the reason we stay on exchange instead of moving to office 365 at our company. Moving to office 365 would cost us more in the long run than our entire combined it salaries

    • @dodibtw
      @dodibtw 3 หลายเดือนก่อน

      for some companies it could be cheaper to host on cloud/datacenter providers due to space/security constraints

  • @dancarter5595
    @dancarter5595 4 หลายเดือนก่อน +1

    Financial institutions have strict regulatory requirements for disaster recovery.
    A separate vendor for DR backup is likely to have been one of them.

  • @seculi7757
    @seculi7757 4 หลายเดือนก่อน +3

    Probably the developer of the tool got an email/reminder in his inbox to extend the time on the pensionfund`s cloud, but after leaving the company no longer had access to his company email.

  • @camillaericajuliarandal332
    @camillaericajuliarandal332 4 หลายเดือนก่อน +4

    The fact that they didn't have the tool force you to specify the flag is just bad coding standards. Any tool even if it is only for internal testing just make the tool force you to specify --development/testing --production and never have a default for those flags

    • @Labushu685
      @Labushu685 4 หลายเดือนก่อน +2

      Especially if you are ever going to use it in prod. You either NEVER use it in prod, or you do in special cases. If you do use it in prod, there can't be such a default case.. it has to be a prompt. If you have to use the tool in prod, it's obviously because the client needs much higher resources than what the normal dashboard allows, so it's gotta have extra special attention. And also, why no reminders that it's set to delete? Why is that date not shown anywhere? Why are they starting such a big client instance seemingly "on a whim"? Even if they did check the command, they gotta also double/triple check the end-result.. just so very sloppy.

    • @davidk.8434
      @davidk.8434 วันที่ผ่านมา

      This is a flaw in management. Coding standards, tests, specs, all slow people down. Screw it! Use an internal hack tool for prod. Move fast, break things.... Unprofessional culture. Fish rots from the head

  • @andythedishwasher1117
    @andythedishwasher1117 4 หลายเดือนก่อน +10

    Bro you've never used VMWare? Honestly pretty sick possibilities. There's a learning curve, but it's fun.

    • @Kane0123
      @Kane0123 4 หลายเดือนก่อน +3

      Steep learning curve. Even steeper pricing recently!

    • @DerekSmit
      @DerekSmit 4 หลายเดือนก่อน

      ​@@Kane0123is an open source alternative available that is close in functionality? Would be cool to play with a bit

    • @DoctorMandible
      @DoctorMandible 4 หลายเดือนก่อน

      ​@@Kane0123how does openware compare?

    • @andythedishwasher1117
      @andythedishwasher1117 4 หลายเดือนก่อน

      @Kane0123 Yeah you're not wrong there. I'd love to see some equivalent open source handles.

    • @NostraDavid2
      @NostraDavid2 4 หลายเดือนก่อน +1

      All I remember from VMware from 10 years ago were the idiotic prices for broke me, so VirtualBox it was and I never bothered using anything else.

  • @grmpf
    @grmpf 4 หลายเดือนก่อน +2

    I think it's kind of funny that, when UniSuper initially only called it a "third-party service provider", pretty much everyone probably imagined some awful company with great B2B marketing but low tech quality that never stood out with anything good ever, and then it turned out said third-party service provider was literally Google. 💀

    • @davidk.8434
      @davidk.8434 วันที่ผ่านมา

      Arguably that also describes Google

  • @chrismingay6005
    @chrismingay6005 4 หลายเดือนก่อน +2

    I bet the dev(s) that pushed for the third party provider backups is feeling mighty proud of themselves now

  • @AG-ur1lj
    @AG-ur1lj 4 หลายเดือนก่อน +1

    24:48 - How could insurance be anything but a scam? How could you possibly profit off paying for my car wreck or medical procedures, except by charging more than they cost to cover them?

  • @jeffreyblack666
    @jeffreyblack666 4 หลายเดือนก่อน +1

    Backups for backups is standard. Even long ago, the standard was onsite backups and offsite backups.

  • @geforex28
    @geforex28 4 หลายเดือนก่อน +1

    Im sure it’s the manager who took the bonus saying « I took the decision to have backups, when my engineers advised for it »

  • @Kane0123
    @Kane0123 4 หลายเดือนก่อน +29

    Australia mentioned let’s go!

  • @18ccase
    @18ccase 4 หลายเดือนก่อน +1

    Ive had to start doing back ups to the backups. Ive been screwed twice now from not having a third backup, and at this point, im afraid something might happen where i need a fourth

  • @memenchomezov
    @memenchomezov 4 หลายเดือนก่อน +1

    Conclusion: Never write a code/tool that will delete the database also the backup at the same time. If everything needs to be deleted automatically then delay the backup deletion with at least one week (it still will be deleted automatically, but if it turns out that it should not be deleted developers will have one week to stop the automation).

  • @youtubeuser1052
    @youtubeuser1052 4 หลายเดือนก่อน +1

    Public cloud isn't cheap, but you can't compare the price to just the price of buying a computer. You can't run a national pension fund on a computer sitting in a corner of your garage in the original cardboard box without being plugged into power, without any cooling, without any network connection. You need to add in the cost of proper data center space, power, cooling and networking. It's still not cheap, so it's more appropriate for temporary capacity.

  • @Peaches4Rent
    @Peaches4Rent 4 หลายเดือนก่อน +2

    If this is an internal tool then the reason for deletion would be because internally engineers can spin up a server which send reminders to owners for renewal

  • @IvanDeMarino
    @IvanDeMarino 4 หลายเดือนก่อน

    Writing documents for nobody to read but for your level uppers to approve is the CORE of working at Google.
    You read the situation perfectly mate

  • @thisiskdn
    @thisiskdn 4 หลายเดือนก่อน

    Never thought I'd get to see Prime impersonating Alexis from Schitt's Creek, complete with hand gestures and the voice. Mad props my man

  • @DoctorMandible
    @DoctorMandible 4 หลายเดือนก่อน +9

    "Backups for your backups" shouldn't be necessary for such an expensive service.

    • @thecollector6746
      @thecollector6746 4 หลายเดือนก่อน +1

      Amazon RDS if I remember correctly keeps 6 back ups of data......supposedly. I don't know what Google's persistence policy is....or if the data bases that were deleted are user or platform managed.

    • @blackguy-ij7ij
      @blackguy-ij7ij 4 หลายเดือนก่อน

      💯

    • @maskharat
      @maskharat 4 หลายเดือนก่อน

      @@thecollector6746 Even if google had 20 backups, when the command to delete everything rolls in, everything, including those 20 backups, will be gone. A backup for a cloud service provider is a second cloud service provider.

    • @Doogle7821
      @Doogle7821 4 หลายเดือนก่อน +1

      actually pretty smart when your financial firm controls pensions for an entire country. I don't care how good or big the 3rd party data vendor is you never put all your eggs in one basket when talking 100s of billions of dollars

    • @Tw33ty271
      @Tw33ty271 4 หลายเดือนก่อน +1

      This is like, risk management 101, and they almost certainly had a legal obligation to do so for exactly this reason.
      The concept is called vendor risk management, and the reason you need to think about it when this much money is involved is that even if Google does everything right, there's still inherent risk involved.
      For example, what if they went out of business? What if there's a regulatory or sanctions change that prevents the vendor from operating in your country?

  • @wdavid3116
    @wdavid3116 4 หลายเดือนก่อน +1

    Unisuper does seem like a spot on company in their handling of this situation as discussed. It likely should be noted that given the company is a major financial provider handling billions of dollars it almost certainly is that way due to stringent government enforced requirements placed on them. I would be very surprised if they would legally be allowed to operate without having secure offsite backups.

  • @steffenbendel6031
    @steffenbendel6031 4 หลายเดือนก่อน +4

    Worst weather forecast for cloud users: it will be sunny!

  • @vishaldinesh
    @vishaldinesh 4 หลายเดือนก่อน +1

    "don't roast us" I know I'm on TH-cam but can't stop laughing XD XD

  • @Shogoeu
    @Shogoeu 4 หลายเดือนก่อน +1

    Renting looks expensive, but remember it includes networks, UPS and support.

  • @cthecheese1620
    @cthecheese1620 4 หลายเดือนก่อน +1

    Just want to throw out there that having 1 back up physical, digital, and off site has been normal for over 20 years.
    In a world where back ups are one dumb CEO away from being considered not necessary, we should not consider the idea that someone has too many back ups.

  • @Exilum
    @Exilum 4 หลายเดือนก่อน +1

    The end was absolutely hilarious

  • @tux1968
    @tux1968 4 หลายเดือนก่อน +2

    When a company is more interested in censorship and DEI than backups.

  • @letopizdetz
    @letopizdetz 4 หลายเดือนก่อน

    According to their site they have 800+ Employees and 647k clients. I know some financial institutions still making old school backups on tape rolls and storing them underground in vaults. you know, just in case. Some jurisdictions require you to go waaay back in case of a trial / investigation.

  • @youtubeenjoyer1743
    @youtubeenjoyer1743 4 หลายเดือนก่อน +50

    Oopsie. We are sowwy 😢

  • @deraxelturrelkeign
    @deraxelturrelkeign 4 หลายเดือนก่อน

    @2:41 forced company contributions are coming out of your compensation. Even other forms of company contributions are coming out of your compensation effectively. The benefits for other forms is that they are pretax or not taxed.

  • @nobbyfirefly57
    @nobbyfirefly57 3 หลายเดือนก่อน +1

    Google when they fuck up: oops! Sowwy! 😊
    Google when you fuck up: 🤬

  • @kuhluhOG
    @kuhluhOG 4 หลายเดือนก่อน +4

    24:06 ABM = Arbeitsbeschaffungsmaßnahme
    translated: work procurement measure

  • @autohmae
    @autohmae 4 หลายเดือนก่อน

    15:41 I think it was meant to create demo/test environment for their customers to try out the service, because a year is very long for a developer to do testing with creating/deleting.

  • @nighthawk6468
    @nighthawk6468 4 หลายเดือนก่อน

    "Is our data safe??? Has there been a breach???"
    "Oh trust us your data is safe, no one has it... not even us!"

  • @3Rton
    @3Rton 4 หลายเดือนก่อน

    Nothing in this video was as unexpected as the robo boy switching to rap god for the finisher, and that includes prime miming Eric Cartman

  • @wadecoldkiller
    @wadecoldkiller 4 หลายเดือนก่อน

    woah finally Kevin Fang gets a spot in here. da channel is also my fav

  • @KaffeeSpot
    @KaffeeSpot 4 หลายเดือนก่อน +1

    Just remember. The Cloud is just someone else's computer

  • @KilgoreTroutAsf
    @KilgoreTroutAsf 18 วันที่ผ่านมา

    Someone trusts full access and responsibility over the personal information of a public fund to a third-party, private company based on another country and nobody in this chain of decisions asks any questions or raises any concerns.
    We have officially reached peak stupidity.

  • @untouchable360x
    @untouchable360x 4 หลายเดือนก่อน

    Every company prevent this by paying for redundancy but they don't want to double their costs. It's a leadership decision.