You did WHAT with the dead SSD? And the data?
ฝัง
- เผยแพร่เมื่อ 30 มี.ค. 2022
- Wendell got his hands on a corrupted Fusion io NAND flash drive, and let's just say, it consumed his life for a little bit.
**********************************
Thanks for watching our videos! If you want more, check us out online at the following places:
+ Website: level1techs.com/
+ Forums: forum.level1techs.com/
+ Store: store.level1techs.com/
+ Patreon: / level1
+ KoFi: ko-fi.com/level1techs
+ L1 Twitter: / level1techs
+ L1 Facebook: / level1techs
+ L1/PGP Streaming: / teampgp
+ Wendell Twitter: / tekwendell
+ Ryan Twitter: / pgpryan
+ Krista Twitter: / kreestuh
+ Business Inquiries/Brand Integrations: Queries@level1techs.com
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Music:
Intro and Outro: "Earth Bound" by Slynk
Other Music:
"Follow Her" and "Lively" by Zeeky Beats
[Cocktails and Lobsters] by Alexander Nakarada (www.serpentsoundstudios.com)
Licensed under Creative Commons BY Attribution 4.0 License
creativecommons.org/licenses/... - วิทยาศาสตร์และเทคโนโลยี
![[Full Episode] MasterChef Junior Thailand มาสเตอร์เชฟ จูเนียร์ ประเทศไทย Season 3 Episode 4](http://i.ytimg.com/vi/e7Oqqo90x_w/mqdefault.jpg)
This is Wendell going full Wendell and I love it. This is why Level1Techs is top tier tech TH-cam: genuine expertise and ability, not just knowledge of properties of the product stack
This recovery adventure BADLY calls for a written piece on the side of it. a lot of background info was clearly cut from this, and an article could provide that, because the subject really deserves it...
The editor wrote at 2:33 that the background info is going in a dedicated video, so you are probably going to see that too in a bit
If only L1 had some kind of dedicated presence where users could interact with each other in written form ;)
@@abavariannormiepleb9470 ahahahah the problem with the forum is that i don't have much to contribute 😅
@@salmiakki5638 Its ok. Just come lurk more.
@@abavariannormiepleb9470 Blergh, forums. Especially ones that use the godawful Discourse. FUCK that.
Wild that you got that mostly recovered from that insanity
The abyss was staring back, for sure.
@@Level1Techs Are you curious about such stuff in general? Got a WD DC SN630 7.68 U.2 that kind of died during a regular reboot. “Kind of” because the device still shows up but as raw and as a strange symptom all SMART data has been reset. Tested powering it on for 24 h but that didn’t change a thing. Didn’t try a secure erase yet (like that faulty Intel SSDs about ten years ago) since I wanted to experiment if the data was somehow recoverable.
26:03 minutes ago i thought i was pretty smart. I was confident in my abilities as an network architect
Ive had a long career working in IT.
My confidence is shaken i have a serious case of imposter syndrome
Thanks Wendell
Modern data storage is wild, and cursed as fuck. He's lucky that it wasn't a year 2022 drive - it has only gotten worse since this piece was made.
Funny thing is, parchment and old lead or iron based ink is a far superior storage media than all of our hightech storage solutions. There are many old tomes still existing, even papyrie is better. Harddrives truely die, when their interface vanishes from the market. Ssds are black boxes of data mincing which, if have encryption on board and their memory controller dies, takes all the data with them. Nas, san and all the abreviations are just data mincers prone to silent corruption. Documentation of all kind of digital storage and software based storage solutions is inherently bad and almost always proprietary inhouse stuff. You're better and safer if you keep your personal important data printed out. If you're a middle tier company, you're Fed, you have to rely on poorly written software with bad documentation and proprietary hardware with no documentation, et voila you go full wendell one of these days. The absurd thing is, the paranoia meter increases with every level of additional "data safety", is the data you stored truely and correctly written to the server? Is what I get back, the stuff I gave it? I for me personally have come to the conclusion (for company data), that a regularly made hard backup on a new usb drive every two weeks is a lifesaver for just these situations. What do new 2tb 2,5" usb 3 drives cost against lost or corrupted company data? Nothing in comparison.
Wish I had more info to provide on this, but even in my 7 years of doing iodrive installs and support, plus all the iodrives in my home lab, never ran into an issue. I did have an i7-920 as my first home esx 5 server with an iodrive though.
This ties into my biggest concern for our future. There are one of a kind movies,pictures, and music from 100 plus years ago that we can see and hear today because they were recorded on a analog medium. In the old days grandma had one photo made in her entire life and we can see it but today grandma may have a 100k of pictures but we can't see them because a HDD died.
This is long talk in library science about preservation of documents for future generation. Paper can stand thousands of years but bits in magnetic form or electric charge don't.
@@pubdigitalix Look up "5D optical data storage," or (for some reason) "Superman Memory Crystal." The problem is more or less solved if we can make the equipment affordable in some way.
Nice Wendell. When I got my first NVMe drive I found the namespace and raw access interface and had some fun ideas about what I could do with that, ie slicing it up with namespaces and using a raw flash filesystem on the raw sectors of one of the namespaces, like JFFS2 or logfs. The neat thing about that would be that the filesystem would be completely invisible to all but the most technical of users. BTW I don't think the ECC data is what you're looking for, or will ever find. That's at a lower layer than what you're accessing. You're looking for the FTL (Flash Translation Layer) data which provides the LBA block mapping and manages the SLC cache blocks and the over-provisioned blocks to give performance and absorb media errors similar to how HDDs have spare blocks to mitigate media errors, except SSDs have a lot more. Ok and having watched a little further... you didn't need to load it all into a database. You only needed to write the metadata to the database - that is, the LBA and the byte offset into the full device image. Then you could use the database to linearise the LBAs and stream them from the device image to a nice linear image, which you could then access via loop mount. And instead of a database you could have use a MMap'd file with the space for an array of the LBA locations... that would have been orders of magnitude faster than a DB.
uptooting for visibility .
You wouldn't even need to copy from the device image to a linear image. You could just make a device mapper table that does all that lifting for you in the kernel on the fly.
Or you could make a FUSE module that serves up an image file on the fly, since doing all those individual sectors (due to the sector headers and such) may be a bit much for the device mapper system.
@@Coderjo. Wendell's ultimate goal was to end up with a device image in normal linear LBA order. Device mapper can't achieve that. For starters, device mapper doesn't handle 512 byte or 4K mappings so it's not even capable of the granularity required. Research Flash Translation Layer and you'll better understand what Wendell was trying to undo.
Shocking, Microsoft tried to create a resilient file system that mimicked ZFS and did a piss poor job? Who'd have thought that would happen...
Same thing with Btrfs actually...
Wendell casual ‘janitor’ casually do static analysis and comparative analysis and somewhat successfully reorder a shit ton. Of data , ok the approach is kinda brute forcy but omg great work… that wendell is too humble
At 520 bytes per sector (including 8 bytes of OOB data), I wonder if the FPGA handles ECC using a part of the OOB data that is hidden from you. Looking at a Micron 512Gb (64GB) datasheet from Datasheet Archive, it looks like the actual OOB data (744 bytes per 8k page) would be 46 bytes per sector, not 8 bytes per sector.
Exactly this, Wendell did not really get a raw nand dump, just a layer above where the FTL does bookkeeping.
Thanks for the snip editor Autumn! Will be waiting for part 2 :-)
Maybe the sequence just always adds one, ignoring overflow, assuming the GC would catch it before it loops again. So instead of grabbing the highest one, you'd need to grab the highest sequence value, that doesn't have one IMMEDIATELY after it (or in the first sequence, in the case of sequence.MAX_VALUE). IE if 8 was the max, and you had 1, 2, 7, 8, you wouldn't use Sequence 8, you'd use sequence 2. Just a thought! Very interesting content!
I had the same thought, though I was figuring on identifying which sequence numbers where introducing corruption. Presumably these would be the very highest numbers, and it would be possible to draw a line below them.
Awesome stuff! Thanks for posting, I know that just added to the overall stress of having to keep all this in your head and moving forward!
Love this sort of deep dive. Hope to see a future update video/article once you wrap it up!
Did something similar many years back to cope with ntfs corruption that was unrepairable on a friend’s failed platter drive. Partial success, but a fun educational exploration.
I love that this came out on world backup day. Fantastic work. Makes me want to dig up my old 80 and 160gig FusionIO drives and see if I can get them running.
Python can handle concurrency just fine if you use the Multiprocessing library (included in the Python standard library). Unlike the Threading library, it launches multiple python processes in order to make sure you're not time-slicing a single CPU core. These processes are not bound by the GIL, so in order to pass data around between the "threads", you need to use the data structures provided by the Multiprocessing library. I've found that producer/consumer thread models work quite well with this library.
When was that added to the Python STL, though? I don't know Python that well, but he might have to use Py2.6 or something, and if the MP lib is newer than that, it doesn't help.
@@jayands Check Python PEP 371. It was added around 2008
@@jayands 12:47 He used Python 3.10
Fantastic work on a fantastically interesting problem!
The "Jeff" from CraftComputing did a video about these drives some time ago, he got a bunch of those drives of various gens and even some software, maybe he knows something
This video is awesome. I love the depth!
I think using a database is the heaviest way of doing this, computationally speaking.
If you actually have the raw flash dump and you have different versions of various blocks then I would have first created a full 1.5TB image using the first-found method for any block. When a collision happens you create a file for the conflicting block and an info file stating where it came from in the source flash dump (or just do it with the name itself). You would track the blocks, of course, with a simple bitmap - 8 bits per block would be better than one so you can store a few flags - found, conflict, erased, etc...
A single pass through the flash dump and you'd have a 1.5TB (corrupt) image file that will represent the raw data you want to recover, a bunch of 520B files representing potentially newer blocks where conflicts exist, and a bitmap with your state per block.
Your next move is to then determine which of the conflicting blocks is the correct one and write that data into the image. The final move would be to find the erased block data and zero out those blocks in the image. Then write the image to a drive and try to mount it.
... all this, of course, assuming I'm not missing something...
really like to see more support for fusion io drives. I have been using many of the sx300/350 card. Those things are tanks compare to NVMe drive. Their write life is in PB range, long pass their EOL/EOS date.
don't MLC based nvme ssd also have endurance in PB range?
@@lordjaashin yes, But fusion io will not reduce speed during continue write.
Check out the Micron 9300 MAX series, extremely durable NVMe drives.
I've been doing computer janitor stuff for a while, so I know what an FPGA is. I still hear "field programmable gatorade" every time, though.
Maybe in the future we'll be replenishing nanites by drinking some field programmable gatorade :P
I am too dumb o understand anything in this video but man watching Wendell work is a treat!!! Computer janitor of the highest order!!!
🤘
Love that editor's note. GG Autumn. Lol.
... Will be waiting for that video.
I look forward to seeing the awesome engineering behind the fusion io - viewer chris
The opening comment of wondering how I got here every morning nailed my introspective funny bone. Need more of that humor!
I just got about the gist of the whole saga ... but i love how Wendel explains everything even if i don't understand most of it. That's F.A.S.D for yah !
Amazing Wendell. I just got rid of an I7 920 with an ancient Ubuntu 8.04.. This thing could not die !! Great to see something like this being still usefull.
The Jigsaw puzzle comparison is cool!
Wendell, if you have a checksum of any file that was on the drive (i.g. ubuntu usb image), that can be used for validation during the data reconstruction phase.
No matter what, it looks to me that this challenge has shaved 10 years off. You positively look youthened by the experience!!!
What an adventure! Thanks!
wendell and his custom ascii art lol
Those things were the dream back in the day. Insane IO.
My old company destroyed tons of these in our DB servers. Didn't stay very long in the boxes....
Fusion IO are so cool. We used them in our VDI cluster for a long time.
Should we call the data recovery guy?!? I AM THE DATA RECOVERY GUY!!!!
Props Wendell, that's awesome 😎 also Autumn, will we get the awesomeness of the device video?
If you used your dumped sectors directly as a virtual disk and it detected a valid filesystem on it, and could read data, then it is very likely that you did not really get a completely raw flash dump.
The ECC algorithms that I am aware of are stored in a pattern, that does not really resemble the real data. You have to pass that block (or subblock if ecc comes in multiple steps) to the actual algorithm to get out sensible data.
I guess your dump is from blocks which already did go through ECC. For a real raw dump you want the raw nand flash blocks and there OOB (out of band data).
Your data is probably just a level above, that is seen by the upper parts of the flash translation layer -> bookkeeping where each eraseblock / block is, logs etc...
Here is a commonly used algorithm for ECC on nand flash, used in both raw nand mtd devices with linux (cheap storage for embedded devices), and apparently also in proprietary ssd controllers: en.wikipedia.org/wiki/BCH_code
There is this document, which can provide some insight from HW side: www.onfi.org/-/media/client/onfi/specs/onfi_3_1_spec.pdf?la=en
Maybe look at a previous version this one might be too new.
For example the "factory marked bad block" info might just work exactly as described by the standard.
superb video
Oh man, respect, one can really go crazy with such Sisyphean work
Good job!
good lord this was hard to follow. i love it.
I am happy when Wendell throws in a "wheeeew!" at the end of a video.
Ahh this reminds me so much on my personal experience with my first nas in 2009, 6tb in 4 drives jbod'ed , one drive failed... bruh. I was greedy, wanted full capacity of 6tb, got what I deserved, painstakingly recovered the remaining 3 drives with many files being trunkated. Learned from my mistake, switched to raid 5, had only 4.5tb but safety, added another layer of safety by buying a second nas, back uping the first one. Data you DO NOT HAVE doubled, is data you do not care about. Had one of those nas fail again, but laughed because THERE WAS ITS sister nas. Anyway, it managed to recover itself when I swapped the failed drive. I do never want, to manually repair filesystems, copy blocks and chain chunks again, NEVER. That was the price I paid.
Damn... I hope the client paid a lot for that one.
those 45million pieces were obviously just optional extras. nice jigsaw work!
I have that same i7 920 system Asus P6T Deluxe board. It's running as a home hypervisor now. It will not die!
I grabbed two of these drives out of the scrap bin at work. Never got around to poking at them. Super cool. Do you want them?
@2:18 The look of Love!
Right on!
The one I have from that era is a OCZ Revo Drive 240 GB. They were purchased by Toshiba, who then did not update drivers so I put it in a server.
If I am remembering right, ReFS does not enable file data integrity chksum by default, only matadata. That might explain why it is not reporting errors. I once had to retroactively go through an existing volume to turn it on for existing files since turning it on for the volume does not affect existing files. Weird design choice when "resilient" is literally in the name of the fs.
Yep, that's the default unfortunately. Have to enable it per volume through powershell.
It's even worse if you have the second option enabled enforced (witch it is by default and force enables on every new file created regardless of enforced folder setting) once the integrity enabled (set to true)
if you get 1 bit of checksum fail in a file it removes the whole file from the name space and doesn't actually free up the space you have to delete the volume and recreate it to get the space back (all you get is an event log that it did it and a resulting missing file)
This was a year or 2 ago in whatever Windows server version for that time
@@leexgx I don't think I experienced that myself. But ReFS did protect me and warn me when a file failed the integrity check. I deleted it and pulled from another ReFS protected backup to replace it.
I was on WIndows Server 2016.
Though I did notice a suspicious amount of space being taken up... maybe it was that and I didn't know it.
You can make a movie out of this: Raiders of the Lost Data... 😏
🤭🤣🤣
Editor Autumn's editing style is starting to become rrcognize, just like Editor Amber.
Thanks for slugging it out in this one dear Editor. I love when the team can nerd out on something unrestricted.
Wendell, the Indiana Jones of ancient NAND data.
11:32
TH-cam auto CC be like: Jupp, "little indian" is the logical choice here.
The last time I had to do anything hairy with data recovery was early 2000. Reading interesting SunSolve articles for fun (and having a much better memory than I do now) saved my bacon.
Great video, though really would have liked to get more backstory on why you needed to recover anything in the first place? What was the drive used for originally anyway?
This video makes me smile.
I think I had an i7 920 in the PC that I built in 2009. Served me well for many years.
Good job.
Wonderful video! I'm guessing you know this already, but ReFS by default only checksums metadata.
This is a marvellous video. Delicious.
i have no idea what you said, but it was very interesting lol. think i should go back to eating paste now.
Yes! More like this ”show and tell” story time.. I like this video.
My God, Wendell...you could cure cancer! And this is the kind of stuff you do just for fun?! You sir, are a (very) humbling inspiration to the rest of us who used to think we knew quite lot about computing!
No problem Autumn. Maybe next week for that part?
Well this took me back to DOS and early Windows disk / (manual) file recovery using Disk Edit. Worked surprisingly well by just replacing one char in the fat to "recover" deleted files :)
Omg 😆 this takes me back to changing words in my c64 games by starting at track 18, then paging till if find what I want elsewhere
sounds like fun, I had some valuable source code on a Encrypted HDD for a customer. Being a old school system guy I went down a rabbit hole to recover the data without using the recover company laughable quote ($$$$). What a pain, so good job.
Nice job.. nice puzzle.. i would hate&love to do that.
@Wendel, lol i know this is an onder video, but.. those sector '5M'-ish that you found so many of.. could those point at Error-correction blocks? Or sectors/blocks that are relocated or erased? Or maybe a few blocks in that sector that are located in another nand block, to prevent write-wear on those nand-cels as not the whole row would get written.
I have neither the time nor the patience to do this. Wendel is a legend
You Sir, are a braver man than me, and I love you for IT -:D
Have a look at using asyncio that is in the standard lib as well for concurrency
So FusionIO is still from the times when flash controllers didn't apply some AES for every page written? That's what makes it possible to approach them the way you did.
On the modern side, I've seen some nvme failures where 4TB nvme device froze and came back after reboot as a ~300MB device. Didn't spend much time on trying to figure out what went wrong ...
I wanted to say something - however my brain has left my brainpan at about 12:55 when Wendell was speaking about garbage collection placing data someplace else.......
I want that video on the engineering of the drive, please.
yes please wtf
Hey, did you try R-studio on that raw image? IIRC, they have pretty extensive toolset for data recovery from solid-state media
Heh... I've reconstructed countless arrays, linux md, lvm and proprietary ones... that is much easier than reconstructing the sector mapping. For arrays there pretty much always exist multiple copies of the data, and you have to figure out which one is the least inconsistent... recovering sector mappings involves some interesting combination of filesystem knowledge, and being able to verify the validity of data in files mapped by that filesystem... been there as well. Was quicker than the parallel (but also succesfull) restore of the backup if you know what you are looking at.
I have the same copper zalman cooler try it on Ryzen it works great with the am3 compatible horns
14:00 that sector with 150 000 coppies of one sector I wonder if data inside is especially bad for the ssd health.
it really sounds for me like software USB killer kind of deal. Sector writen in a loop that never responded with correct crc. Maybe even that's what killed the drive. I wonder if that is possible.
I've got a 768gb one of those from an HP server being binned. Some of the chips are plastic imitations just so the card looks cool still :)
@6:48 to 6:55 rubbing your bare fingers against those flash dies isn't going to help with your corrupted data issues. That's pretty brave of Wendell to do that considering I'm to scared to even touch/install my SSDs without grounded anti static mat and anti static wrist strap.
He dumped the data allready, so i think its no problem
What's the flashing blue light over the monitor for? Just curious. It's my nature.
Python has multiprocessing package and gevent for cooperative concurrency. Running "many processes via bash script" it's a bit weird :)
After this experience be sure to only use GRAID Supreme for superior data availability ;)
Some years I had a 3 SSD RAID 0 where 1 of the 3 striped drives failed. A tech place said they'd look at it for $4k with no guarantee. (At a quarter of my income that year, I passed.) They said the Indilinx Barefoot controller in the drive had failed so the data could not be accessed. Any idea now of how it might be accomplished today? I had one program on it that I really wanted to retrieve.
Happy World Backup Day everyone. How appropriate to release this video today. ;)
Well that's bad news for my 24 TB ReFS Veeam backup volume.
Great video, really interesting!
Yeah, test your backups. You may be surprised.
@@Level1Techs Actually, now that i think about it should be OK. ReFS was built on a iSCSI LUN backed by ZFS.
ReFS supports fast clone and that weighed out the other cons of doing such a thing.
24:42 "This has not been a fun experience" - I don't believe you, we know you 🤣
Woot!
Wendell you are a Legend !! I have some repair tools ... they consist of a Simple Prayer to God... But for this problem God will struggle.
It was my understanding that solid state used a file location system sorta like and is but simple, so no location data was left?
Um you can do multithreading in python, also you can use asyncio if you want... (i wrote search engine scrapper and build system for AUR to build all packages using multithreading in python...)
I got two of these used without considering the driver situation… with more recent nvme ssds matching and exceeding the performance of these drives, I can’t fathom using this for my home servers.
Consumer SSD's usually can't match the write endurance of these Fusion IO drives. Definitely something to consider.
Yeah, these will out live you under a constant write workload. I have a centos 7 box that's just going to live as a Fusion-io SAN for the rest of my life.
True, but I personally struggle to find a use that actually takes advantage of it. I’ll hold onto them till I have something useful to do with it..
Level9000Techs
This poor thing broke Wendell just. A bit
Have you tried Spinrite from Steve Gibson? I am sure it would resurrect the drive. Steve is the master of drives.
that $10 walmart lamp in the background is great
I'd probably have used a NoSQL key value store instead of a relational database, then you could probably retrieve all blobs for that sector number.