Former NetApp engineer here. I worked on these systems (installing, using and selling) for about 16 years and it was a joy seeing you guys setup these old things. A couple notes for people in the comments: 1) Yes, you can add bigger drives to storage arrays, but the big thing to keep in mind is your rebuild times for failed disks. The bigger your drive, the longer it can take to rebuild a drive from RAID, and if you don't have enough spares, or get a multidrive failure, you could wind up with data loss. This is the reason you typically don't see large 16+ TB drives in enterprise arrays, and when you do there is usually a requirement for 2 or 3 hot spares per pool. 2) Why buy something like this when you can get a smaller NAS box that hold has the same capacity? Data accessibility. You do not what to see what happens to a Synology or QNAP when a couple thousand users try to access it at the same time. 3) Linus is right - they are NOT the target customer for enterprise arrays. They need a big vault, but not the access times. And they can afford to spend the time fixing and building their own solutions - they actually MAKE money from the roll your own solutions via views. 4) NetApp is, and has been historically, the most expensive enterprise storage solution on a per TB basis. They are great if you need CIFS, NFS and block storage (FC, FCoE or iSCSI) in one array. But if you just need block or file, there are much better and cheaper solutions. Fun note here - for a very long time, NetApp was the only storage solution that ran actual CIFS - they actually licensed the protocol from Microsoft. This eliminated the weirdness you can sometime get, especially back in the old days, of running Samba in an AD environment. Overall - this was a fun trip down memory lane with this. Great job getting it setup and running on your own!
it's just so cool that every tech enjoyer and worker get on LTT to share their experience. Internet and Linus does wonders. it must have been hearth warming to see a part of your life documented by THE Linus :)
old fart here; "the weirdness you can sometime get, especially back in the old days, of running Samba in an AD environment" gave me a stress response. Thanks. lol
Also, netapp disks are formatted with extra two bytes per... Sector? I don't know, but anyway it means all the data on the drive is checksummed, which is great but probably a headache for generic Linux.
Jake, you summarized Netapp perfectly. I think only Cisco and Oracle are worse for licensing every single thing. Netapp also licenses capacity, so you pay per gigabyte of space.
The irony is that they could have restored the licences. Just call the old owner and ask him to login to his NetApp account and download them. They are locked to the serial numbers on the controllers... Also naming the IOMs (the dummest divices in existance that have 0 control over the drives) controllers and the RAID controllers - servers... Come on Jake , you are the more knowledgeable of the 2 , get it toggeder.
"It should be illegal to sell hardware that requires software that you need an account to access." - Love it, Jake. We in the TPM space feel this statement so hard.
A couple years ago we had an air compressor company stop by to sell us replacement 150hp air compressors (we use a lot of air) to replace our old (very old) units. I just about couldn't keep myself from laughing at them when they said the units would have to be connected to the internet and would not make air unless we had a current subscription plan with them. I literally couldn't even. As a fairly competent millwright at the time, I offered to my boss to tear all the computer controls out and rewire them from scratch to match the controls on our old units, but he didn't go for it. Subscriptions are literally cancer.
@@whoawhoawhatsthis It actually gives me a base to start researching from, since I hear them talk about things in the video and start googling to learn more. I've picked up a lot just off that.
@@mikyhtx it depends on the paycheck and what the job includes. either he gets paid for 2 jobs, or once for 2 half jobs. i myself am in a position where i do a bit of everything and it is both interesting and pays better than if i had "one" job
One of my favorite things with this channel over the years is watching Jake get more comfortable in front of the camera and even better at explaining technical details at an understandable level. Not that he was ever bad at either, but watching someone flourish is always fun
We ran something like this when I worked at a university. We dropped one in our building, another at a building down the street, and another on another side of the country. Each night, they all backed up to each other. Video editing teams on the campus would dump all their footage to the server and we could edit completely from the network. The brilliance was getting people stop storing the university's historic footage on portable drives sitting on desks...above a restaurant. At one point, one of servers sat in my actual office. A student worker wanted to test performance so he wrote and read files from it all weekend. I came in the next Monday and my actual office was 110F.
It's not a good idea to run these outside of a server room. It will be full of dust within months. It needs air filtration and air conditioning to work reliably in the long term.
Wrote and Read all weekend? I hope he didn't hit all the drives all the time. That's like smoking a 4 packs a day for a week. That server lost like 40% of its remaining life expectancy
I used to be a data Center technician and a few of my clients were oil and gas and had 2-3 dozen racks of fully populated netapp disk shelves like that in their DCs. They were usually around for 3-5 years before getting replaced with higher density models. It was effectively a lease program from netapp, I did get to see a sneak peak of Zen before it was officially on the market which was pretty cool.
Kinda leasing.. we have a large amount of Netapp in our company. Usually what happens is that they will End of Life (EOL) the unit and you have to replace it. EOL prevents support and when you have a large enterprise you need the support and the ability to get spares as well. Netapp will not ship any parts or give any support for EOL systems. So you are kind of stuck. We have been researching AWS, Azure, Oracle and Google but the pricing model is what knocks us back. Instead of a one time payment and use the equipment for 4-5 years we have a nonstop monthly payment.
Please make the "it should be illegal to sell hardware that needs a software and key" a bigger point. Literally tons of ewaste is being created by these enterprise companies especially CISCO. They will tell everyone it's for YOUR safety and security, but it's just greed. That netapp server is just garbage, which is really sad.
PERFECTLY GOOD HARDWARE WASTED BECAUSE OF A STUPID SOFTWARE LOCK, and consumers of all ranges still gets gaslit into buying the *newer and faster hardware with more software/hardware interlocks* for their supposed benifits.
@@generic6099 This is actually why the DMCA actually made it LEGAL to hack/crack such equipment when it was past the manufacturer's support cycle. It makes it reusable, doesn't take any money from the company, and reduces e-waste if taken advantage of.
Only this is true enterprise kit. It's designed to make NetApp's customers money. They might spend a fortune to buy and operate the equipment but then they're going to make a fortune in turn. Everyone in the chain with the exception of the ultimate end customer, usually the general public, is going to make their coin and take their pound of flesh in setups where these things are typically used.
@@JeffDeWittIt just isn't working. Enterprise customers already pay huge prices for literally everything. They will continue to pay for a comparatively small licensing price. Non-enterprise customers and the environment are hit the hardest and the companies have no reason to care.
@@JeffDeWitt Unfortunately, for some "enterprise stuff" it costs more in time and people than just to pay the huge price. The part that surprises me the most is them not willing to even sell the license because it was bought second hand.
I used to work as a hardware maintenance engineer for a company in Ireland and used to look after the likes of "legacy" IBM and Dell systems. These had gone past the support life that the OEM would provide unless you paid them ridiculous amount of money. I used to deal with all the same issues that the lads are coming across, licensing, hardware procurement etc. It's all a huge pain in the hole to force companies to keep paying the OEM to maintain them.
Same, but there are companies out there that specializes in supporting out of support system. The main issue is there is not likely to be any firmware updates to them, but we had old IBM servers running with good support from 3rd party. That said, we only had PSU and HDD problems, but they had the systems we still used in stock for a "quick" replacement. Boy did it make a difference in speed and time used to replace those old systems :)
@@Joliie oh yeah, I used to work for a 3rd party, I realised I didn't specify in my comment initially 😅 you're spot on, we mostly dealt with HDDs and PSUs, we very seldom had issues with network cards. One of the challenges with the service was also knowing the command line for the iSeries and the xSeries systems. It was good fun 😊
15:11 can we just shoutout startech for making every weird adapter under the sun? Every time I have some stupid idea they're at the top of the amazon search.
As a former Netapp Certified Engineer, it’s really funny to see you trying to understand how to connect and use it. Also the Ethernet cables for the shelves (ACP) are not any more necessary with the recent versions of Ontap.
@@Sammysapphiracompany in in is embarking on a 6 year plan to decommission all it's data centres around the globe and migrate to AWS. So we may have done 1 or 2 of them in 10 years... 🤷♂️
@@Sammysapphiramost companies moved to cloud. The big companies who still remain on prem build their own storage stack. Netapp is still used. Pure storage is one of the more 'newer' and fancier on-prem system. I still think that Dell/EMC has the biggest market share for data center storage though...
I would love to see more home server stuff, something like a practical and affordable diy nas setup for a small family. A simple build with some of the configuration for the essentials of a family-oriented homelab. Including things like file and computer backups, media server ect.
You could repurpose your own gaming PC if you have a high core count and enough RAM with proxmox. Basically proxmox is essentially free VMWare made by an Austrian Company. Granted their support hours follow Austrian time but its a great free tool that lets you cluster everything together and multiple VMs.
I tear down enterprise computing equipment like this as my job rn. The licensing problem is the tip of the Iceberg, especially when you start to consider networking equipment. Some of it can't be reset without the root password and require a reflash to even log in. I got started taking tech seriously after learning how to build a PC from Linux back in the NCIX days. I love your mission and your philosophy on life/work
When i was in IT, I spent 2 days in an old server room (that we were moving out of) just zeroing a bunch of servers we were going to surplus. That was a thing that took a lot of time.
I had a similar task at a defence contractor when we were preparing desktop computers for return at end of lease. Hundreds of desktops had to be wiped, re-imaged with the original OS, any front panel USB ports reconnected, and any extra drives removed. Equally, the new replacement desktops had to be imaged and have their front panel USBs unplugged. Whatever the IT setup, its still labour intensive process, plugging and unplugging PCs, checking serial numbers, etc, and it's hard to justify creating a really lean process for something that was done in bulk once every few years. The machines that were used in high security areas were easier to deal with. the hard drives were removed and destroyed.
Lol, just getting my home server set up right now, four 4TB drives that used to have data on them have been zeroing for about three hours right now and scale says its only 36% done. What did I do in the meantime? Well, I'm moving house so I showed the house to two potential buyers. Real estate is so not my gig.
@@NavySeal2k Firstly, doing it yourself doesn't meet government data destruction requirements, Second, the disks to be destroyed were sent to another company so the process can be certified and verified. Thirdly, as I said, these were the fastest computers for us to deal with...
I rescued some older netapp fibrechannel disk shelves and an iSCSI SAN, turned out there was a relatively easy nvram patch to make it think it was an engineering test unit and unlock all the licenses by default (thanks netapp for leaving the configs for every model in the firmware image, even though they were IBM and Sun branded netapp shelfs). installed sas to fc interposers and maxed out all the units with common off the shelf drives.
I bought the same setup for my company. Was awesome to see you guys run into the same exact issues as i did. i never could get access to the fans and man o man the fans spin! fast forward a year and filled with 12tb drives. i use half the array to replicate to itself. I love it!
@@giovannimantegna769 But in anything 'smart' it's more an issue that everything is run in some random 'cloud' server somewhere so after a few years the company switches that server off and all your smart stuff becomes e-waste with no (reasonable) way to fix it... Here it's just fully functional hardware that is meant to be sold to large companies that would rather pay for a support contract than risk having downtime, kind of like an insurance package. So naturally NetApp doesn't want to sell or deal with small parties that won't buy their product. While it is annoying, nobody really is going to use it like that anyway. More annoying are HPE and Dell locking BIOS updates behind a paywall... You buy a server with 2 years of support, it has a bug in the BIOS firmware that makes it crash once every week or so for two years. After two years they bring out a fixed BIOS update that you can't download because your support contract has ended... (True story) Basically they sold you a broken server (actually 250 servers...) that they didn't fix for 2 years only to then tell you that after 2 years of problems you need to pay them even more to finally get the server you thought you bought two years ago...
I am so glad you guys tried to get the normal NetApp environment up and running before retrofitting everything. The learning experiencing of working with HA hardware/software is a valuable experience. One important step after getting everything working is testing it. This includes going full chaos monkey and unplugging random cables and seeing how it reacts vs. expectations. Though those one finger salutes were warranted for NetApp given the enterprise lock downs on licensing. 13:10 More modern SAS disks now have multiple arms so that two operations can be performed simultaneously. To further improve parallel operations, that half the stack of arms can act independently of the other half, permitting up to four operations on a single set of platters. Awesome for hard drives but far behind even SATA SSD in terms of raw IOPs. 19:47 The interposers permit both controllers to see a single SATA drive. You are correct in that most controllers can operate a SATA drive however in that chassis only one of the two will ever actually see it. One of the benefits of SAS is its support of multipathing to the drives which is critical for HA. That means both controllers have access to all the drives simultaneously and if one controller fails, the other still has access. Another way of looking at it is that a 12 Gbit SAS drive actually has 24 Gbit of aggregate bandwidth across two links. This is how modern SAS SSD can keep up with NVMe to a degree. However, the new hot swappable NVMe form factors like E3S also support multipathing to provide HA functionality. Throwing 22 TB SATA drives into these does make sense for bulk storage of video. Using all 96 bays would permit ~1.8 PB of capacity after formatting/redundancy. Since LMG is mostly a video shop, I'd disable compression. The controllers will still try to compress that video data which just consumes power without much of a result. I would also consider moving smaller files to a different storage pool build around SSDs while keeping compression on.
I'm a computer flipper and almost always rely on Marketplace and Ebay for my parts. I sometimes use specialty places like Jawa, but only when I'm desperate as their prices are non negotiable and almost always too high (my experience). Glad to see something like this antique can be had on Marketplace, it's always why I almost daily check on there 'cause you never know.
I just picked up a Dell R210 ii with 16Gb of Ram for $50, 8mi from me...from Marketplace. I couldn't believe I finally get a deal like that and that close
10:00 that is the huge problem power consumption. One of those 600GB drives takes 10-16W... times 24 per shelf, times 4 shelfs is 1-1.5kW (JUST the drives alone). That is 2-3 times more then my complete high-end gaming setup with 5800X3D, tuned 4090 AND OLED TV.
This was a super solid pick up for small business mass storage. For personal home use, Linus is definitely correct. A nice little mid to full tower desktop with 4-8 drives or a micro ATX/ mini itx nas chassis with sleds makes a great custom rig. For me, I am dabbling in home lab gear, so we (roommate co-op) picked up an older sandy bridge super micro 12 disk server. It's a bit warm and the cores are not particularly fast, but as a pure NAS solution on unRAID it's perfectly good for our use case and easy to use. As a side note, I personally have smaller shares for my virtual machines in ZFS and a windows storage share for my hyper-V config. So I have very diverse set up MOSTLY for personal educational purposes.
I had one of these storage shelves. Turns out the fans are not just crazy loud, they also draw a lot of power. Like, A LOT. Swap them for Noctua NF-A8 FLX (2 per PSU) and the system becomes a lot more manageable both in terms of power draw and noise. With an all-SATA setup and dual PSUs, I had it hooked up to an unraid server with an IBM M1015 HBA. Worked like a charm. Only downside for me was that Unraid could not spin down drives or read SMART data (I was using interposers).
Best advice before start building a homelab: decide what you want to learn. Most people just create a homelab because they like buying and playing with hardware, but dont have any real goals that they want to reach. 🤔 (For me its learning about IPv6, High availability network and storage, ZFS and FreeBSD)
I'm a field engineer for NetApp, the disk controllers are called IOM modules and the server controllers are called Controllers. This is so awesome to watch!!
@@floydlooney6837 when a power supply is missing the system kind of panics and ramps up. So plugging in all the power supplies will fix that. Also when the system is at loader it ramps up as well.
When I worked for Konica Minolta in one of their datacenters, we used NetApp for storage. Cisco blade servers down one row of cabinets, booting off of and VMs running off of NetApp tiered storage down another aisle, all connected via multiple 40g backbones. Pretty fun experience!
I buy used NetApp drives all the time, I re-flash the drive firmware and replace the proprietary 520 byte sectors size with standard 512 byte sectors. One of my servers (an HP Proliant DL360) has been running 8 of them for 5 years nows, somehow without a single drive failure even after moving the server 3 times! They must choose their drives well since they've been so reliable (compared to other drives I've used).
@@MaximNightFury They are, to flash the drives I had a PCIe to ExpressCard adapter I plugged into my laptop (Thinkpad T430), then I had a SuperMicro HBA I hookup so I can do 4 drives at once just using my laptop (since my server RAID controllers will not recognize them as valid drives). Boy did it take forever too since the tool I used had to do a slow wipe of the entire drive afterwards.
The shelves are worth roughly $300 to $400 each empty. They are quite popular in the home lab space since it's a cheap way to host a large number of drives. Linus actually got a great deal. Maybe not so much once you account for the time it's going to take Jake to remove 96 drives.
Reminds me of my HP days. Hauling server hardware around MS campus and deploying unreleased goodies. Most lab managers loved me due to being proactive on distributing the hardware instead of hoarding it in one lab. I did a lot of research into storage limits and wondered why no one did daughterboard adapters to socket SD cards into 3.5 bays. The solutions that have been made in the trailing dozen years still haven't really did the right things or taken off. Right things being controller and cooling.
13:10 Ok just out of curiosity: wouldnt a bigger platter with a dead inside be better? Lets say right there that disc has 5 cm radius of arm movement covering pi * 5^2 cm^2 = pi * 25 of data storage. If we had instead let’s say 8 cm with dead 3 cm in the middle we’d have 5 cm arm movement again, but covering pi * (8^2 - 3^2) = pi * 55 Basically more than doubling the covered surface per arm movement. Of course overidealized not taking disc rotation time into account, smaller one also has a tiny dead center, bla bla. Ok now that I’m writing this I’m wondering how much wrong surface waiting times would come into play. Lets say on the small one you have 8 bits, the most inner ring has only 8 bits, so u have 100% hit rate. On the wider concept one you’d have millions of bits making a lower hit chance. However… on the inner small one your have to wait 2 rotations worst case if u hit the second bit and have to wait all the way till it starts again while on the bigger one your only have to wait roughly 1 rotation total… My head hurts…
With a more powerful actuator they could get the same seek times with a larger platter. If other commenters can be trusted the reason they don't do this is because a larger platter spinning at high speed would be too mechanically unstable. Personally I'd rather they still made slow-spinning 5.25" hard drives for bulk storage.
Regarding the interposer to go from SAS to SATA: I am pretty sure that another important reason for it is the multiple controllers that access the disk. SATA doesn't offer an ability for multiple controller connections, which is important for the high availability fail-over to work. The interposer offers that multiple connection access while using a SATA disk. Also, regarding the idea about hiring a person to manage the storage for you and therefore not needing the high availability: I don't think that makes sense. If the array is mission critical and it needs to be back online ASAP if it is down, then you still need high availability features even if you hired someone to manage it. (I'm probably just incorrectly reading into that comment though).
This helped me out a lot! I acquired a Netapp 12 drive disk shelf from a business I did some work for. Its just been sitting in my office for about a year now not doing a thing. Without the license key they are basically useless, as they found out in the video! So, thanks for pretty much answering all my questions I had about repurposing this paperweight thats sitting in the corner of my office!
I knew there was gonna be a call to Wendell with this one, but I do enjoy watching them learn the hard way that enterprise stuff is managed by teams of experts for a reason.
As someone who used to work for a big web hosting company that used NetApps, it is possible to trick a netapp into accepting *any* drive, you just have to do a low level format that changes the size of the sectors, if I remember right. But anyway, we had a specific machine running linux we used to basically turn normal SCSI drives into NetApp drives. (our netapps were all scsi)
There are several gotchas with buying enterprise hardware on the "grey market". You won't have a maintenance contract, which also prevents you from getting some or all of the software and / or licenses. If you want to get this, most often the vendors will say you need a maintenance contract. But, they will backdate this to when the maintenance lapsed. So if you buy the hardware from a thirs party in e.g 2023, and the maintenance lapsed in 2020, the vendors will make you backpay from 2020.
If you buy something that is bog standard like a NetApp, EMC, or other storage vendor, you don't need "a Jake". Jakes cost a lot, and often their time is better spent on other custom things instead of keeping shoestring / duct tape hardware working. This is like managed vs unmanaged services.
i like videos like this, that are more tech oriented and go into details of the respective hardware and software. The extra depth may not be suitable for the average audience, but for others it may be really nice to watch.
I haven’t built a computer in a while but after watching this channel for years I was able to without any hiccups. Much thanks to all your mistakes that helped me learn.
I love deals when people just want something gone and literally any amount of money is a bonus for them. It’s a win win. I get a lot of tools like that.
I remember back in 1999, the job I was at in Oak Ridge got a 4TB EMC storage array (Before Dell acquired EMC). The unit cost our company about 8 million dollars. Had a built in laptop to manage the array locally or you could manage it remotely, which I did and was able to get the array programmed to our business' needs in a few days. Never touched an array device of this size before and was absolutely a beast. Was louder than the Sun e10000's we had running. Insanity. yes, 4 TERABYTES
Working with NetApp for like 3 years. The Storage Products were the one who brought me in IT. Its still very nice to work with it. Nice finally see a Video on LTT.
Those Netapp Heads went eosl thats why they wanted to get rid of it, you wont get any parts or support from netapp anymore (very expensive anyway like 3k+ a month per 2heads, depending on your contract with netapp). The Netapp Disks are formated in their own Format (WAFL) with a different block size (iirc 520 size instead of 512 what is common) so you have to reformat them first (if even possible, never tried that)
I have always suggested to friends to buy computer parts off facebook marketplace although this isn’t what this is, it is always nice to see a successful working purchase from it.
FR, where i live you always find the best deals in marketplace, just gotta know how to tell a scammer from their profile and how to purchase something from a stranger with no intermediary
Be mindful of the SATA vs SAS is the third pin connector won't put your drives to sleep or to be discovered. Since your array was already SAS, i'm just putting it out there cuz i know lots of people had the same issue...
The smaller platter is pretty standard on 15k drives, on top of seek times it’s also smaller to reduce the stress on the spindle when spinning that fast. Remember how anything over 56x cds would cause them to shatter?
16:52 Could the connector be the SAS SFF-8088 to SFF-8644 instead of QSFP? The QSFP connector shape is the similar, but considering the age of the array and the disk types, it might not be that.
We had something similar at work around 2005. It was about half as tall as that one, loaded with enough 15,000 rpm SCSI drives to reach 1 TB. The dumb part was it was used to store MP3s for a radio network's music library, and all the files were in one directory, at least when they first set it up. It was so slow! 😂
Also smaller disk platter allows for it to spin faster or have more head room. It could possibly even prevent vibration or warping at high speeds of the platter disks.
I really really want to see more DIY and Enterprise server videos. Data hoarding and management along with how all of this stuff works would be awesome to learn. Stuff on with Access and SQL. Be kind cool even though we have stuff like True NAS.
That's a drool worthy TrueNAS build. Getting the rack and the four shelves for $1000 was a bargain. Fully populated with 20TB drives you can turn that rack into 2PB. Holy smokes!
You can use Linux to rewrite the bit sector size on all those drives and use them on standard equipment thereafter. I can forward the command prompts to do it but i suspect you guys could figure that out. To recap. Netapp doesn't have proprietary firmware on the drives. They are just formatted with a non standard sector size
This story reminds me of buying an Alienware in 2008 off ebay, which the seller scammed me because the BIOS was corrupted. I had everything I needed to get it working again besides the firmware which little did I know at that time that Dell bought Alienware and would not provide the firmware because I needed an account to access which was only possible if I purchased it from Alienware directly.... 15 years later and no one as fixed these consumer laws.... What if we had the support of the a media company to do something about this for the greater good?
It is so refreshing to hear Jake talking more like a professional techy by not swearing in one of LTT videos after all he is representing Linuses company THANKS Jake
Once you change the shelf id, you need to powercycle the shelf for the change to take effect. I can see blinking shelf id's so that could be an issue. The previous os probably is limited to 16 disks per bus, need to change that to 255, like the linux on has.
It's understandable. LMG is not some small upstart anymore. Linus has to split off various duties from himself to run the business. He can't be editing, writing, fixing, managing, renovating, buying, merchandising everything himself as he could in the early days. It makes sense to split the day-to-day infrastructure off to someone who is dedicated to it. Jake is that someone.
But pirating NetApp firmware is like watching youtube with uBlock on! Anyway, I am expecting that someone have already sent required files to Linus already.
The reason you might need the SAS-SATA interposer is if you're dual pathing and want to use larger capacity SATA drives. The interposer adds that capability to the cheaper SATA drives. Totally *non* optional in that use case. If you don't care about dual pathing (and honestly, that's one of the selling points of these shelves) then you could try without it, but then if you lose one of those paths to the shelf (something in the daisy chaining, or you lose a controller), you're knocking that drive offline which could affect the entire shelf, etc.
Protip: make sure old equipment supports updated encryption/hashing and firmware updates without a license.....BEFORE u buy......lots of people trying to sell deadend platforms
The problem was to reset it to factory default. It probably came with all the licenses still on. Just the password went in the way. The license keys are delivered to the original owner. Maybe they still had them. I agree, this practise of not freely sell these systems to transfer ownership and get access to everything that was originally sold is so bad it should be sued against.
It depends really on who the seller was, if it was the original company that ran the server then it's also a maybe, lots of documents get lost in the sauce, especially something like server admin password when a company gets liquidated. Your best bet is if the company that ran it and sold it to you is still in business and sold it because they upgraded their servers, or somehow getting a hold of the server admin that used to work at the company where the server was running.
If the documentation is from 2015, then it means you googled it and clicked on something and it’s all the old documentation that somehow google deems higher priority. Better to hit the new documentation site; much cleaner and easier and specific to OS. Plus system manager isn’t required to manage newer versions of the OS, you just hit the website of the controller and the manager pops up. 9:09
Sorry to be poking at details, but usually the ones you call servers are called controllers, and the shelfs/expansions modules are called IO modules (IOM)
I found two of these for $100 3 years ago on craigslist. I thought it was junk but wanted the racks! I kept them around but I was in a apartment sooooo I couldnt ever power it on cuz the power plugs arent standard in my apartment. Had to move and they had to go cuz I had no way of knowing if it would be worth keeping. Wish this video was around when I still had the hardware. Mine came with all the cables still in it but no drives in it. They only thing I've kept is the daisy chain power strips mounted in the rack but pretty sure they need 240v to work? I've been trying to share my rack/setup so they could roast it for awhile now 😅
The "only runs specialized NetApp firmware" on the disks thing *is* actually for your benefit -- NetApp treats your data as *the* most important thing, and not losing it as being the single most important thing they do, and the shipping firmware on a lot of disks is frequently just not up to that task (sometimes for stupid reasons like "disk reports data written but it's not actually completely on disk yet", sometimes for much less stupid reasons like the deep specifics of how the disks deal with something like a read error). You don't want NetApp for the price, you want NetApp because you want the closest thing possible to a guarantee that your data won't disappear. I remember troubleshooting an issue a decade ago where my company's (unmaintained, thanks to layoffs) NetApp kept rebooting about once a day -- and it turns out, it was doing that intentionally, because we'd run out of spare disks in the array, so the next disk failure would leave us a single failure away from catastrophic data loss. As far as the NetApp was concerned, availability wasn't as important as protecting our data, so it rebooted itself daily specifically to get the attention of someone/anyone to maybe go poke at it and see why it's doing that. And it worked! We had new disks within a day, and the thing was quickly made happy, and our data was safe. So, the expensive NetApp-certified disks aren't just a money grab, there's a reason -- though they most certainly do make a small fortune on it, too.
Yeah, enterprise licensing is easily the worst part of working in IT. It's extremely bad across the industry to where there's even a market for vendors to manage it for you if you don't have the staff to dedicate to that task as it's a full time job in of itself. Netapp doesn't have anything on Cisco, Oracle, and Microsoft though. Those three in particular are what I would consider to be 'hostile' for their licensing shenanigans.
I concur. I work in school IT support. Although I don't directly work with licensing for our IT infrastructure and software, I have worked a bit with the licensing for the software for our marquee signs. Its annoying mainly because the company that made the software doesn't exist anymore. The license is tied to the machine that it is installed and is always a hassle when we had to upgrade computers. It also doesn't help that the software was meant only for Windows XP, so we had to use some workarounds to get it to install on Windows 10/11. Luckily, we've moved away from that software.
@@JJFlores197 I know that pain. The pain of dealing with software from companies that go under is never fun and it's always expensive to find (or build) a replacement.
22:58 sudo sg_format --format --size=512 to make 520 sector size "normal".... pretty annoying thing. This is also required when disk is incorrectly formatted with Data Integrity Feature (DIF)... It's not uncommon.
As answer to that statement at 10:50, it is actually illegal, both in Canada and the U.S you sell something that they need to then buy something else to make it work
I personally think it is better to have many low capacity drives, then few high capacity drives. It takes less time to repair the array with low capacity drives. I think it improves the access speed when the information is spread across more drives.
After looking at these videos and seeing how the MOTHER VAULT came together, I decided to take the plunge and did my own TrueNAS setup... a 1U server and a 2U JBOD for a total of about 35 2.5" disk capacity (currently doing 2 SATA, 15 SAS HD, 3 SAS SSD cache). Got tired of trying to recover USB drives that failed without warning and keeping 2 and 3 different drives everywhere. This is my end game for now. Maybe if space warrants it, I will upgrade to a 45 drive 4U server off of ebay that holds 3.5" drives. Power and noise are an issue but eventually they will go to a sound proof room. Between archiving my various videos & photos and providing a Jellyfin server for my movies, I am very happy so far.
Former NetApp engineer here. I worked on these systems (installing, using and selling) for about 16 years and it was a joy seeing you guys setup these old things. A couple notes for people in the comments:
1) Yes, you can add bigger drives to storage arrays, but the big thing to keep in mind is your rebuild times for failed disks. The bigger your drive, the longer it can take to rebuild a drive from RAID, and if you don't have enough spares, or get a multidrive failure, you could wind up with data loss. This is the reason you typically don't see large 16+ TB drives in enterprise arrays, and when you do there is usually a requirement for 2 or 3 hot spares per pool.
2) Why buy something like this when you can get a smaller NAS box that hold has the same capacity? Data accessibility. You do not what to see what happens to a Synology or QNAP when a couple thousand users try to access it at the same time.
3) Linus is right - they are NOT the target customer for enterprise arrays. They need a big vault, but not the access times. And they can afford to spend the time fixing and building their own solutions - they actually MAKE money from the roll your own solutions via views.
4) NetApp is, and has been historically, the most expensive enterprise storage solution on a per TB basis. They are great if you need CIFS, NFS and block storage (FC, FCoE or iSCSI) in one array. But if you just need block or file, there are much better and cheaper solutions. Fun note here - for a very long time, NetApp was the only storage solution that ran actual CIFS - they actually licensed the protocol from Microsoft. This eliminated the weirdness you can sometime get, especially back in the old days, of running Samba in an AD environment.
Overall - this was a fun trip down memory lane with this. Great job getting it setup and running on your own!
+1
it's just so cool that every tech enjoyer and worker get on LTT to share their experience. Internet and Linus does wonders. it must have been hearth warming to see a part of your life documented by THE Linus :)
that's why there is raid DP who cares
old fart here; "the weirdness you can sometime get, especially back in the old days, of running Samba in an AD environment" gave me a stress response. Thanks. lol
Also, netapp disks are formatted with extra two bytes per... Sector? I don't know, but anyway it means all the data on the drive is checksummed, which is great but probably a headache for generic Linux.
Jake, you summarized Netapp perfectly. I think only Cisco and Oracle are worse for licensing every single thing. Netapp also licenses capacity, so you pay per gigabyte of space.
This comment should get more likes
That's crazy
The irony is that they could have restored the licences. Just call the old owner and ask him to login to his NetApp account and download them. They are locked to the serial numbers on the controllers...
Also naming the IOMs (the dummest divices in existance that have 0 control over the drives) controllers and the RAID controllers - servers... Come on Jake , you are the more knowledgeable of the 2 , get it toggeder.
No, Dell/EMC are by far the worst.
so does EMC starting with VNX2
So glad to see Linus’s oldest son Jake is still doing well
I thought Jake was his boyfriend
@@MaximNightFurycringe furry
@@MaximNightFury Jake is Linus' son but Yvonne's second husband.
You can see the company is doing well... as Jake has had a haircut
@@MaximNightFury What a cute couple
"It should be illegal to sell hardware that requires software that you need an account to access." - Love it, Jake. We in the TPM space feel this statement so hard.
*cough* 2nd hand apple products *cough*
A couple years ago we had an air compressor company stop by to sell us replacement 150hp air compressors (we use a lot of air) to replace our old (very old) units. I just about couldn't keep myself from laughing at them when they said the units would have to be connected to the internet and would not make air unless we had a current subscription plan with them. I literally couldn't even.
As a fairly competent millwright at the time, I offered to my boss to tear all the computer controls out and rewire them from scratch to match the controls on our old units, but he didn't go for it.
Subscriptions are literally cancer.
I swear Linus and Jake doing server things is something I know almost nothing about yet its entertaining as hell.
Same...I actually love the server videos... almost all of it goes above my head, but interesting to watch.
@@whoawhoawhatsthis It actually gives me a base to start researching from, since I hear them talk about things in the video and start googling to learn more. I've picked up a lot just off that.
100% i would like to pay jake to build me a server, even though i dont need it at all. ( i'd be okay with a 2tb hdd) Bur this duo is fun.
Same 😂
100%
Not only does Linus save money by paying Jake to manage his servers, he gains money by turning it into content.
Truly a genius man 😢
Jake work for Linus so he gets paid managing the servers is just one part of his job
@@rokibucayeah but imagine doing two jobs and only getting paid for one.
@mikyhtx many people have multiple tasks as their single job responsibility....
You don't know that he's not being paid more for his "extra" work.
@@mikyhtx it depends on the paycheck and what the job includes. either he gets paid for 2 jobs, or once for 2 half jobs. i myself am in a position where i do a bit of everything and it is both interesting and pays better than if i had "one" job
One of my favorite things with this channel over the years is watching Jake get more comfortable in front of the camera and even better at explaining technical details at an understandable level.
Not that he was ever bad at either, but watching someone flourish is always fun
Waiting for our Linux Queen to come back in front of the camera as a main. ❤
Every video i watch with jake just makes me wish i could be friends wiht him hahah
He even teed up that sponsor segue pretty well!
Now he just needs a gym arc to become gigachad
We ran something like this when I worked at a university. We dropped one in our building, another at a building down the street, and another on another side of the country. Each night, they all backed up to each other. Video editing teams on the campus would dump all their footage to the server and we could edit completely from the network. The brilliance was getting people stop storing the university's historic footage on portable drives sitting on desks...above a restaurant.
At one point, one of servers sat in my actual office. A student worker wanted to test performance so he wrote and read files from it all weekend. I came in the next Monday and my actual office was 110F.
It's not a good idea to run these outside of a server room. It will be full of dust within months. It needs air filtration and air conditioning to work reliably in the long term.
OMG and I can't imagine the sound either. When an enterprise device goes in to 20k RPM mode with 80 fans.
Wrote and Read all weekend? I hope he didn't hit all the drives all the time. That's like smoking a 4 packs a day for a week. That server lost like 40% of its remaining life expectancy
@@Reac2 it was years ago. As far as I know, it was fine.
@@Reac2 writing and reading for 2 days will make it lose 40% of its life expectancy? So if they did it for a few weekends it would have died?
I used to be a data Center technician and a few of my clients were oil and gas and had 2-3 dozen racks of fully populated netapp disk shelves like that in their DCs.
They were usually around for 3-5 years before getting replaced with higher density models.
It was effectively a lease program from netapp, I did get to see a sneak peak of Zen before it was officially on the market which was pretty cool.
@AlexanderHenry-hz3tdwhy the spam?
Kinda leasing.. we have a large amount of Netapp in our company. Usually what happens is that they will End of Life (EOL) the unit and you have to replace it. EOL prevents support and when you have a large enterprise you need the support and the ability to get spares as well. Netapp will not ship any parts or give any support for EOL systems. So you are kind of stuck. We have been researching AWS, Azure, Oracle and Google but the pricing model is what knocks us back. Instead of a one time payment and use the equipment for 4-5 years we have a nonstop monthly payment.
"How much money did we waste on these?"
"$20"
Linus is genuinely happy that, for once, it wasn't hundreds or thousands.
Or a hundred thousand like a lot of the lab equipment.
Please make the "it should be illegal to sell hardware that needs a software and key" a bigger point. Literally tons of ewaste is being created by these enterprise companies especially CISCO. They will tell everyone it's for YOUR safety and security, but it's just greed. That netapp server is just garbage, which is really sad.
What do you expect for something that old. Now the new NetApp AFF-A800, A250, NS224, those are lighting fast.
So very true. Tons of e-waste ends up in landfills every year due to it. Should be made illegal.
PERFECTLY GOOD HARDWARE WASTED BECAUSE OF A STUPID SOFTWARE LOCK, and consumers of all ranges still gets gaslit into buying the *newer and faster hardware with more software/hardware interlocks* for their supposed benifits.
@@generic6099 This is actually why the DMCA actually made it LEGAL to hack/crack such equipment when it was past the manufacturer's support cycle. It makes it reusable, doesn't take any money from the company, and reduces e-waste if taken advantage of.
Such behaviour from companies should be illegal. Just locking basic functions of harddrives behind login credentials.
Only this is true enterprise kit. It's designed to make NetApp's customers money. They might spend a fortune to buy and operate the equipment but then they're going to make a fortune in turn. Everyone in the chain with the exception of the ultimate end customer, usually the general public, is going to make their coin and take their pound of flesh in setups where these things are typically used.
No, not illegal, but people just shouldn't play their game and either put them out of business or force them the change their behavior.
@@JeffDeWittIt just isn't working. Enterprise customers already pay huge prices for literally everything. They will continue to pay for a comparatively small licensing price. Non-enterprise customers and the environment are hit the hardest and the companies have no reason to care.
@@seigeengine Spoken like someone who doesn't have a need for enterprise grade storage.
@@JeffDeWitt Unfortunately, for some "enterprise stuff" it costs more in time and people than just to pay the huge price. The part that surprises me the most is them not willing to even sell the license because it was bought second hand.
I used to work as a hardware maintenance engineer for a company in Ireland and used to look after the likes of "legacy" IBM and Dell systems. These had gone past the support life that the OEM would provide unless you paid them ridiculous amount of money. I used to deal with all the same issues that the lads are coming across, licensing, hardware procurement etc. It's all a huge pain in the hole to force companies to keep paying the OEM to maintain them.
Same, but there are companies out there that specializes in supporting out of support system. The main issue is there is not likely to be any firmware updates to them, but we had old IBM servers running with good support from 3rd party. That said, we only had PSU and HDD problems, but they had the systems we still used in stock for a "quick" replacement. Boy did it make a difference in speed and time used to replace those old systems :)
@@Joliie oh yeah, I used to work for a 3rd party, I realised I didn't specify in my comment initially 😅 you're spot on, we mostly dealt with HDDs and PSUs, we very seldom had issues with network cards. One of the challenges with the service was also knowing the command line for the iSeries and the xSeries systems. It was good fun 😊
Ah Ireland, a nation of data centres and storage engineers.
@@WolfetoneRebel1916 don't forget incredibly hodgepodge internet access!
15:11 can we just shoutout startech for making every weird adapter under the sun? Every time I have some stupid idea they're at the top of the amazon search.
As a former Netapp Certified Engineer, it’s really funny to see you trying to understand how to connect and use it. Also the Ethernet cables for the shelves (ACP) are not any more necessary with the recent versions of Ontap.
I liked you labeled yourself as *former" NetApp Cert. Engineer. The companies I work with all ditched them a decade or more ago.
@@guytech7310 What do companies use for storage these days? Does everyone just rely on AWS or other cloud infrastructures?
@@Sammysapphiracompany in in is embarking on a 6 year plan to decommission all it's data centres around the globe and migrate to AWS.
So we may have done 1 or 2 of them in 10 years... 🤷♂️
@@Sammysapphiramost companies moved to cloud. The big companies who still remain on prem build their own storage stack. Netapp is still used. Pure storage is one of the more 'newer' and fancier on-prem system. I still think that Dell/EMC has the biggest market share for data center storage though...
Former NetApp remote support engineer here. Takes me back to the old days...
I would love to see more home server stuff, something like a practical and affordable diy nas setup for a small family. A simple build with some of the configuration for the essentials of a family-oriented homelab. Including things like file and computer backups, media server ect.
Interesting
Craft computing does a lot of home lab content. Mostly a mix of obsolete enterprise gear and weird chinese market stuff.
@@mgkleym Serve the Home is another good channel to look at.
You could repurpose your own gaming PC if you have a high core count and enough RAM with proxmox. Basically proxmox is essentially free VMWare made by an Austrian Company. Granted their support hours follow Austrian time but its a great free tool that lets you cluster everything together and multiple VMs.
Depending on needed storage and data throughput... unRaid on your choice of hardware may be a great option.
I tear down enterprise computing equipment like this as my job rn. The licensing problem is the tip of the Iceberg, especially when you start to consider networking equipment. Some of it can't be reset without the root password and require a reflash to even log in.
I got started taking tech seriously after learning how to build a PC from Linux back in the NCIX days. I love your mission and your philosophy on life/work
@AlexanderHenry-hz3tdstop Spamming
When i was in IT, I spent 2 days in an old server room (that we were moving out of) just zeroing a bunch of servers we were going to surplus. That was a thing that took a lot of time.
@@TheInfidel_SlavaUA why would it need many man hours?
Just start the process and do something else.
I had a similar task at a defence contractor when we were preparing desktop computers for return at end of lease. Hundreds of desktops had to be wiped, re-imaged with the original OS, any front panel USB ports reconnected, and any extra drives removed. Equally, the new replacement desktops had to be imaged and have their front panel USBs unplugged.
Whatever the IT setup, its still labour intensive process, plugging and unplugging PCs, checking serial numbers, etc, and it's hard to justify creating a really lean process for something that was done in bulk once every few years.
The machines that were used in high security areas were easier to deal with. the hard drives were removed and destroyed.
Lol, just getting my home server set up right now, four 4TB drives that used to have data on them have been zeroing for about three hours right now and scale says its only 36% done.
What did I do in the meantime? Well, I'm moving house so I showed the house to two potential buyers. Real estate is so not my gig.
O_o takes 20 seconds to take a drill to a Harddisk, you can speed this up by stacking them 3 high. We never give away harddisks.
@@NavySeal2k Firstly, doing it yourself doesn't meet government data destruction requirements,
Second, the disks to be destroyed were sent to another company so the process can be certified and verified.
Thirdly, as I said, these were the fastest computers for us to deal with...
I rescued some older netapp fibrechannel disk shelves and an iSCSI SAN, turned out there was a relatively easy nvram patch to make it think it was an engineering test unit and unlock all the licenses by default (thanks netapp for leaving the configs for every model in the firmware image, even though they were IBM and Sun branded netapp shelfs). installed sas to fc interposers and maxed out all the units with common off the shelf drives.
I bought the same setup for my company. Was awesome to see you guys run into the same exact issues as i did. i never could get access to the fans and man o man the fans spin! fast forward a year and filled with 12tb drives. i use half the array to replicate to itself. I love it!
Jake is 1000% correct here, it really should be illegal to sell hardware this locked down... 🤬😡
worst is in the smart home sector where remote access is subscription locked
@@giovannimantegna769 that's not affecting millions of dollars of hardware.....yet
yeah .... blame it on the "socialists" ... oh wait !!!! XD XD XD
@@giovannimantegna769 But in anything 'smart' it's more an issue that everything is run in some random 'cloud' server somewhere so after a few years the company switches that server off and all your smart stuff becomes e-waste with no (reasonable) way to fix it...
Here it's just fully functional hardware that is meant to be sold to large companies that would rather pay for a support contract than risk having downtime, kind of like an insurance package. So naturally NetApp doesn't want to sell or deal with small parties that won't buy their product.
While it is annoying, nobody really is going to use it like that anyway. More annoying are HPE and Dell locking BIOS updates behind a paywall... You buy a server with 2 years of support, it has a bug in the BIOS firmware that makes it crash once every week or so for two years. After two years they bring out a fixed BIOS update that you can't download because your support contract has ended... (True story)
Basically they sold you a broken server (actually 250 servers...) that they didn't fix for 2 years only to then tell you that after 2 years of problems you need to pay them even more to finally get the server you thought you bought two years ago...
I am so glad you guys tried to get the normal NetApp environment up and running before retrofitting everything. The learning experiencing of working with HA hardware/software is a valuable experience. One important step after getting everything working is testing it. This includes going full chaos monkey and unplugging random cables and seeing how it reacts vs. expectations. Though those one finger salutes were warranted for NetApp given the enterprise lock downs on licensing.
13:10 More modern SAS disks now have multiple arms so that two operations can be performed simultaneously. To further improve parallel operations, that half the stack of arms can act independently of the other half, permitting up to four operations on a single set of platters. Awesome for hard drives but far behind even SATA SSD in terms of raw IOPs.
19:47 The interposers permit both controllers to see a single SATA drive. You are correct in that most controllers can operate a SATA drive however in that chassis only one of the two will ever actually see it. One of the benefits of SAS is its support of multipathing to the drives which is critical for HA. That means both controllers have access to all the drives simultaneously and if one controller fails, the other still has access. Another way of looking at it is that a 12 Gbit SAS drive actually has 24 Gbit of aggregate bandwidth across two links. This is how modern SAS SSD can keep up with NVMe to a degree. However, the new hot swappable NVMe form factors like E3S also support multipathing to provide HA functionality.
Throwing 22 TB SATA drives into these does make sense for bulk storage of video. Using all 96 bays would permit ~1.8 PB of capacity after formatting/redundancy. Since LMG is mostly a video shop, I'd disable compression. The controllers will still try to compress that video data which just consumes power without much of a result. I would also consider moving smaller files to a different storage pool build around SSDs while keeping compression on.
I'm a computer flipper and almost always rely on Marketplace and Ebay for my parts. I sometimes use specialty places like Jawa, but only when I'm desperate as their prices are non negotiable and almost always too high (my experience). Glad to see something like this antique can be had on Marketplace, it's always why I almost daily check on there 'cause you never know.
I just picked up a Dell R210 ii with 16Gb of Ram for $50, 8mi from me...from Marketplace. I couldn't believe I finally get a deal like that and that close
10:00 that is the huge problem power consumption. One of those 600GB drives takes 10-16W... times 24 per shelf, times 4 shelfs is 1-1.5kW (JUST the drives alone). That is 2-3 times more then my complete high-end gaming setup with 5800X3D, tuned 4090 AND OLED TV.
*5:41** IS THAT A FIGHTER JET TAKING OFF*
This was a super solid pick up for small business mass storage.
For personal home use, Linus is definitely correct. A nice little mid to full tower desktop with 4-8 drives or a micro ATX/ mini itx nas chassis with sleds makes a great custom rig.
For me, I am dabbling in home lab gear, so we (roommate co-op) picked up an older sandy bridge super micro 12 disk server. It's a bit warm and the cores are not particularly fast, but as a pure NAS solution on unRAID it's perfectly good for our use case and easy to use. As a side note, I personally have smaller shares for my virtual machines in ZFS and a windows storage share for my hyper-V config. So I have very diverse set up MOSTLY for personal educational purposes.
Definitely cheaper options out there but if you want a solid case with support for drives you can't go wrong with a fractal design R5 or R7!
I had one of these storage shelves. Turns out the fans are not just crazy loud, they also draw a lot of power. Like, A LOT. Swap them for Noctua NF-A8 FLX (2 per PSU) and the system becomes a lot more manageable both in terms of power draw and noise. With an all-SATA setup and dual PSUs, I had it hooked up to an unraid server with an IBM M1015 HBA. Worked like a charm. Only downside for me was that Unraid could not spin down drives or read SMART data (I was using interposers).
Same Here. Damn interposers
LTT needs a channel that is dedicated to home lab / home server and networking!!!!
This!!! I agree 100%
Best advice before start building a homelab: decide what you want to learn. Most people just create a homelab because they like buying and playing with hardware, but dont have any real goals that they want to reach. 🤔
(For me its learning about IPv6, High availability network and storage, ZFS and FreeBSD)
LLT (Linus Lab Tips)
No no, call it
Level Two Tech (LTT)
@AlexanderHenry-hz3td hail satan
I have been running one of these units at home for my media library and love it to death. Got 12 bays populated with refurbished 10TB sas disks
16:15 You can see Linus go through all 5 stages of grief as soon as Jake says "luckily"
Gotta love a Linus and Jake excursion into IT equipment!
Gotta love how no one asked!
@internet_userr no one asked for your opinion
They usually have 1 or 2 videos a month together, so it's not unusual.
Information technology technology
you must be new choncho
I'm a field engineer for NetApp, the disk controllers are called IOM modules and the server controllers are called Controllers. This is so awesome to watch!!
Was the 🖕 part also awesome to watch? 🤣
Are they always so loud?
@@floydlooney6837 If you plug in all power supplies the fans slow down.
@@Jehty_ kinda lol. I actually love my job.
@@floydlooney6837 when a power supply is missing the system kind of panics and ramps up. So plugging in all the power supplies will fix that. Also when the system is at loader it ramps up as well.
It's lovely to see the fun these guys have and the respect Linus has for Jake's skills and innovation.
Thx! Know the chief evangelist, nice guy! Btw: Greetings to Oli F. 🎉😂
When I worked for Konica Minolta in one of their datacenters, we used NetApp for storage. Cisco blade servers down one row of cabinets, booting off of and VMs running off of NetApp tiered storage down another aisle, all connected via multiple 40g backbones. Pretty fun experience!
I buy used NetApp drives all the time, I re-flash the drive firmware and replace the proprietary 520 byte sectors size with standard 512 byte sectors. One of my servers (an HP Proliant DL360) has been running 8 of them for 5 years nows, somehow without a single drive failure even after moving the server 3 times! They must choose their drives well since they've been so reliable (compared to other drives I've used).
520 byte sectors sound like hell
@@MaximNightFury2 extra bytes are for checksums
@@MaximNightFury They are, to flash the drives I had a PCIe to ExpressCard adapter I plugged into my laptop (Thinkpad T430), then I had a SuperMicro HBA I hookup so I can do 4 drives at once just using my laptop (since my server RAID controllers will not recognize them as valid drives). Boy did it take forever too since the tool I used had to do a slow wipe of the entire drive afterwards.
where do you get firmware for disks?
The seller must've thought Linus was the worst negotiator, for offering to pay double of what they thought they were offering 😅
But they still sold it for $1000? Lmao I'd be pissed
@@shannonduckmanton243In the original video, they meant to sell it for $500 and accidentally added another 0
I legit cannot think of something in LTT history that has made me laugh as hard as that moment in the video.
@AlexanderHenry-hz3td which fairy tale is this from?
The shelves are worth roughly $300 to $400 each empty. They are quite popular in the home lab space since it's a cheap way to host a large number of drives. Linus actually got a great deal. Maybe not so much once you account for the time it's going to take Jake to remove 96 drives.
Reminds me of my HP days. Hauling server hardware around MS campus and deploying unreleased goodies. Most lab managers loved me due to being proactive on distributing the hardware instead of hoarding it in one lab. I did a lot of research into storage limits and wondered why no one did daughterboard adapters to socket SD cards into 3.5 bays. The solutions that have been made in the trailing dozen years still haven't really did the right things or taken off. Right things being controller and cooling.
13:10 Ok just out of curiosity: wouldnt a bigger platter with a dead inside be better?
Lets say right there that disc has 5 cm radius of arm movement covering pi * 5^2 cm^2 = pi * 25 of data storage.
If we had instead let’s say 8 cm with dead 3 cm in the middle we’d have 5 cm arm movement again, but covering pi * (8^2 - 3^2) = pi * 55
Basically more than doubling the covered surface per arm movement.
Of course overidealized not taking disc rotation time into account, smaller one also has a tiny dead center, bla bla.
Ok now that I’m writing this I’m wondering how much wrong surface waiting times would come into play. Lets say on the small one you have 8 bits, the most inner ring has only 8 bits, so u have 100% hit rate. On the wider concept one you’d have millions of bits making a lower hit chance. However… on the inner small one your have to wait 2 rotations worst case if u hit the second bit and have to wait all the way till it starts again while on the bigger one your only have to wait roughly 1 rotation total…
My head hurts…
With a more powerful actuator they could get the same seek times with a larger platter. If other commenters can be trusted the reason they don't do this is because a larger platter spinning at high speed would be too mechanically unstable. Personally I'd rather they still made slow-spinning 5.25" hard drives for bulk storage.
Jake and Linus both do such a good job bring this all down to my level. Their knowledge knows no bounds
Regarding the interposer to go from SAS to SATA: I am pretty sure that another important reason for it is the multiple controllers that access the disk. SATA doesn't offer an ability for multiple controller connections, which is important for the high availability fail-over to work. The interposer offers that multiple connection access while using a SATA disk.
Also, regarding the idea about hiring a person to manage the storage for you and therefore not needing the high availability: I don't think that makes sense. If the array is mission critical and it needs to be back online ASAP if it is down, then you still need high availability features even if you hired someone to manage it. (I'm probably just incorrectly reading into that comment though).
Linus and Jake giving the finger to NetApp is the highlight of this episode
Makes me want to go and do the same. I dont think this was good ad to netapp. Even though most of us watching wouldnt be buying server stuff.
This helped me out a lot! I acquired a Netapp 12 drive disk shelf from a business I did some work for. Its just been sitting in my office for about a year now not doing a thing. Without the license key they are basically useless, as they found out in the video! So, thanks for pretty much answering all my questions I had about repurposing this paperweight thats sitting in the corner of my office!
I knew there was gonna be a call to Wendell with this one, but I do enjoy watching them learn the hard way that enterprise stuff is managed by teams of experts for a reason.
As someone who used to work for a big web hosting company that used NetApps, it is possible to trick a netapp into accepting *any* drive, you just have to do a low level format that changes the size of the sectors, if I remember right. But anyway, we had a specific machine running linux we used to basically turn normal SCSI drives into NetApp drives. (our netapps were all scsi)
There are several gotchas with buying enterprise hardware on the "grey market". You won't have a maintenance contract, which also prevents you from getting some or all of the software and / or licenses.
If you want to get this, most often the vendors will say you need a maintenance contract. But, they will backdate this to when the maintenance lapsed. So if you buy the hardware from a thirs party in e.g 2023, and the maintenance lapsed in 2020, the vendors will make you backpay from 2020.
Jake is that person that Linus hired full time to take care of their mission critical infra. 😅
Why is that funny?
He was hired as a video writer. Took over some of the infrastructure duties over the years
@@Locutusperhaps because Jake does a terrible job of it (love you Jake, sorry :'()
If you buy something that is bog standard like a NetApp, EMC, or other storage vendor, you don't need "a Jake". Jakes cost a lot, and often their time is better spent on other custom things instead of keeping shoestring / duct tape hardware working. This is like managed vs unmanaged services.
@@philb5593 Still not funny. It's great that he was able to move up in LMG.
i like videos like this, that are more tech oriented and go into details of the respective hardware and software. The extra depth may not be suitable for the average audience, but for others it may be really nice to watch.
I haven’t built a computer in a while but after watching this channel for years I was able to without any hiccups. Much thanks to all your mistakes that helped me learn.
Definitely my favorite kind of LTT content, love seeing videos like this.
I love deals when people just want something gone and literally any amount of money is a bonus for them. It’s a win win. I get a lot of tools like that.
these are the best kind of videos LTT makes! :)
I remember back in 1999, the job I was at in Oak Ridge got a 4TB EMC storage array (Before Dell acquired EMC). The unit cost our company about 8 million dollars. Had a built in laptop to manage the array locally or you could manage it remotely, which I did and was able to get the array programmed to our business' needs in a few days. Never touched an array device of this size before and was absolutely a beast. Was louder than the Sun e10000's we had running. Insanity. yes, 4 TERABYTES
@AlexanderHenry-hz3td Muhammad is gay.
1:45 Isn't it like a JBOD NAS where all the JBODs are connected to a central system that distributes storage to connected systems?
Working with NetApp for like 3 years.
The Storage Products were the one who brought me in IT.
Its still very nice to work with it.
Nice finally see a Video on LTT.
10:57 Jake summarizing all of my complex and highly nuanced feelings about NetApp in 1 second.
Those Netapp Heads went eosl thats why they wanted to get rid of it, you wont get any parts or support from netapp anymore (very expensive anyway like 3k+ a month per 2heads, depending on your contract with netapp).
The Netapp Disks are formated in their own Format (WAFL) with a different block size (iirc 520 size instead of 512 what is common) so you have to reformat them first (if even possible, never tried that)
I have always suggested to friends to buy computer parts off facebook marketplace although this isn’t what this is, it is always nice to see a successful working purchase from it.
FR, where i live you always find the best deals in marketplace, just gotta know how to tell a scammer from their profile and how to purchase something from a stranger with no intermediary
Be mindful of the SATA vs SAS is the third pin connector won't put your drives to sleep or to be discovered. Since your array was already SAS, i'm just putting it out there cuz i know lots of people had the same issue...
The smaller platter is pretty standard on 15k drives, on top of seek times it’s also smaller to reduce the stress on the spindle when spinning that fast.
Remember how anything over 56x cds would cause them to shatter?
You'd expect the company to not want the bad publicity and just throw the old software at them.
But then it would devalue anyone that bought it originally. They’d rather keep their current customers happy then worry about LTT’s viewer base.
Based on the cost and intended demographic for these servers, I doubt the "general public" that is 99% of LTT's audience matters at all to NetApp.
Yea, that's not how Enterprise licensing works.
None of their customers are on this channel though. Their customers are golfing IT managers, not down in the trenches IT geeks.
If you did a budget build for an old server like this, step by step, and aimed at small business owners I bet we'd eat that up. I know I would.
They did videos about their storage array servers. Petabyte Project
I love buying up old tech and reusing it like this. Sometimes you can get really lucky and get perfectly good hardware for an amazing price.
16:52 Could the connector be the SAS SFF-8088 to SFF-8644 instead of QSFP? The QSFP connector shape is the similar, but considering the age of the array and the disk types, it might not be that.
We had something similar at work around 2005. It was about half as tall as that one, loaded with enough 15,000 rpm SCSI drives to reach 1 TB. The dumb part was it was used to store MP3s for a radio network's music library, and all the files were in one directory, at least when they first set it up. It was so slow! 😂
Also smaller disk platter allows for it to spin faster or have more head room. It could possibly even prevent vibration or warping at high speeds of the platter disks.
Smaller disks also create less drag and use less power as a result :)
I'm certain the smaller platter just means the arms don't need to travel as far.
@AlexanderHenry-hz3td Does Allah make good hard drives?
@@autohmae if the disk can spin faster it allows for faster read speeds in addition to faster seek times.
I really really want to see more DIY and Enterprise server videos. Data hoarding and management along with how all of this stuff works would be awesome to learn. Stuff on with Access and SQL. Be kind cool even though we have stuff like True NAS.
I love how Riley always does the sponsor spots because oh my lord is it entertaining to see him.
That's a drool worthy TrueNAS build. Getting the rack and the four shelves for $1000 was a bargain. Fully populated with 20TB drives you can turn that rack into 2PB. Holy smokes!
You can use Linux to rewrite the bit sector size on all those drives and use them on standard equipment thereafter. I can forward the command prompts to do it but i suspect you guys could figure that out.
To recap. Netapp doesn't have proprietary firmware on the drives. They are just formatted with a non standard sector size
Love the server related videos! Wish you guys could do even more of 'em :D
This story reminds me of buying an Alienware in 2008 off ebay, which the seller scammed me because the BIOS was corrupted. I had everything I needed to get it working again besides the firmware which little did I know at that time that Dell bought Alienware and would not provide the firmware because I needed an account to access which was only possible if I purchased it from Alienware directly.... 15 years later and no one as fixed these consumer laws.... What if we had the support of the a media company to do something about this for the greater good?
So he tried to lowball them, and ended up paying twice what they actually wanted for it. And they say Karma isn't a thing.
This is a good lesson on the ongoing trend to "never own anything and be happy" that these companies get away with.
I'm not sure you know what karma is.
@@marcogenovesi8570 yeah i literally dont get what he means
@@marcogenovesi8570 He's talking about corporations and the trend that you don't own what you buy.
Just wanted to mention in the 1st vid they also got a bunch of extra stuff that they said $1k was an absolutely steal
It is so refreshing to hear Jake talking more like a professional techy by not swearing in one of LTT videos after all he is representing Linuses company THANKS Jake
Once you change the shelf id, you need to powercycle the shelf for the change to take effect. I can see blinking shelf id's so that could be an issue. The previous os probably is limited to 16 disks per bus, need to change that to 255, like the linux on has.
So a software layer in that BSD build assumes the old SCSI2 limit of 16 IDs per 16 bit bus ?
Always love take your kid to work days. Jake is starting to teach Linus things he forgot now.
It's understandable. LMG is not some small upstart anymore. Linus has to split off various duties from himself to run the business. He can't be editing, writing, fixing, managing, renovating, buying, merchandising everything himself as he could in the early days. It makes sense to split the day-to-day infrastructure off to someone who is dedicated to it. Jake is that someone.
Looking forward to part three with the actual deployment! It would be somehow used for the lab, I guess?
LTT probably could find someone with a license to access the download but they would probably not tell us
They probably could but at this point they proved that the hardware can be repurposed without all of that license non-sense.
Netapp allows license transfers in this case... Just get it in writing from the seller and you are good to go...
Why bother when they can replace the server with one that's much higher performance, has no proprietary BS, and the software is free?
But pirating NetApp firmware is like watching youtube with uBlock on!
Anyway, I am expecting that someone have already sent required files to Linus already.
The camera-tracked video overlays are really nice :)
The reason you might need the SAS-SATA interposer is if you're dual pathing and want to use larger capacity SATA drives. The interposer adds that capability to the cheaper SATA drives. Totally *non* optional in that use case. If you don't care about dual pathing (and honestly, that's one of the selling points of these shelves) then you could try without it, but then if you lose one of those paths to the shelf (something in the daisy chaining, or you lose a controller), you're knocking that drive offline which could affect the entire shelf, etc.
Protip: make sure old equipment supports updated encryption/hashing and firmware updates without a license.....BEFORE u buy......lots of people trying to sell deadend platforms
The problem was to reset it to factory default. It probably came with all the licenses still on. Just the password went in the way. The license keys are delivered to the original owner. Maybe they still had them. I agree, this practise of not freely sell these systems to transfer ownership and get access to everything that was originally sold is so bad it should be sued against.
It depends really on who the seller was, if it was the original company that ran the server then it's also a maybe, lots of documents get lost in the sauce, especially something like server admin password when a company gets liquidated. Your best bet is if the company that ran it and sold it to you is still in business and sold it because they upgraded their servers, or somehow getting a hold of the server admin that used to work at the company where the server was running.
I hate when things should "just work", because when they don't, the troubleshooting is typically an absolute nightmare.
If the documentation is from 2015, then it means you googled it and clicked on something and it’s all the old documentation that somehow google deems higher priority. Better to hit the new documentation site; much cleaner and easier and specific to OS. Plus system manager isn’t required to manage newer versions of the OS, you just hit the website of the controller and the manager pops up. 9:09
I love watch tech savvy people work. I would really enjoy working with you guys if I lived there.
I like these vlog styles video's about things they buy, or the lab.
NetApp's PR team suddenly started having a panic attack as soon as this video got uploaded
@AlexanderHenry-hz3td Allah means God, at least learn about fantasy settings well
Sorry to be poking at details, but usually the ones you call servers are called controllers, and the shelfs/expansions modules are called IO modules (IOM)
Honestly its a good time learning about servers with jake and linus❤
Call me a nerd (aren't we all?) or a time-waster when I should be working, but this was super entertaining to watch.
I found two of these for $100 3 years ago on craigslist. I thought it was junk but wanted the racks! I kept them around but I was in a apartment sooooo I couldnt ever power it on cuz the power plugs arent standard in my apartment. Had to move and they had to go cuz I had no way of knowing if it would be worth keeping. Wish this video was around when I still had the hardware. Mine came with all the cables still in it but no drives in it. They only thing I've kept is the daisy chain power strips mounted in the rack but pretty sure they need 240v to work? I've been trying to share my rack/setup so they could roast it for awhile now 😅
The "only runs specialized NetApp firmware" on the disks thing *is* actually for your benefit -- NetApp treats your data as *the* most important thing, and not losing it as being the single most important thing they do, and the shipping firmware on a lot of disks is frequently just not up to that task (sometimes for stupid reasons like "disk reports data written but it's not actually completely on disk yet", sometimes for much less stupid reasons like the deep specifics of how the disks deal with something like a read error). You don't want NetApp for the price, you want NetApp because you want the closest thing possible to a guarantee that your data won't disappear.
I remember troubleshooting an issue a decade ago where my company's (unmaintained, thanks to layoffs) NetApp kept rebooting about once a day -- and it turns out, it was doing that intentionally, because we'd run out of spare disks in the array, so the next disk failure would leave us a single failure away from catastrophic data loss. As far as the NetApp was concerned, availability wasn't as important as protecting our data, so it rebooted itself daily specifically to get the attention of someone/anyone to maybe go poke at it and see why it's doing that. And it worked! We had new disks within a day, and the thing was quickly made happy, and our data was safe.
So, the expensive NetApp-certified disks aren't just a money grab, there's a reason -- though they most certainly do make a small fortune on it, too.
Yeah, enterprise licensing is easily the worst part of working in IT. It's extremely bad across the industry to where there's even a market for vendors to manage it for you if you don't have the staff to dedicate to that task as it's a full time job in of itself. Netapp doesn't have anything on Cisco, Oracle, and Microsoft though. Those three in particular are what I would consider to be 'hostile' for their licensing shenanigans.
I concur. I work in school IT support. Although I don't directly work with licensing for our IT infrastructure and software, I have worked a bit with the licensing for the software for our marquee signs. Its annoying mainly because the company that made the software doesn't exist anymore. The license is tied to the machine that it is installed and is always a hassle when we had to upgrade computers. It also doesn't help that the software was meant only for Windows XP, so we had to use some workarounds to get it to install on Windows 10/11. Luckily, we've moved away from that software.
@@JJFlores197 I know that pain. The pain of dealing with software from companies that go under is never fun and it's always expensive to find (or build) a replacement.
19:42
Linus: "...gotta screw in from the bottom."
Jake: "I have a thing for that."
Linus and Jake: "AYYYY!"
LOL
22:58 sudo sg_format --format --size=512 to make 520 sector size "normal".... pretty annoying thing. This is also required when disk is incorrectly formatted with Data Integrity Feature (DIF)... It's not uncommon.
“This seems safe”
Famous last words
good job explaining for once what are we looking and how it works instead of diving straight into technicalities
As answer to that statement at 10:50, it is actually illegal, both in Canada and the U.S you sell something that they need to then buy something else to make it work
I personally think it is better to have many low capacity drives, then few high capacity drives. It takes less time to repair the array with low capacity drives. I think it improves the access speed when the information is spread across more drives.
After looking at these videos and seeing how the MOTHER VAULT came together, I decided to take the plunge and did my own TrueNAS setup... a 1U server and a 2U JBOD for a total of about 35 2.5" disk capacity (currently doing 2 SATA, 15 SAS HD, 3 SAS SSD cache).
Got tired of trying to recover USB drives that failed without warning and keeping 2 and 3 different drives everywhere. This is my end game for now. Maybe if space warrants it, I will upgrade to a 45 drive 4U server off of ebay that holds 3.5" drives. Power and noise are an issue but eventually they will go to a sound proof room. Between archiving my various videos & photos and providing a Jellyfin server for my movies, I am very happy so far.