Rescuing an Abandoned Server - Building my new offsite backup!
ฝัง
- เผยแพร่เมื่อ 3 ก.ค. 2024
- In this video we take a look at an old server of mine which has been sitting untouched for over 3.5 years. As you can imagine, it was in quite a dire state so in this video, we clean it up, install a few upgrades and then install a clean copy of Proxmox VE to set this machine up as my new offsite backup server!
Buy items shown in this video on Amazon (Affiliate):
- StarTech SSD Bracket: geni.us/o6Qe
- Battery Powered Air Duster: geni.us/XjUhD7t
www.camerongray.me/
/ camerongray1515
Chapters:
00:00 - Introduction
01:10 - First Look at the Server
02:48 - Looking at the Hardware
06:42 - Cleaning
08:21 - New Hardware Upgrades
13:58 - First Power Up
15:15 - Installing Proxmox VE
16:18 - Proxmox VE Tour
23:10 - Conclusion
AFFILIATE LINKS NOTICE:
Product links under this video marked “(Affiliate)” are affiliate links where I may receive a small commission on qualifying sales. Affiliate programs that I am a member of include, but are not limited to: Amazon Associates, eBay Partner Network and AliExpress Affiliates.
As an Amazon Associate I earn from qualifying purchases.
Purchasing through these affiliate links will not cost you any more money, however the commission earned significantly helps fund the production of videos on my channel. - วิทยาศาสตร์และเทคโนโลยี
Buy items shown in this video on Amazon (Affiliate):
- StarTech SSD Bracket: geni.us/o6Qe
- Battery Powered Air Duster: geni.us/XjUhD7t
You got me wondering if that StarTech SSD Bracket will also work in an HP DL380 G7 and / or an IBM X3650 M4 !
Realistically, for SATA SSDs that card should work for any machine with a suitable PCIe port and a pair of free SATA ports as all it uses the slot for is power and then looks like a regular SATA SSD to the SATA controller. Using an NVMe slot with it is a bit more hit-or-miss and may end up requiring a newer machine for full functionality.
@@Gerrit-Maxworks in any system as the power it uses is passive from the slot, you just need 2 sata ports for mirrored boot (I thought the HP dual sata m.2 cards I got was expensive but startech are the same or more then £30-40) unsure why they are so expensive when they require less parts then a dual nvme version £5-10 usually (guess it's more classed as niche item)
seems like a pretty rare thing for a hackerspace to offer it's members server colocation at their facility, definitely a very nice perk
I know at least one that does it. The racks there looks impressive, but are full of super old crap. I am afraid it could take on fire at every moment and security is not priority there. I would consider Dell R series with a secure plane at the front to protect hard drives from being stolen. I prefer my home rack over cheap colocation. I would get on an expensive one without a blink, if I got at least one customer who would even out the costs. Oh well.
M.2 drives "Literally taking up space"
Can confirm. Need new home for sata cables I'm drowning in them!!.....
Can’t even give them away
@gandi69 It cracks me up its like USB c cables when the first ones come out, savour them like they are the most precious thing on earth, then you can't even give them away. Man, how things change.
@@ConfidentialMeerkattime to sell em to iPhone users
@zeruty the thought had occurred to me. Knowing apple, only their "special" cable will work though! They did mention other cables would cause damage to the device. Which I'm sure goes against the standard if they alter it in anyway.
SO out of my league and understanding, but I always enjoy your content none the less. I wish I had a fraction of your knowledge and understanding. Well done you Cameron!!
Amazing 😃😃 the thought of your server happily humming away forgotten, hmm someday my owner will connect to me 😅
Using the SATA SSD drives was a nice touch. I just built a Lenovo Server and used NVME drives. I wished servers supported bifurcation but my Lenovo X3550m5 does not. Would have been nice to not be forced to use one slot per SSD.
Glad to see an old server still getting some use, that is perfect for a backup server and doesn't break the bank. One thing I've been surprised about over the years is just how much dust a server can hold and still continue to work lol.
"its control o, of course its control o that would be very clear "
I don't know why but that made me laugh out loud :D, i guess because as a professional computer janitor i have to deal with the same crap all the time.
The kind of video nobody expect but it was fun!
just a tip :
with your NVMe slot, as you're using ZFS, you could look into grabbing an old 16GB Optane module, and setting that as a ZIL device for your HDD array.
it's a nice cheap way to accelerate random writes for a small array, at least if "syncing" is either used by the share type, or syncing is set to forced.
Good stuff. Looks like a fun project.
Pretty cool server rescue. I have a HP DL360 G6 server. I love those servers, I use mine for running Promox. Server will be happy at its new place. Thank you for sharing. Have a good weekend Cameron.
G6 are absolutely solid, but I don't like cable clutter in a 1U units.
Love this! Reminds me of myself and all the servers I have colocated in random places 😆
"Good Monitor for weird Signals" is such a Sysadmin thing lol
Love these tech deep dives
Blast from the past that server Cameron! :-)
Haha, small world! I still have all your emails from the colo days, what a throwback!
Intersting video, thanks for sharing Cameron, I'm sure I've walked past that hacker space with my son a while back and wondered what it was.
Another computer janitor here. 🙂
Hacker space? I’m looking for me info on that
This bracket for SSD is really nice. Saves airflow and allow to do a proper raid on two disk. From what I know, SATA controller on actual pci card have to split pci bandwith explicitly in half for both slots and very few card do it properly.
You can run proxmox backup server on a container. It supports encrypted and deduplicated backups. Pretty neat.
Those UltraStars are data center grade drives so they're awesome
About serial numbers. It is a good idea to have some note, preferably on paper and on the boot drive of the server itself. About mapping of serial numbers, and caddies locations, and their use. Because once the drive fail, it is likely you will not be able to read its serial number or wwn. So you will not know which failed, unless you go over all the other ones, and deduce it by exclusion.
I use Zabbix to monitor my drive health (although most monitoring software can be used for this) - This tracks the history of various drive health parameters which can provide pre-fail warmings but means that even if a drive completely fails and disappears from the system, I can still see what its serial number was through the data that Zabbix has collected while the drive was still operational.
@camerongray1515 your points are correct on zabbix but I'm pretty sure what movax meant is to have a physical note saying which drive is what serial (can also work for ram sticks, not all hardware knows to tell you which slot but I believe zabbix can follow that too)
Can be a post it note on top of the server, in a notepad saying "server x hdd slot y serial aaabbbccc"
Or a label on the front of the drive caddy with the serial- my favorite for fast info, though not the most aesthetic
The reason is that if your nas os/ipmi/zabbix reports a faulted drive with serial abc, you don't need to start looking for the specific physical drive to replace.
Imagine a raid 5/10 (or zfs equivalent) array losing a drive and you don't know which bay the drive is in, and the faulted drive led doesn't work/exist.
You must turn off the server, if you don't have extrrnal bays you need to start looking for the drive, but if your drives positions are noted/labeled, you can just hot swap the bad drive.
Imagine forgetting you need parity and you start pulling out the drives- that's gonna end up very badly for the pool
@davidl6566 That's what I did in this video - all drive caddies are labelled with the drive's serial number.
Thank you you save the E-waste, I try to buy the old 90s-2000s PC case as NAS, of course need to upgrade as DRAM3 with LGA115x from second hand for reuse as NAS, because have HDMI port
The integrated SmartArray controllers in the G6 HP/HPE servers can only recognize
That is SATA AHCI mode, to be precise.
You probably do not need VyOS. You can just setup NAT and WireGuard, on the host itself. It is just a mater of setting few routes, iptables rules, and that is it. But having VyOS as VM is also an interesting option. Also nice for learning. I do use VyOS at home (on a physical box), and really it is nothing special. It is just a frontend to iptables. I will be switching back to pure Debian and few small custom scripts instead to setup everything on startup.
I definitely don't *need* it, but I feel it's always good to separate out responsibilities and have all of the routing and networking stuff in a dedicated VM rather than trying to force Proxmox to also act as a router. With "appliance" style distros such as Proxmox VE, TrueNAS, XCP-ng.etc I prefer to leave the bare metal OS as untouched as possible rather than trying to install all manner of packages on it which could be broken with future software updates since these appliance OSs generally aren't expecting their underlying config to be heavily changed. It also means that when I'm exposing ports to the internet for things such as Wireguard, I'm exposing a VM and not the full underlying host. VMs also have the benefit of being easy to backup or migrate to a different host whereas if I had a complex setup within Proxmox itself, this would need to be manually replicated to a new server if I come to upgrade in the future.
It's no secret that router OSs such as VyOS, PFSense, OPNSense, OpenWRT.etc are nothing really more than a frontend to existing Linux/BSD software, but this doesn't mean that they don't provide a benefit. VyOS provides a very clean and consistent configuration interface all managed from a single configuration file. It also provides a very helpful mechanism for applying and separately saving configuration changes with the ability to rollback (commit-confirm) which can be an absolute lifesaver! Of course, I could easily set this up on bare Debian, but not sure what benefit that would bring over something like VyOS outside of a very niche situation where VyOS doesn't support a certain feature.
These "appliance OSs" are also very helpful when multiple people may be managing the system. They can all view the configuration through the single interface and understand it relatively easily versus a custom setup where you're going to need to document exactly which service and config file handles which particular thing and you'll inevitably have someone bodge something in a config file that's buried away that later gets forgotten about and causes all manner of issues down the line. With something like VyOS - If I'm on an unfamiliar machine, I can just run "show configuration" and BAM! I have a full view of the entire configuration of that router.
Been looking for them StarTech dual m.2 sata things
I ended up getting HP dual m.2 sata Card (not really cheap) working fine in the dell R330 8x2.5 servers I got(really cheap) , only annoying thing is the sata ports are in 2 awkward locations
Your hp is quite old but most servers have a life cycle controller F11 in dell's case if local (it's it's own mini os that's built into the bmc/iDrac out of bands management) "Hp Intelligent Provisioning" I believe is what hp call it and usually allows remote updating in short amount clicks (but from the looks of your raid controller showing the drives as 2tb it's probably older then what I have got, recommend making sure your raid controller is in "it" mode as currently setup will likely require you to manually pass throw the drives at boot to change a drive)
11:39 ive looked into booting from nvme in older servers, the solution is to boot a bootloader on a flash drive that has the nvme drivers
Looks like the state of the servers at one of our offices. Ice cold due to air con and filthy
Yeah it seems like ice cold aircon is a thing of the past for servers.
Also! BIOS-based machines do have a 2.2TB limit for storage devices. Thus explaining why your hard drive capacity is shown as lower than expected.
I tought exactly the same, since he mentioned that it didn't have UEFI.
That limit is only for the boot drive. I use 4TB HDDs for storage on a non-UEFI system and the BIOS detects them correctly
@@Pasi123 Wait the BIOS? It doesn't show as a lower capacity? The OS should be able to handle the proper size.
I'd love to know more about how you use wireguard to connect to this remote server. I have an old sever I'd love to use a remote BU server for critical files as an offsite BU.
WOOOOooooo!! I found the same thing in BOI
How are you sharing your /hdd/storage through a container? I use NFS which isn't compatible with containers. I see you are running the fruity pc's so NFS might not be an option for you. What are you using to mount?
Great video. Any info on the hackerspace your a member of ? Haven’t heard of that before would be interesting to have colo space !
Hackerspaces are a very regional thing so you'd need to look for one local to you, most larger cities will have one of some form, although not all will be so server/networking/IT focussed. They're more of a club that you'd use for projects and socialise with the members of.etc, not really the sort of place you could use purely as a colo facility.
Really enjoyed this one. Out of interest, what’s the context for you having access to this offsite location? Is it part of a shared working space or something unrelated?
I’d love convenient access to such a store room at a club or hobby location for this purpose 👀
It's part of a hackerspace/makerspace - a club for people who enjoy various types of tinkering/tech stuff and offers various workshop spaces with tools and resources- Traditional woodworking tools, 3D printers, amateur radio equipment.etc. This one in particular has a lot of members who are interested in servers/networking stuff so it offers facilities for those such as colocation. There's usually some sort of similar club in most large cities, but the resources they'll offer will tend to vary.
You can also access OPNSense through the serial console from a proxmox host. Not sure how it is to setup OPNSense with the command line vs VyOS.
Exactly - My current OPNSense firewall doesn't even have a graphics adapter so the only option is to set it up over the serial port. However, to actually make configuration changes you really need to use the web interface. I wanted the ability to make any required configuration changes over the console as, if I break the network config, the serial console may be the only way I'm able to fix the issue without physically travelling to the machine.
Amazing Video! :)
How did you label the Drive Caddys?
I used a Brother E550W label printer with the extra wide 24mm tape. That particular printer allows you to set the exact length of a label which allowed me to create labels that fit almost perfectly - only had to manually cut around 1mm off of the bottom of each label to get it to fit.
If the router vm goes down how do you get to the management interface? Is that on a different network or such? As you mentioned you run a public ip to the server.
BTW does proxmox allow multiple Vlans and such internal with virtual switches? :)
How much does the power bill run you? I live in Sweden and running servers are super expensive.
The hackerspace has a shell server that I can access over SSH and from there I can SSH into the bare Proxmox host allowing me to fix the router. This was my main motivation for using VyOS instead of a web managed router OS - it's much easier to fix when all I have is an SSH session.
Power is provided by the hackerspace so doesn't cost me anything except for my regular membership fee. Although the general rule is to just be sensible with power and not to take advantage. So a relatively modern server, up to maybe a relatively basic dual CPU machine would be fine, but don't go running a quad CPU machine with a bunch of GPUs unnecessarily or run a 20 year old server for something that could run faster on a Raspberry Pi!
I haven't tried VLAN configuration with Proxmox bridge interfaces, it looks like it would be possible and you can make a bridge interface "VLAN aware" so that it'll pass traffic, but it's not something I have first hand experience using.
@@camerongray1515 that sounds awesome I would love something like that here in Sweden. I run my own servers but on hp microserver builds I have two do far a windows box and a pfsense box. But they are local at home so I can do everything from within the network. Then I have a cisco switch and some routing guff going on.
I would rather enjoy being able to run a server like that at a place. Just to have a off-site backup is worth the money. I don't have that today. But I absolutely need to figure out that soon.
Actually thinking about putting together a esxi build with a Amd optron 16core machine I got with 32gb ram. Planning to do eight 4tb ironwolf drives for storage on adaptec 5805. And then having like the os and such on a few ssd drives like you did it. I do allot of video editing so videos eat storage like no tomorrow. And really need a solid backup solution.
I love servers and learning things I don't know how to do :)
it's funny how those gen7 drive cages always break their tabs off
What are you paying for the off-site server per month? And what connection speed do they give you? Can you post a lonk to their services? Been looking for a decent colocation.
This is part of a hackerspace - essentially a sort of club for people who are interested in tech/electronics/radio.etc that I've been a member of for many years. It's not the sort of place you could (or should) really use purely as a colocation provider. It's also not equivalent to an actual colo provider - no UPS, running in a store room, cooled by an extractor fan or in particularly hot weather, a portable AC unit and power needs to be shut off annually for the building owner to carry out electrical inspections. However, if you look around, lots of smaller datacentres offer relatively affordable 1U colocation plans for relatively power efficient servers.
Very nice. What's the reason to choose Proxmox above TrueNAS Scale or UnRAID?
Proxmox was because I wanted this machine to primarily host VMs with decent capabilities around networking. TrueNAS Scale would also do that but is more suited to acting as NAS and hosting network shares which I'm not really doing here - I'm dumping files onto it over SSH but that's really it from a file storage perspective. I run TrueNAS at home on my NAS and it's great, but it's nowhere close to Proxmox when it comes to being a hypervisor. As for Unraid, I can't quite put my finger on it but I just don't have much trust or faith in it. It's more of a gut feeling sort of thing rather than anything concrete, but while the filesystem is neat, it doesn't seem to know what it wants to be - is it a NAS, a hypervisor, a machine for virtualizing multiple desktops with GPU passthrough for gaming? And at least last time I looked into it, which was a while ago, almost everything ran as root which is a bit of a red flag to me. It's probably ideal for many home servers but it's not something I've ever seen deployed in any serious business environments and when it comes to me picking software to run in my home network, I stick to things I'd also be comfortable deploying in a business.
There was no facility for ilo (or similar) at this colo provider?
The machine has an iLO but I'd never bothered to set it up at the time as it's horrifically outdated, insecure and uses a TLS version so outdated that practically no modern browser/OS will connect to it. I could of course connect an external IP KVM to it, but the machine isn't particularly far away so I can easily enough just get to it in person.
Hmm. I've got like a VPS on a cloud at the moment. Should I get a used server and colocate it to a datacenter? Or should I just stick to cloud-based solutions, or even just plain Google drive or something for my off-site backup. I'm just a student currently, tinkering with stuff.
I'd probably stick with cloud storage and VPSs unless you have a good reason otherwise. Backblaze B2 offers very low cost cloud storage. With a colocated server, you're responsible for all of the maintenance so if something like a drive were to fail, you'd need to buy a replacement drive yourself then either travel to the datacentre to install it or pay someone there to install it for you. This was why I stopped colocating this machine and moved to hosting everything in the cloud. The only reason I'm using this machine now is because I essentially get the colocation for free as part of my hackerspace membership and it's not too far away so I can easily get to the machine 24/7 to perform hardware maintenance.
@@camerongray1515 Awesome to see your reply. Might get some used server for myself to tinker with :P but will stick to cloud storage and VPSs to host my offsite backup! Probably like AWS glacier or smth. Since it's 'cheap'. As well as your B2 suggestion!
Colocation is pretty expensive, as power prices are going up. One big advantage of a metal is guaranteed performance, because cheap VPS providers are often over-provisioning their servers quite a lot. There is a limit to even Vmware live migrations if one o(or more) of the fellow VM's start some CPU/io intensive work.
THREE AND A HALF YEARS?!?
Do you use the two SSD for boot and vm storage? If so are you worried about wearing the ssd out?
I'm not particularly concerned - none of the VMs I'm running perform any significant writes to the drives, you'd probably see more wear if the drive was used heavily in a regular PC. It's also relatively easy to monitor the remaining lifespan of an SSD from its smart data - these WD green drives report a percentage of reserved space remaining. As long as you monitor this - you can easily enough detect when a drive is approaching the end of its lifespan and pre-emptively replace it.
I have had some speed (cloning) issues when having proxmox OS and the VM storage on the same device. Wondering if it will perform better if I create a zfs pool with two SSDs. But my SSDs are Samsung EVO which are not great from proxmox I understand. @@camerongray1515
dramless WD Green as proxmox drive is kinda funny :D At least do not use them as VM drive.
Did you notice the random sparking in the background at the Maker Space?!
I saw that while editing - I'm pretty sure it's just an activity LED - that shelf has a bunch of desktop PCs/tower servers on it so looks to me like a blue activity LED on one of those. The colour adjustments I did on those clips of the monitor to make the monitor's image look better seem to make blue LEDs turn white, the LED on the monitor itself was also bright blue but looks almost white in the video.
How can you find a place like this to host your server offsite?
My hosting here is part of a hackerspace which is essentially a club for people interested in technical stuff. However, if all you're wanting is to run a server offsite, the term you'd be wanting to look for is "colocation" - many datacentres offer it, some will only offer larger chunks of rack space (so you'd rent a full, half or quarter of a rack) but some will offer "single server colocation" where you can pay a monthly amount to have them host your sever in their datacentre.
Old Net/Sys Admin here.... Love the video. Out of curiosity, did you replace the thermal paste on the CPU??? And... Is that Server Room climate controlled???
Didn't bother replacing the thermal paste - was tight on time and realistically, the temperatures are fine and any improvement would be minimal. The room is normally cooled with a simple extract fan with cool air being pulled in from the unheated corridor through vents in the doors. There is also an AC unit in the room, but this is on a thermostat and is only used if the room gets particularly hot.
Is there a way to make this server quiet by replacing the fans and powersup with something else? These U servers are loud!
Not really - 1U servers are loud because they need high static pressure fans to force the air through the restricted space. The row of fans in this machine are responsible for cooling everything from the hard drives, to the CPU to any installed expansion cards. If you were to replace the fans, you'd likely end up overheating. If you need a quiet server, you really need to go to at least 2U since then you can install active cooling on your CPU and fit quieter case fans, but even then you'd likely need to build it yourself rather than going for a pre-built.
@@camerongray1515 We had these shitbox 1U servers that would continually overheat - friend of mine in the office took his power drill to the cases drilling hundreds of holes above the fan, everyone thought he was completely mad but they held temp after that. Well that and a large desk fan supplying the rack in our makeshift server room where one of the AC units had failed.
For backup, why not use Proxmox Backup Server?
why wouldnt you not use hardware raid1 on the boot drives? by doing software raid you are writing the same data twice and loss of hardware level recovery?
Nowadays, software RAID can perform equally well as hardware RAID and filesystems such as ZFS offer additional benefits in terms of caching and data integrity. The boot drives are in a ZFS mirror so I do have hardware redundancy. Additionally, the "RAID controller" integrated into this machine, along with those included with motherboards is still CPU driven and requires drivers under the host OS, often known as "FakeRAID". Real hardware RAID would require a dedicated controller.
Hardware raid on remote location would require a battery to work reliably between power outages. I am pretty sure, that new batteries for G6 controllers are pretty rare.
You're not going to replace the thermal paste on the CPU
I didn't have any with me but it's not something I'm particularly concerned about. I "borrowed" the CPU from this machine to test a motherboard shortly after I got it so I know that I used decent paste then. Realistically, the temperatures are more than fine and I'd likely see no real improvement from replacing the paste, maybe a couple of degrees at best.
i think those ssd you have are better because if something goes wrong on writing and the pc shuts off the data is on the drive. having ssd with cache gives you no benefits only failure points. i have a dell r630 that is sitting in the maker space i am in the power is on this place also a big problem but i am done with battery buffered solutions and usv systems they do work but if this happen to often they break and this is the chase on my place. i spend more money on batterys as on the whole server. my r630 has 2 xeon e5 2699v3 and 768gb ram and uses only sas ssd drives. and having caching aktiv made my day in the end harder. i can revover half written data but if the os tels me i am done writing and it is not is one thing thats me annoy.
second is cache slows you down if you have big data chunks. i am involved in a project of weather stations we have about 5000 stations end each of them send once in a month a dataset of 2gb
i have in the end 10tb of data i need to process in one single run. this takes 3h only for the run to get the results but beforehand i need 1 week to feed the machine and 1 week afterwards to check if the result was correct. in that time i had the server sitting in the maker space i had several times a power outage while i was feeding the machine. i had many trouble to find the missing files where was not written but sitting in cache and waiting to be written on disk. the os tells you i am done you can unplug the usb drive but the drives are already working on it so store the data. ok this may might a probem i have created by my selfe because i installed a pcie usb3.2 card with 40gbit usbc and i use a usbc 40gbit nvme caddy with a 250gb nvme in it. ok i see the point my drives are slower as the usb drive but still caching dumb.
prob needs a million updates lol
Are you recording at 60fps now then? 😜
30fps - New camera can only output in 5.9K at either 25 or 30fps but thankfully I can lock the shutter speed to a "safe" speed for a 50Hz mains frequency to avoid flicker while squeezing out the extra 5fps. It'll do higher frame rates but it'll drop the resolution to 4K and, more significantly, will crop the image in to an APS-C sensor size which isn't ideal with the lens I have.
Why not update BIOS and so on?
I just didn't have time while filming the video but will do it next time I'm back in front of the machine.
(Long answer is that I had got my days mixed up and was actually meant to be going out with friends when I was filming this so it turned into a case of "just get Proxmox installed and get out of there")
The planet thanks you for wasting 3.5 years of power..
All I know is that this thing was running non stop for over 400 days and that's alot..
Time to troll. Microsoft mouths drop to the floor. No server stays up that long. :)
A lot of Windows 2012 are doing just fine.
@@lpseem3770 I have had really good luck with 2012, 16 and 19 though they are simple server setups. The most problem I had was with 2012 would stick on updates and you would have to manual install the updates due to them getting out of order? or whatever it was doing. Then on 2016 or 19 had that stupid update that one some of mine removed the settings from the nic. Though I only have a few of each.
Does your wife know you stole her hair dryer?
first