Migrating my PERSONAL SERVER from TrueNAS to Proxmox + Cockpit
ฝัง
- เผยแพร่เมื่อ 19 ก.ย. 2023
- Today I'm taking my 10 servers and hopefully working that list down to just 7! JUST SEVEN!
So, driven by my desire to consolidate my critical services into one box so I can lab away with the rest of the boxes, I am taking the time to shut down some of the most critical servers in the house and re-home them, then disassemble the parts for the next project.
As a result, I'm going to show you some behind the scenes computer building, shuffling, and the actual setups I'm using for storage and virtualization.
The blog post includes a full guide on the more unique configurations I'm using, so be sure to check that out too!
www.apalrd.net/posts/2023/ult...
Find the ProtoArc XKM-01 here! #protoarc #ProtoArcFoldableKeyboard #ProtoArcXKM01
www.protoarc.com/collections/...
Support me on Ko-Fi if you enjoy my content and find it useful:
ko-fi.com/apalrd
Feel free to chat about my upcoming projects on Discord!
/ discord
I can only add 5 cards so here are all of the links to the videos I promised I'd add a card for:
Terramaster NAS - • Terramaster NAS as low...
Bigstor the Proxmox Backup Server - • Setting Up my PROXMOX ...
Bigstor gets a LTO Tape Drive - • Should I use TAPE BACK...
Tour of the closet behind me - • Home Network / Homelab...
3-node Cluster video - • $100 3-Node Proxmox HI...
New Desktop (Old one died) - • Updating my WORKSTATIO...
Old video about my old server - • Rebuilding my older ZF...
My previous Proxmox NAS video - • Turning Proxmox Into a...
#homelab #proxmox #truenas #datastorage - วิทยาศาสตร์และเทคโนโลยี
![[UNCUT] ฝันรักห้วงนิทรา | EP.9 (3/4)](http://i.ytimg.com/vi/ikd1tEhXbp8/mqdefault.jpg)
Your Proxmox Cockpit NAS video was paradigm altering for my homelab, lol. It allowed me to ditch a TrueNAS Scale setup I was using and reclaim a TON of compute and ram.
It works well too!
@@apalrdsadventures Indeed! I just finished putting the finishing touches on a 60tb build in a Fractal Node 304 (which was a gigantic pain populating all 6 internal 3.5" bays) and am going to be implimenting that setup on that machine as it's been working flawlessly on my old "production" unit which is an old 2nd gen Core i7 laptop with only 16gb ram in it.
How much more resources did you reclaim because I am using truenas scale and ZFS is usally using 32gb/64gb of ram I have on my server when nothing is happening/being hosted. Mostly just using my server for plex and some other stuff but host game servers a few times a month when my friends want to play a game.
@@dogearooski I had 64gb allocated to the VM and it liked to use a lot of it. I don't have a direct comparison because when I implemented the cockpit setup I did it on a different host. But. that host machine only has 16gb of RAM total so it's definitely efficient enough to run with little ram.
Same here. I am loving the cockpit solution for an smb share
Timely! I've just completed my own migration from TrueNAS Scale to Ubuntu 20.04 + Cockpit + 45Drives. Excellent!
NIce video. I recently junked all my eBay "Lab" equipment and built a vSphere cluster from enthusiast level PC components with three 4TB SSD's. It was a lot of fun and it offers a ton of flexibility. I'm now looking to swap vSphere to Proxmox, which is why I'm here. It was really nice to see someone else's journey.
First thing, I LOVE you cat!
I wished I had known you were going to do this earlier. I just decommissoned my HPE DL380G8 and sold it, and I would have shipped it to you for free.
ProLiant DL380p Gen8
Dual 750W PSU's
Ilo4 HP Integrated Lights-Out
2 x Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz for 32 threads
16 x 8GB 1333Mhz C3-10600 DDR3-1333 ECC Registered RDIMM for 128GB
8 X 600GB 15K SAS drives.
4 x 1gb lan ports
1 x 2tb NVME Drive in an NVME TO PCI-E converter. Used for Macruim backups.
Replaced it with an DL380G10.
My desktop drives died after reaching 10 power-on years. It was a 1TB WD Black and a 500GB Seagate Baracuda. Both were used their last 5 years in ZFS and both were cached by a 80GB and 40GB L2ARC and 5GB and 3GB ZIL/LOG based on a 128GB sata-SSD. My L1ARC had a hit rate of ~98%, while the L2ARC still had a hit rate of ~40%. So in my configuration the disks collected power-on years, but they were somewhat retired, They only had to do all the writes. Most of the reads were done by the memory and SSD caches and that is probably, why they did grow that old, a healthy and serene life :) :)
That cat looking at you once in a while thinking, who the f... are you talking too :D Priceless
YES! I was soooo waiting for this 🎉
I was a cool video to watch... but I was hoping to see more from the terminal side of things - backing up data pool, setting up UIDs, etc. Is this coming?
It's more in the blog post where I have the commands / config files detailed out, I spent some time ironing out the tutorial part there
@@apalrdsadventures great! I wasn't aware, I'll check it out.
@@apalrdsadventures I've checked out the blog post - it is great! Thank you. Now I just have to finish the hardware part of the equation and I can start switching from Synology.
I see The Producer is regularly inspecting the set. 😉
At 20:47, you stopped talking and the only thing that I could see on the screen was a cat that demanded being pet, but all the server stuff from before that makes total sense.
Don't worry he gets plenty of attention
@@apalrdsadventures I'm glad. Jokes aside, it was a pretty fun video. I hadn't heard of Cockpit before, so that was an interesting bit for me.
Regarding shadow copy - can you browse versions on macOS?
What are you using to measure power consumption?
Ambitious
I want to have a all in one server with proxmox and a couple of vms/LXCs (with samba for nas usage) inside. I have one nvme x4, one m.2 key e WiFi and two data ports. Could you please suggest where to put what without sacrificing redundancy? Proxmox on one nvme with zfs and vms/LXCs and the samba data storage on both satas in raid1 with zfs? Have heard proxmox should be kept alone on one drive? I would appreciate a suggestion greatly!
I have that exact same A300 DeskMini setup as my Proxmox host. Love it. Ryzen 2400G, averages ~20 Watts @ 230 V with a few VMs, some LXC and Docker containers, costing me ~€5 a month. Super quiet with Noctua fan, since it has a "real" BIOS and you can adjust fan curves, unlike with those 1 liter PCs. I have to practically put my ear on the case to hear anything. Granted, single 1Gbit Realtek NIC, only 4 cores, forced to use SO-DIMMs, limited IO and no expansion options, but so very silent and cheap to run.
great video. I just got my own asrock deskmini h470 running. superb machines
I've bought x300. Still in the box, waiting for a weekend. Bummer you can't fit 15mm 5tb drives - but other than that amazing machine. I'm planning to run this from USBC power bank as a poor man's UPS.
im running with 2usb drives 5tB.. works great. just as storage. got 2 nvme drives for the machines @@marcinneuman4239
I'll try to downsize to 2tb drives I already have (plus 1tb mirrored nvme) and use 5tb for backup in a separate machine.
Can you use iscsi with the config Proxmox + Cockpit?
Question about HA. I have a 3 node proxmox cluster. All of them have a 1G connection to a switch and a usb 1G usb to Ethernet for a my CT or VM different vlan. I am planning to migrate my HA from a pi to the proxmox cluster. It on another vlan, also have a zigbee dongle on thant pi. Is there a way to migrate HA and use the same vlan on proxmox whitout a nither Ethernet cable? And if something happens to the pc where HA is and migrates to another pc how can i make the zigbee keep working? Thanks
Yes. You'd need dongle for every node and PVE 8.
Pls, do not shorten "HA" like that when talking virtualization.
Can you go a little more in depth on the proxmox + cockpit setup. I am not quite grasping this setup and the advantages it provides over say something like unraid.
What OS are you running on the TerraMaster -- TOS, Proxmox, something else? Thanks.
Proxmox
You had mentioned something about truenas doing truenas things and wanted to move away from them is there any change they made? I am not sure what you had ment about that
In general, their attitude is to keep a complete database of how the system should be configured and regularly wipe the system / regenerate configs with each update, so it's relatively impossible to 'go outside' of their ecosystem if you have any more unique scenarios (or even just need special hardware driver packages).
Depending on your scale that might be a good thing, but in my case file sharing is a very small part of what I want to do, and TrueNAS just isn't good at the VM / CT side of it. I need fairly basic file sharing, and a good hypervisor, and TrueNAS is not that and aggressively does not want me to make it what I want.
Tks for the video! One suggestion: it would be much more easier to undertand some of your toughts if you present a diagram showing all these components. At some points in the video i feel my self lost due to the big and complex structure.
I'm looking to abandon my TrueNAS server, as well. Mainly I want to condense my 2 servers into 1, primarily for power savings. But also I'm not using Samba in my environment anymore. And that's really the only reason I'm using TrueNAS anymore. Everything else is just easier to do at the command line, on a system that isn't hellbent on being an appliance and making your life miserable if you step out of scope in the slightest.
How do you monitor the power consumption?
I have one of these with logging in InfluxDB + Home Assistant: www.brultech.com/greeneye/
@@apalrdsadventures Thank you!
How did you pass through the coral to the Frigate container? (If I understood right you are using a container and not a VM?)
Currently the coral is mounted in an m.2 to PCIe carrier on the Zimaboard, running bare metal, so I haven't done the passthrough yet. But it involves installing the gasket/apex DKMS driver, compiling it for the kernel, and loading it, then passing the /dev/apex devices into the container.
Four things:
1) That ProtoArc keyboard is a great idea, but it isn't exactly what I would call inexpensive. (it's a $90 keyboard and mouse combo) Great idea though. Maybe I'll get that for myself for Christmas.
(I would be interested to see how well that keyboard handles simultaneous key presses (for games and such).)
2) My mass server migration event happened over Christmas last year (2022)/into January of this year (2023) where I migrated 5 servers down to 1. That took my power consumption down from 1242 W to ~570W now. (It's a dual Xeon E5-2697A v4, 16-core/32-thread CPU with 256 GB of DDR4-2400 ECC Reg. RAM and 36 HDDs.)
However, the server became so busy with managing all of the data and VMs and containers (I'm upto 4 containers and 10 VMs running off of that thing, plus, it is responsible for managing two ZFS pools, for a total of something like ~200 TB of space), it became so busy and loaded that I ended up having to fire up one of my old(er) 4-bay QNAP NAS units back up, to take a little bit of the edge load off of the main Proxmox server.
As of this morning, there is a new proposal that I am mulling over where I might have to re-commission my HPC cluster head node back on, because Debian isn't great for dealing with 100 Gbps Infiniband (CentOS was better at it), so I might end up spinning that system back up, so that it can be my Infiniband subnet manager. Not 100% sure about that yet. I might test it with a CentOS VM to see if the VM can deal with being my Infiniband subnet manager, that way, it will make my 36-port 100 Gbps Infiniband switch stop power-cycling the fans in the sense that it will ramp out and ramp down, roughly every hour-ish, because it thinks that the Mcast group has been deleted and recreated, repeatedly.
So I will have to test that out.
Unfortunately, I also know that virtio-fs (which Proxmox supports) and NFSoRDMA doesn't play nicely together, so that will be a bit of a bummer. Dunno yet. Will have to test that out.
3) re: watching your own videos back so that you can see how you did something
That's what I use OneNote for.
I have notes on how to do stuff (especially in Linux) ALL. THE. TIME.
My cluster deployment "script" is a OneNote tab/page.
My OneNote is on my QNAP server and then I've pointed my NextCloud turnkey container to it, so that I also run my own local cloud provider service as well so that I can get access to those documents without going through SMB or NFS (and I can also live-edit them with NextOffice).
4) Going from 10 to 8, then 7 servers is great.
There are a few things that I know that I CAN'T migrate/virtualise:
CUDA doesn't migrate/virtualise well. (GPU accelerated CFD can't find the RTX A2000 6GB, despite me getting GPU passthrough working in Proxmox.)
Some of the Infiniband stuff (again, e.g. RDMA), doesn't migrate/virtualise well.
So yeah.
Also whilst you can virtualise the gaming systems, to connect to it remotely without paying for Parsec, becomes a LOT more limited. (e.g. I haven't found a great way to be able to play Minecraft via a virtualised gaming system, over remote desktop. The mouse inputs gets misinterpreted by the VM.)
Regarding your point 3: I'm a big fan of Obsidian; all the notes are stored as plain Markdown files so it's truly portable in case Obsidian ever stops being maintained.
For scripting cluster deployments, I just started learning Ansible and it's pretty awesome. Biggest pain point I've noticed is that while it tries to be idempotent, when I'm working on a script and something goes pear-shaped, I can quickly get out of sync with the playbook, and it's best to wipe the OS and start over from a clean slate. Nice to do on a VM, impractical for bare metal of course.
@@joelv4495
Yeah, I'm hearing more and more people reference Ansible.
I don't know if I would necessarily use it, because I LIKE the ability to see the result of the commands that I am copying-and-pasting in, to make sure that stuff is working -- because as we ALL know -- with Linux -- it is very easy for a newer version of something to break something else.
I tried, ONCE, to deploy my four HPC cluster nodes simultaneously using tmux and it had mixed success.
Some commands were able to be replicated/cloned correctly whilst others failed to paste in properly across all 4 windows.
But it was a neat concept though, to be able to do something like that.
You do really great videos. But I'm still confused the best way of having these 3 things:
1. Proxmox running VMs with the VMs backed up to "something".
2. A file share location which I can save and access my files from all my network devices
3. The "something" which backs up both the VMs and the file share
Currently I am just running ESXi with random limited backups of my different devices, so I need to sort it out!
What would be your recommendation?
(Then I'll go and watch your respective videos again!)
Here's what I am currently doing:
1. Proxmox VE backs up to Proxmox Backup Server. PBS is a separate machine with a separate zfs pool dedicated to backups.
2. File share location is the Cockpit container running in Proxmox VE, every system mounts the 'lab' share (including the Proxmox host, but also all of my other systems) where I keep ISOs and the like, and there's also two shares for video production and everything else ('projects') that I can login to over Samba. SMB is what I chose since it has wider compatibility with clients (i.e. windows, but also iOS / Android / ..).
3. For me that's the Proxmox Backup Server. In the blog I have some explanation of how I do backups in Cockpit in a way that works decently well with PBS and also lets me have file-level restore and separate schedules for each dataset.
My only gripe is that with PBS the data pool has to be read in 100% for every backup. Maybe it's not a big problem, but it seems like a not the best method - especially for the data that is mostly archive and not changed that often. One big advantage that everything is under one backup system. I'd love to learn more about ZFS send/receive as it seems like a better system for backing up data.
For VMs it does not need to, as long as the VM wasn't shutdown since the last backup. It uses the qemu dirty data bitmask.
For CTs and Hosts it does read the data 100%. It's not ideal, but for me, it's a tradeoff that I'm okay with since I really like other features of PBS compared to other solutions I've used.
I don't have any VMs. It's CTs and data. I don't mind reading it once a week, but I'd like to have additional less brute-force method for daily backups to the dedicated backup machine. Do you have any opinion on using ZFS send/receive?
@@apalrdsadventures thanks for the info. Much appreciated.
I was going to start digging into TrueNAS Scale, so I'll start with this approach instead and see how it goes!
What's the cat's name? Great video, loving your content.
Sherlock has white paws and Danny does not
@@apalrdsadventures They're both very handsome cats.
They are, unfortunately they are not friends with each other
"I have 10 servers in the house doing server things" 👌
because reasons
19:56 I have to respectfully disagree, 3 years of power on time is nothing for those drive. My 4TB WD RED started having signs of the end after having ~7,5 years of power on time......
Also so far im pretty happy with truenas scale, it does not expose docker directly but it still works pretty nicely at least for me.
spend 15 min doing structured wiring organization - too many servers is a nice problem to have - ideally you will want 2 clusters of 3 boxes lower power cpu - i just had a raid fail so you do want dual nas to run netfs - I think that is probably the biggest thing to do for aspiring cluster fanatics - big and fast netfs with redundancy
Ideally the firewall and backup server are separate, then a cluster, plus one machine left to do hardware level stuff (like testing PCIe devices).
@@apalrdsadventures oh yeah - ideally you want/need at least 11 boxes - ha opnsense, ws, dual nas and 6 machines in 2 clusters - you can run backup proxmox on nas as vms - you could run on 7 boxes minus the extra cluster and nas but for something like a small biz you would want the extra cluster and nas, toss in a pihole but you could virt this too - it does hammer home why people want to have low power machines - the elec can add up - prices for parts like 5700g will drop once zen5 gets closer - realistically i think you want everything to run on at least 6 core chips to give yourself some leeway - the 5600g looks good to me. you should do some content on new pc and how that worked out - realistically going with a 6th/7th gen refurb may be more realistic for nodes - half the cost - a 7600 amd chip may be best if you have the dosh - only 50 bucks more than 5700g, still a 65w chip you can underclock and a good upgrade path when zen5 drops in price in a year or two - pc prices should drop quite a bit after zen5 - jan/feb could be best time to build but when considering cluster or system for smb people may as well just dive in and make the best of it - i think it is more important to get a working system than to worrry about power consumption too much - it is a consideration but not overriding concern - optimixation may save you 10w per box at most and for the entire setup that is about 100w savings - it is not all that significant at the end of the day truth be told - stability and performance are bigger priorities - and time management - and production
Ten servers to seven ?. Not enough, should be to two!.Well, I used an X99 sever with E5 2680V4 14 core machine with 64GB of ECC DDR4 RAM. I reduced from 6 to two!.But had 14 drives and 2 NVME disks.
Do you have a channel too? Where is the video showing this beauty?
Re-upload? Why i think i already saw this intro?
It was live for like 45 minutes on accident last week, I hadn't finished the written article
I watched the whole thing then 😆@@apalrdsadventures
The irony... I am neck deep in a migration from TrueNAS on bare metal to TrueNAS in a VM on Proxmox.
Consolidation is also the reason. I was running seperate nodes for Proxmox with the baremetal TrueNAS as backing storage. Now, TrueNAS is *just* the storage/pool/users manager and everything either hairpins on the same host or connects over the network to TrueNAS.
PBS backs up the TrueNAS VM.
TrueNAS configs are persisted in git.
VMs have local storage and shared backing storage via TrueNAS.
I mostly kept TrueNAS because it does a decent job managing the ZFS pools and sharing things out. I don't care for their "apps" and "kubernetes" implementations.
Here I am doing the opposite! I’m consolidating to save on electricity costs, though. My proxmox cluster will be replaced with a single TrueNAS server.
Fix your audio mate. It's low and feels very muted.