Dude, I love your videos. This one is my favorite. Your excitement and frustration is genuine yet you are able to entertain all the way through. Don't change.
Building a mirrored ZFS for the boot drive would solve this issue. Also I believe when you have hardware passthrough enabled it has to reserve all the memory. The error on passthrough might be caused by using that gpu as the display out on the host, That usually causes issues. Sometimes blacklisting the gpu will work causing proxmox to run headless. But then you can run into the GPU reset bug and it still won't work. Best solution I've found is throwing in a cheap video card for the host to use, and make sure to set it as the default video output device in the bios. Then GPU passthrough probably should work.
"Building a mirrored ZFS for the boot drive would solve this issue." Yes and no. It would "solve" the boot (NVMe) SSD dying issue (sort of), but trying to get the GPU passthrough to work with ZFS root is a PITA. (I know, because I tried. And as a result of my attempt, I ended up reverting back to a more conventional HW RAID6 that's managed by my MegaRAID SAS 12 Gbps 9361-8i instead. (Because fixing a HW RAID6 is easier than fixing a ZFS raidz2 sometimes.)) "Also I believe when you have hardware passthrough enabled it has to reserve all the memory. The error on passthrough might be caused by using that gpu as the display out on the host, That usually causes issues. Sometimes blacklisting the gpu will work causing proxmox to run headless. But then you can run into the GPU reset bug and it still won't work. Best solution I've found is throwing in a cheap video card for the host to use, and make sure to set it as the default video output device in the bios. Then GPU passthrough probably should work." Depending on the combination of CPU and GPU that you're using, for some, you need to make sure that you have resizable BAR enabled in the BIOS, and that both your CPU, motherboard, and GPU supports it. For the Proxmox console itself, you really DON'T need video output since all it tells you at the end is the IP address (which you probably already know) :8006 to log into the web UI where you're going to do everything else anyways. The better way to check, IMO, is to install and enable openssh-server (# apt install -y openssh-server) and then making sure that you are able to ping the system and ssh into it. If you pass that check, then you're able to do a LOT more over ssh than you would be via the local console anyways. (Once you have blacklisted nouveau and a bunch of other things (as a result of enabling GPU passthrough), the part where it tells you to log in from the :8006 might not even work, but the system itself is otherwise fully operational. So, again, the console output isn't always necessarily indicative of a fault nor problem with your system. Again, ping and ssh are better able to ascertain that than said console output. (If you have a server with IPMI, that can help as well. Or you can also get and deploy either PiKVM and/or TinyPilot KVM to perform a similar function, if your motherboard doesn't have IPMI built-in. Just be aware that those other "IPMI"-like tools needs a physical connection to the motherboard's power switch headers, etc., and therefore; unlike having multiple servers that have IPMI built-in, you won't be able to easily nor readily plug the PiKVM nor TinyPilot KVM from system to system to system without having to unplug the connections to the power switch headers on the motherboard to the power switch headers on another motherboard.) Having the IPMI is like having a cheap GPU, except it doesn't take up a PCIe slot. There are plenty of forum thread posts about BAR on the Proxmox forums.
Re: backing up Proxmox system. You gotta remember it's an IT appliance; the individual nodes are considered worthless. The VMs and the cluster (if exists) is what matter to backup. Proxmox itself should be backed up by knowing how it is configured. Whether that is a configuration manager (IE: Ansible, Chef, or saltstack), a script that configures it, or just a piece of paper with all the changed settings listed.
Fully agree. That's why I keep deployment notes in my OneNote. Another thing that might also be useful (potentially) is dumping the history of your bash commands that you've used, and also dumping a list of the packages that you've installed as well. (That way, if you DO need to recover the system, you can just "replay" the apt install "script" with all of the packages that you have installed post initial install (or, again, if you've saved a record of it in your deployment notes.) The vast majority of the packages that are installed on the system -- it will likely skip (or say that it already has the latest or a newer version installed and therefore; nothing to do), but it will install the "delta" basically.
Probably easier to just to an image of the proxmox install to a spare drive you can swap in, and create regular image backups. Then you aren't jumping through hoops to get up and running again.
@@hycron1234 Three things about this: 1) Depends on where you end up storing or hosting said image. 2) You assume that there isn't a hardware failure that will prevent the successful redeployment of this image. 3) You also, further assume, that there aren't any hardware changes between the system that was imaged vs. the system that it will be imaged unto (which may or may not be true). (IF the hardware changed between the source of the image and the destination, then there is a possibility that restoring the image may not work 100% of the time.) If you have a clone of (all of) the hardware, then you could redeploy the image, without needing the second server to be on all the time, as a live mirrored copy as a part of your HA cluster.
A couple things, I have Blue Iris running on a 2016 VM using an ancient server tower with 7 cameras, and still only peg the cpus at 13-15%. My thoughts were to migrate to a PROXMOX server and here's why. They have HA (High Availability). This is an opportunity for you to set up a secondary server, even a lower speed system that your PROXMOX can fail over too. Even if all it does is backs up your VMs live so you can selectively choose which VMs you wish to run. *Edit...did you GLUE the false fronts to your drawer boxes? Noooooooooo!
Im running mirrored nvme boot drives, its kinda nice to have knowing that the system will always boot. Also running proxmox internal backup-tool too weekly backup vms to both my large raidz3 pool and a external usbdisk.
I love seeing videos like this I spent so many nights in college doing all this as well!!! The one thing I can recommend to you that I learned the hard way is always have a backup of your backup 😅 I currently have a 500GB 2.5 HDD plugged into my proxmox that runs a weekly cron job that backups all my VMs and that is then copied to my apple iCloud because it is the most reliable and don’t forget step 1 of the 3-2-1 method is having one offsite :) Backups, Backups, Backups, Backups
Also backup your host as well. It's one thing to back up your containers and VMs, but not enough TH-camrs talk about backing up the underlying/original host itself.
a good reason for not automating your life with any pc, you should have home assistant in a cluster for redundancy and a better backups - i had a raid fail recently myself - tough to mitigate
You should be able to create a Proxmox cluster or ceph cluster. That should give you a good way to keep Home Assistant working if your Proxmox server ever goes down again.
This is why i run vcenter. You can pay for a vmug subscriotion which gives you a 365 day trial keys for all the vmware products which is a year key. Wheree if younpurchased them would be in the many thousands per year. I have a 4 node HP DL380G9 vcenter cluster with zfs unraid running the SAN that all my vms operate under. Full backups of everything. Vcenter goes down, just attach vms to a new vcenter, servers go down, backyp zfs snapshots, and ccenter snapshots wirh full vm backuos too
You can use Veeam Agent for Linux to do a bare metal backup of Proxmox to a SMB/CIFS share then if it ever goes down, you boot from a Veeam recovery ISO, point it to the backup and it will put everything back exactly as it was. It's better than straight cloning your SSD because the backup file only takes the space you need on disk rather than the size of the disk and you can do compession, dedup, etc. Completely free to use the agent one as many machines as you want. They have a tool called Backup and Recovery which centralizes the management but it's only free for up to 10 devices.
The magic words your looking for to back up directories, etc. from one linux/unix install to something remote is "Rsync" You can set it up to run and grab whatever you want and pump it onto another server or even a PC if you need it there. Rsync is one of many tools, but that rabbit hole should get you going on cross backups, remote backups and help you figure out your crash plan. But ultimately you should use a config. manager that can execute a play book for automated bare metal installs. It takes some doing but you can set that up with something as simple as a raspberry pi or other low power SBC. There are lots of write ups geared at the home lab situation to establish your crash plan, just incase the situation warrants it, whether that's an upgrade, replacement or recovery. Other hot buzz words are "hardware agnostic", "PXE", "Bare Metal Automation", etc, etc.
migrate your home assistant stuff to something like a small nuc, so other people within your home doesnt suffer at least. then you'll figure out the rest
A bad ssd can strike whether it's in a NUC or a server chassis. A ZFS mirror of his boot drive would have allowed him to replace the drive without any downtime.
I'll backup the VM occasionally and have a Chromebook set aside specifically for the HAOS VM on it - otherwise its e-waste, but if my PVE box goes down I can jsut open up the chromebook and im right back in action...its a decent backup option i think and keeps some electronics out of the trash heap
I just have a strange feeling that having a server with high AF fans in an environment with both a water heater and a AC unit is not the best. Ngl the chaos is entertaining! Hope you get things sorted soon.
My Proxmox boot drive is a hardware managed 4x HGST 1 TB SATA 3 Gbps 7200 HDD HW RAID6 array (via Avago/Broadcom/LSI MegaRAID 12 Gbps SAS 9361-8i). That way, I am fault-tolerant upto two HDDs dying and I also won't have the KNOWN finite write endurance limit that SSDs have. I have a separate 8x 6 TB raidz2 pool for the VM OS boot drives, which is separate from my main 8x10 TB + 8x10 TB + 8x6 TB mass storage ZFS pool which consists of three 8-wide raidz2 vdevs. But yes, having to try and fix a dead boot drive that everything else depends on is a PITA.
You can use the free Veeam Linux agent and back-up your SSD to an external NAS / Storage / USB whatever.. if your SSD fails,, replace SSD, boot with the veeam recovery usb image, choose baremetal recovery, point to latest backup file, recover, boot and your live again
Do you really need a GPU? I'm running Blue Iris w/ 15x1080P cameras and give it 12 cores of a Xeon E5-2640 V4, 8GB of RAM, and the VM hovers around 13% CPU utilization.
Using a consumer grade SSD for Proxmox and then vms that spams stuff to disk (logs) is a desaster waiting to happen. SSD's have limited writes. Its far better to install 2 machanical drives in a raid 1 for the boot device. Set a cron to tell smart to report the drives health in an email and now you have an early warning system while not having to worry about limited writes. Their is a reason datacenters aren't fully SSD, while faster, data retention blows
I believe a "Complete backup for Proxmox" is called RAID 1. Also, you wouldn't have to worry about the "Mafia" and their lasers if you just drove under the speed limit. You know, like you're supposed to.
RAID 1 is a backup? LOL. No it's not. Try deleting a file and trying to get it back after it's gone. Or edit a file and save it. RAID 1 just replicates the command to both drives.
That sucks! I'm loving my 2-node Proxmox HA Cluster w/ QDevice VM running on my UnRAID server. Always install your OS in a mirror if possible which in the case of Proxmox is a ZFS mirror. Also, always install on an SSD when possible. Create a Proxmox share on your UnRAID server and attach it to Proxmox using NFS. Then schedule backups to your UnRAID server from Proxmox of all of your VM's. I currently have Untangle, Blue Iris, Nextcloud, Emby, and about 30 different containers running on it. ETA: Also, the OS should be the only thing installed on the drive. Don't use it for VM storage as it just isn't worth it. You can then get by with using 100GB SSD's or smaller enterprise SSD's which are insanely cheap.
Have raid 1 for your boot drive and this shit won't happen Also just clone the whole boot drive after setup and keep that or alternatively just image the drive and have a preconfigured iso on a bootable drive ready to go
My proxmox boot drives are in a zfs mirror as well as my VM storage. These VMs are backed up to a standalone instance of proxmox backup server and then replicated to my NAS so I have to lose 3 copies to not be able to restore a VM. Its easy enough to reinstall proxmox but my main concern is the data/VMs and losing all the configuration etc. I'm working on a 4th copy of my cricial don't want to lose stuff to OneDrive but for what I have already I'm pretty happy with the setup.
Proxmox has a couple of backup options and it’s meant to backup the VMs and CTs. And it’s always good to backup home assistant itself. I run an SMB backup to TrueNAS and backup to google cloud. Then Proxmox Backup Server backs up the VM. Also, start using enterprise-grade SSDs as consumer SSDs are not stable enough for constant read/writes that a server will do. I’ve learned this the hard way.
I've got hundreds of containers and VMs. Never lost more than 10 minutes if one failed completely because, well, BACKUPS. I don't understand the untrained. I really don't. And I have many servers at home to move things around too. Even clustered. It's almost like the software SUPPORTS THIS.
I'm late but I enjoyed this video. I too have the same usb sabrent and nvme m.2 heatsink. I hope you haven't lost the screw to attach it to the motherboard
What do you have for backups? I've been using a combination of Proxmox Backup Server and backups to NAS for years with no issues. Also, for cloning maybe keep a bootable USB around with Rescuezilla or Clonezilla.
Bro, I feel your pain. Recently my dual Xeon Proxmox died on me. You at least got through to the software side of things. Mine is just not taking power to start. It does seem to have power to run the flashy lights on the motherboard but nothing else. I'm dreading digging into fixing it because I have a feeling I'll have built a whole new system by the time I've got it fixed. I don't really want to do that. I did backup my VMs off host so I could reconstruct it but I'm not looking forward to getting to that part. So, I'm sending you a virtual beer and giving you the cheers sign. Good luck to us.
i recently bought a Z840 to replace my Z600 (which i will probably now add in to have a little cluster thing) so i get ya...that Z600 had some pains...Ill say though, I love this Z840 so much that I might just buy a second one, they're just THAT good (for me that is)
Pro for xcp-ng: It can do live migration, I think, better than Proxmox can. Cons for xcp-ng: Doesn't support virtio-fs (despite it being derived from a RHEL derived Linux) and Oliver Lawrence is a bit of a dick.
If you stay with a single NVME boot drive, I'd consider a $34 Optane NVME. Rated at 2 million hours between failure. Definitely not the safest option, but at least you are using a drive that is specifically designed to be durable. A better option would be a mirror of two Optane NVME drives. Plus, at $34 each, and a PCIE NVME adapter at $20...less than a C-Note.
hmm... good time to invest to add *3 proxmox PC (make it in to H.A cluster....) *you can do H.A cluster with 2.... but a lot of issue.... but the quorum thing is a pain....
Did you turn off ballooning memory for your Windows VM? Because at 20:13 it doesn’t look like you did, and iirc that’s an issue when doing PCIe passthrough
I concluded it is better to keep Blue Iris on it's own hardware, otherwise a bit like having all your eggs in one basket. If you had HA then I'd probably do it.
My dude, if you have a house that relies on proxmox and home assistance that much you need to have your primary home assistant Drive setup on a mirror raid. Doing that you could have had it fail over to that system or at least pull out the bad Drive and you'd be back online with the mirror Drive and after a quick rebuild you would have been back on your feet. 😅
I want to mirror my setup, and should have with the sata ssd. but now I have all my nvme slots filled so dont have any room for more nvme drives. which means I either have to go to sata ssd for mirror or figure something else out
@@Bytemybits Mirror the entire server with a second server running proxmox with HA. You can even run Proxmox on a rPi if you have the storage. Just get that sucker copied over.
Okay I'll comment a solution, setup a fog server box and boot your proxmox master over pxe once a week and have it dump the raw disk image to whatever storage you gave it
1. Don't speed. Problem solved. 2. Restore your Proxmox and HA VM from backup. 10 minutes later, problem solved. I don't see what's so hard about any of this???
but that doesnt actually backup proxmox right? Like, to do my Windows VM, I had to edit proxmox files. all those configs would still have to be redone right?
@@Bytemybits You will have to redo the system configs or copy those files from your current setup whenever you make a change. So I have to redo my GPU passthrough if I reinstall Proxmox but my containers and VM's are easily recreated.
Did you (or are you going to) fix the MyQ Home Assistant issue. I had recalled you set this up prior and now with the API changes in MyQ, it doesn't work with Home Assistant. Is there a fix?
What model Samsung was that that ‘went South’? (Samsung has had a rash of failures for drives made in a certain timeframe between certain months of 2021 thru Oct of 2022 ( I think?)
for clones - i would do a complete clone if you're worried about it...look up how to use the "dd" command and dd your current install to an .iso image and then if you have a failure you can then use the same "dd" command to restore to a new disk. I have also gone to larger and even SMALLER disks using the dd command, though to go to a smaller disk you'd need to play with file system size first and admittedly I have done it on windows and mac, not sure i have with linux.....anyways dd is slow and kinda brute force but it just goddamn works....other than that i think you can do a bit-for-bit backup using macrium reflect if you have a windows machine somewhere
Please record with dstat or iostat over a longer period of time. iotop would also be good with aggregated monitoring. Proxmox is very sensitive to RAM going south also. When a stick of RAM starts going south it can create errors and the way proxmox is designed with self healing, it can cause an endless read/write loop occurring, which has had history of taking out peoples SSDs over a very short period of time.
Samsung Evo/qvo are perfect for Linux use (seem to have random death issues after year or 2 or if they have long uptime I killed 840 and 850 ssd's running for more then 30 days, the MLC based samsung pro are fine, NVme 980 and higher is TLC and may have same problems as evo, as the 980 pro is just a super nvme evo plus) boot should be redundant, storage ideally raid6/z2
plug it in go to device manger and see if you see the drive there and of you see it change the letter and then try it again it is something 18:45 wow you have any desk installed on your computer you keep that installed and it won't be long you will be hacked always uninstall it when done using it crazy boy lol be safe.
@@Bytemybitsyou don't need Enterprise hardware for your setup. But you may want to look into ansible to help configure your PVE install if and when this happens again.
2 things: 1. My BI dedicated and going through Sherlock Holmes crap to troubleshoot. Going to check drives. 2. I ROFLd when I heard "c*nt hair" first time ever on YT.. Thanks for making my day!
This is a fantastic lesson in why people buy components w/ ECC memory. The filesystem corruption is likely caused by memory instability OR the consumer SATA controller. Also moving to a filesystem that runs consistency checks are part of it's normal operation like ZFS which Proxmox supports now fully.
how would ECC save a dying SSD? SSDs are not like hard drives, when they die they just die unlike HDDs which can sometimes give you errors for weeks before finally dying.
I honestly believe the SSD didn't have any problems. The bad ram stick just introduced corruption to the system and fdisk flagged the drive as unmountable to prevent further damage. However in the running theme of the channel: Jason threw money at the problem instead of properly reading the error messages.
@@justfasial01 It likely would have caught it faster, before bad data could be read or written. Most data integrity stuff is designed to fail fast and hard rather than slow and quietly.
@@xsillycarnifex When the SSD didn't show in BIOS with the USB adapter I assume the SSD is dead, doesn't look like data corruption but you're right ECC would help avoid that. The ssd looks about 5 years old and just out of it's warranty period, that's when you'd expect it to go bad so it makes sense it just died. ECC has it uses but this looks like just a dead ssd.
Nice thing wirh home assistant, the backup and restore is awesome. Mine backups to another share daily and since its a vcenter appliance, i also snappshot it twice daily and store snapshots on another zfs cluster
Proxmox is the reason I got into unraid in the first place. Its poorly coded installer cuts off the next button on whatever resolution the monitor next to my server uses and every time something goes wrong I have to move the computer out of my rack to a monitor where I can see said button to proceed with the install.
I did this recently with Unraid and cache drive, Basically once my cache drive filled past 90%(downloading a lot of files) it became corrupt. Had to rebuild all my dockers... I did not have a backup...dumb me. Now I have a raid 1 PNY 2-1TB cache. Lesson learned. Do not use SP nvme drives anymore. That is my 3rd drive with corruption.
Dude, I love your videos. This one is my favorite. Your excitement and frustration is genuine yet you are able to entertain all the way through. Don't change.
Building a mirrored ZFS for the boot drive would solve this issue. Also I believe when you have hardware passthrough enabled it has to reserve all the memory. The error on passthrough might be caused by using that gpu as the display out on the host, That usually causes issues. Sometimes blacklisting the gpu will work causing proxmox to run headless. But then you can run into the GPU reset bug and it still won't work. Best solution I've found is throwing in a cheap video card for the host to use, and make sure to set it as the default video output device in the bios. Then GPU passthrough probably should work.
"Building a mirrored ZFS for the boot drive would solve this issue."
Yes and no.
It would "solve" the boot (NVMe) SSD dying issue (sort of), but trying to get the GPU passthrough to work with ZFS root is a PITA.
(I know, because I tried. And as a result of my attempt, I ended up reverting back to a more conventional HW RAID6 that's managed by my MegaRAID SAS 12 Gbps 9361-8i instead. (Because fixing a HW RAID6 is easier than fixing a ZFS raidz2 sometimes.))
"Also I believe when you have hardware passthrough enabled it has to reserve all the memory. The error on passthrough might be caused by using that gpu as the display out on the host, That usually causes issues. Sometimes blacklisting the gpu will work causing proxmox to run headless. But then you can run into the GPU reset bug and it still won't work. Best solution I've found is throwing in a cheap video card for the host to use, and make sure to set it as the default video output device in the bios. Then GPU passthrough probably should work."
Depending on the combination of CPU and GPU that you're using, for some, you need to make sure that you have resizable BAR enabled in the BIOS, and that both your CPU, motherboard, and GPU supports it.
For the Proxmox console itself, you really DON'T need video output since all it tells you at the end is the IP address (which you probably already know) :8006 to log into the web UI where you're going to do everything else anyways.
The better way to check, IMO, is to install and enable openssh-server (# apt install -y openssh-server) and then making sure that you are able to ping the system and ssh into it.
If you pass that check, then you're able to do a LOT more over ssh than you would be via the local console anyways. (Once you have blacklisted nouveau and a bunch of other things (as a result of enabling GPU passthrough), the part where it tells you to log in from the :8006 might not even work, but the system itself is otherwise fully operational. So, again, the console output isn't always necessarily indicative of a fault nor problem with your system.
Again, ping and ssh are better able to ascertain that than said console output.
(If you have a server with IPMI, that can help as well. Or you can also get and deploy either PiKVM and/or TinyPilot KVM to perform a similar function, if your motherboard doesn't have IPMI built-in. Just be aware that those other "IPMI"-like tools needs a physical connection to the motherboard's power switch headers, etc., and therefore; unlike having multiple servers that have IPMI built-in, you won't be able to easily nor readily plug the PiKVM nor TinyPilot KVM from system to system to system without having to unplug the connections to the power switch headers on the motherboard to the power switch headers on another motherboard.)
Having the IPMI is like having a cheap GPU, except it doesn't take up a PCIe slot.
There are plenty of forum thread posts about BAR on the Proxmox forums.
Re: backing up Proxmox system.
You gotta remember it's an IT appliance; the individual nodes are considered worthless. The VMs and the cluster (if exists) is what matter to backup. Proxmox itself should be backed up by knowing how it is configured. Whether that is a configuration manager (IE: Ansible, Chef, or saltstack), a script that configures it, or just a piece of paper with all the changed settings listed.
Fully agree.
That's why I keep deployment notes in my OneNote.
Another thing that might also be useful (potentially) is dumping the history of your bash commands that you've used, and also dumping a list of the packages that you've installed as well.
(That way, if you DO need to recover the system, you can just "replay" the apt install "script" with all of the packages that you have installed post initial install (or, again, if you've saved a record of it in your deployment notes.)
The vast majority of the packages that are installed on the system -- it will likely skip (or say that it already has the latest or a newer version installed and therefore; nothing to do), but it will install the "delta" basically.
Probably easier to just to an image of the proxmox install to a spare drive you can swap in, and create regular image backups. Then you aren't jumping through hoops to get up and running again.
@@hycron1234 Three things about this:
1) Depends on where you end up storing or hosting said image.
2) You assume that there isn't a hardware failure that will prevent the successful redeployment of this image.
3) You also, further assume, that there aren't any hardware changes between the system that was imaged vs. the system that it will be imaged unto (which may or may not be true).
(IF the hardware changed between the source of the image and the destination, then there is a possibility that restoring the image may not work 100% of the time.)
If you have a clone of (all of) the hardware, then you could redeploy the image, without needing the second server to be on all the time, as a live mirrored copy as a part of your HA cluster.
your clusterf-vids are just both funny, chaotic and entertaining.. So... keep posting that shit.
A couple things, I have Blue Iris running on a 2016 VM using an ancient server tower with 7 cameras, and still only peg the cpus at 13-15%. My thoughts were to migrate to a PROXMOX server and here's why. They have HA (High Availability). This is an opportunity for you to set up a secondary server, even a lower speed system that your PROXMOX can fail over too. Even if all it does is backs up your VMs live so you can selectively choose which VMs you wish to run.
*Edit...did you GLUE the false fronts to your drawer boxes? Noooooooooo!
Exactly, this wouldn't even be an issue with HA.
Im running mirrored nvme boot drives, its kinda nice to have knowing that the system will always boot. Also running proxmox internal backup-tool too weekly backup vms to both my large raidz3 pool and a external usbdisk.
This is the way!
yup, I run that backup every sunday night - no RAID for me but my stuff can be lost and im soo broke they say im injured so, theres that
I love seeing videos like this I spent so many nights in college doing all this as well!!! The one thing I can recommend to you that I learned the hard way is always have a backup of your backup 😅 I currently have a 500GB 2.5 HDD plugged into my proxmox that runs a weekly cron job that backups all my VMs and that is then copied to my apple iCloud because it is the most reliable and don’t forget step 1 of the 3-2-1 method is having one offsite :)
Backups, Backups, Backups, Backups
Also backup your host as well.
It's one thing to back up your containers and VMs, but not enough TH-camrs talk about backing up the underlying/original host itself.
a good reason for not automating your life with any pc, you should have home assistant in a cluster for redundancy and a better backups - i had a raid fail recently myself - tough to mitigate
ProxMOX Clustering it is gold.!
You should be able to create a Proxmox cluster or ceph cluster. That should give you a good way to keep Home Assistant working if your Proxmox server ever goes down again.
Also, why not give Frigate a try instead of Blue Iris
This
@@aro56 Why not give backups a try?
I feel like Jason is on speed . or have smoked Liquids undies. 🤣🤣🤣🤣
you should really run a hardware diagnostic on the whole system and start from there
This is why i run vcenter. You can pay for a vmug subscriotion which gives you a 365 day trial keys for all the vmware products which is a year key. Wheree if younpurchased them would be in the many thousands per year. I have a 4 node HP DL380G9 vcenter cluster with zfs unraid running the SAN that all my vms operate under. Full backups of everything. Vcenter goes down, just attach vms to a new vcenter, servers go down, backyp zfs snapshots, and ccenter snapshots wirh full vm backuos too
You can use Veeam Agent for Linux to do a bare metal backup of Proxmox to a SMB/CIFS share then if it ever goes down, you boot from a Veeam recovery ISO, point it to the backup and it will put everything back exactly as it was. It's better than straight cloning your SSD because the backup file only takes the space you need on disk rather than the size of the disk and you can do compession, dedup, etc. Completely free to use the agent one as many machines as you want. They have a tool called Backup and Recovery which centralizes the management but it's only free for up to 10 devices.
The magic words your looking for to back up directories, etc. from one linux/unix install to something remote is "Rsync" You can set it up to run and grab whatever you want and pump it onto another server or even a PC if you need it there. Rsync is one of many tools, but that rabbit hole should get you going on cross backups, remote backups and help you figure out your crash plan. But ultimately you should use a config. manager that can execute a play book for automated bare metal installs. It takes some doing but you can set that up with something as simple as a raspberry pi or other low power SBC. There are lots of write ups geared at the home lab situation to establish your crash plan, just incase the situation warrants it, whether that's an upgrade, replacement or recovery. Other hot buzz words are "hardware agnostic", "PXE", "Bare Metal Automation", etc, etc.
migrate your home assistant stuff to something like a small nuc, so other people within your home doesnt suffer at least. then you'll figure out the rest
A bad ssd can strike whether it's in a NUC or a server chassis. A ZFS mirror of his boot drive would have allowed him to replace the drive without any downtime.
I'll backup the VM occasionally and have a Chromebook set aside specifically for the HAOS VM on it - otherwise its e-waste, but if my PVE box goes down I can jsut open up the chromebook and im right back in action...its a decent backup option i think and keeps some electronics out of the trash heap
@@angelgonzalez2379 that's also needed, but at least he can tinker with the server without bothering the home if he moves it to a dedicated machine
I just have a strange feeling that having a server with high AF fans in an environment with both a water heater and a AC unit is not the best. Ngl the chaos is entertaining! Hope you get things sorted soon.
sorry about your ordeal bro. Looking forward to your video on the temp sensors.
My Proxmox boot drive is a hardware managed 4x HGST 1 TB SATA 3 Gbps 7200 HDD HW RAID6 array (via Avago/Broadcom/LSI MegaRAID 12 Gbps SAS 9361-8i).
That way, I am fault-tolerant upto two HDDs dying and I also won't have the KNOWN finite write endurance limit that SSDs have.
I have a separate 8x 6 TB raidz2 pool for the VM OS boot drives, which is separate from my main 8x10 TB + 8x10 TB + 8x6 TB mass storage ZFS pool which consists of three 8-wide raidz2 vdevs.
But yes, having to try and fix a dead boot drive that everything else depends on is a PITA.
They dont even use laser here, they moved purely to distance/time calculation. mostly cause the only way to block it is to slow down :)
The easy answer is in the proxmox installer you can select two drives and install as a raid one.
Yes please create a video about your Home Assistant!
I do not use SSD drives for applications that have heavy write operations. The flash ram in them has limited numbers of writes. Chaotic.
You can use the free Veeam Linux agent and back-up your SSD to an external NAS / Storage / USB whatever.. if your SSD fails,, replace SSD, boot with the veeam recovery usb image, choose baremetal recovery, point to latest backup file, recover, boot and your live again
Do you really need a GPU? I'm running Blue Iris w/ 15x1080P cameras and give it 12 cores of a Xeon E5-2640 V4, 8GB of RAM, and the VM hovers around 13% CPU utilization.
Using a consumer grade SSD for Proxmox and then vms that spams stuff to disk (logs) is a desaster waiting to happen. SSD's have limited writes. Its far better to install 2 machanical drives in a raid 1 for the boot device. Set a cron to tell smart to report the drives health in an email and now you have an early warning system while not having to worry about limited writes. Their is a reason datacenters aren't fully SSD, while faster, data retention blows
I believe a "Complete backup for Proxmox" is called RAID 1.
Also, you wouldn't have to worry about the "Mafia" and their lasers if you just drove under the speed limit. You know, like you're supposed to.
But that defeats the purpose of owning such a fine automobile like the mustang
Raid is not a backup is a very popular saying for a reason.
RAID 1 is a backup? LOL. No it's not. Try deleting a file and trying to get it back after it's gone. Or edit a file and save it. RAID 1 just replicates the command to both drives.
That sucks!
I'm loving my 2-node Proxmox HA Cluster w/ QDevice VM running on my UnRAID server.
Always install your OS in a mirror if possible which in the case of Proxmox is a ZFS mirror. Also, always install on an SSD when possible.
Create a Proxmox share on your UnRAID server and attach it to Proxmox using NFS. Then schedule backups to your UnRAID server from Proxmox of all of your VM's.
I currently have Untangle, Blue Iris, Nextcloud, Emby, and about 30 different containers running on it.
ETA: Also, the OS should be the only thing installed on the drive. Don't use it for VM storage as it just isn't worth it. You can then get by with using 100GB SSD's or smaller enterprise SSD's which are insanely cheap.
Have raid 1 for your boot drive and this shit won't happen
Also just clone the whole boot drive after setup and keep that or alternatively just image the drive and have a preconfigured iso on a bootable drive ready to go
If you turned everything into smart devices, how did you manage to control switches without HA?
My proxmox boot drives are in a zfs mirror as well as my VM storage. These VMs are backed up to a standalone instance of proxmox backup server and then replicated to my NAS so I have to lose 3 copies to not be able to restore a VM. Its easy enough to reinstall proxmox but my main concern is the data/VMs and losing all the configuration etc. I'm working on a 4th copy of my cricial don't want to lose stuff to OneDrive but for what I have already I'm pretty happy with the setup.
Proxmox has a couple of backup options and it’s meant to backup the VMs and CTs. And it’s always good to backup home assistant itself. I run an SMB backup to TrueNAS and backup to google cloud. Then Proxmox Backup Server backs up the VM.
Also, start using enterprise-grade SSDs as consumer SSDs are not stable enough for constant read/writes that a server will do. I’ve learned this the hard way.
I've got hundreds of containers and VMs. Never lost more than 10 minutes if one failed completely because, well, BACKUPS. I don't understand the untrained. I really don't. And I have many servers at home to move things around too. Even clustered. It's almost like the software SUPPORTS THIS.
I'm late but I enjoyed this video.
I too have the same usb sabrent and nvme m.2 heatsink. I hope you haven't lost the screw to attach it to the motherboard
I feel you pain. I’ve been going through this lately myself.
This feels a lot like my experience with homelabbing.
pain, payoff, repeat LOL
Have you considered using the Home Assistant Google Drive Backup add-on?
What do you have for backups? I've been using a combination of Proxmox Backup Server and backups to NAS for years with no issues.
Also, for cloning maybe keep a bootable USB around with Rescuezilla or Clonezilla.
You wanna avoid io errors in proxmox? Make sure your main lvm doesnt fill up
Bro, I feel your pain. Recently my dual Xeon Proxmox died on me. You at least got through to the software side of things. Mine is just not taking power to start. It does seem to have power to run the flashy lights on the motherboard but nothing else. I'm dreading digging into fixing it because I have a feeling I'll have built a whole new system by the time I've got it fixed. I don't really want to do that. I did backup my VMs off host so I could reconstruct it but I'm not looking forward to getting to that part. So, I'm sending you a virtual beer and giving you the cheers sign. Good luck to us.
i recently bought a Z840 to replace my Z600 (which i will probably now add in to have a little cluster thing) so i get ya...that Z600 had some pains...Ill say though, I love this Z840 so much that I might just buy a second one, they're just THAT good (for me that is)
Cheers to our shared pain lol
For configuration, you need the rabbit hole that is ansible in your life.
I have it where every system I have can be reconfigured in minutes
I don’t use proxmox, I use XCP-ng. But I also have my storage and compute in 2 different boxes. If I lose a host, I can start the VM on another host.
Pro for xcp-ng: It can do live migration, I think, better than Proxmox can.
Cons for xcp-ng: Doesn't support virtio-fs (despite it being derived from a RHEL derived Linux) and Oliver Lawrence is a bit of a dick.
That "Content Cop" intro music :)
My Samsung SSD M.2 drive failed on me last week. I lost Plex and all my proxmox, docker vm documentation. Good times lol
I always mirror my Proxmox boot drives.
If you stay with a single NVME boot drive, I'd consider a $34 Optane NVME. Rated at 2 million hours between failure. Definitely not the safest option, but at least you are using a drive that is specifically designed to be durable. A better option would be a mirror of two Optane NVME drives. Plus, at $34 each, and a PCIE NVME adapter at $20...less than a C-Note.
hmm... good time to invest to add *3 proxmox PC (make it in to H.A cluster....)
*you can do H.A cluster with 2.... but a lot of issue....
but the quorum thing is a pain....
the zigbee thing is the thing. Its connected to a specific dongle and changing to a different one is a whole thing
Sorry I might need some explaination on this motorcycle mugginng situation, im curious as to what happens in this situation
Think he is talking about the police catching him for speeding using a laser radar gun.
Did you turn off ballooning memory for your Windows VM? Because at 20:13 it doesn’t look like you did, and iirc that’s an issue when doing PCIe passthrough
I concluded it is better to keep Blue Iris on it's own hardware, otherwise a bit like having all your eggs in one basket. If you had HA then I'd probably do it.
My dude, if you have a house that relies on proxmox and home assistance that much you need to have your primary home assistant Drive setup on a mirror raid.
Doing that you could have had it fail over to that system or at least pull out the bad Drive and you'd be back online with the mirror Drive and after a quick rebuild you would have been back on your feet. 😅
This is why you should have set things up better. MIrrored array for boot. Raid 6 (or zfs with spares). Or at the least, a NAS with backups.
I want to mirror my setup, and should have with the sata ssd. but now I have all my nvme slots filled so dont have any room for more nvme drives. which means I either have to go to sata ssd for mirror or figure something else out
@@Bytemybits Mirror the entire server with a second server running proxmox with HA. You can even run Proxmox on a rPi if you have the storage. Just get that sucker copied over.
You might want to watch this - cloning system drives.
Turn off memory ballooning for the IRIS vm.
No mirror for the system drive on a critical server. Good job!
The 970 ssd needed the firmware to keep it from dying lmfao.
Did you not think about RAID and disk redundancy before putting all your eggs in that basket? You learned the hard way.
I have been running Blue Iris on a Windows 10 VM for a few months now on UnRaid with no issues. What problems have you had?
If you do figure out how to backup your proxmox server please make a video on it
I love your style and you really make me wanna upload similar videos
that home assitant theme is awesome. Can I have it? =)
Okay I'll comment a solution, setup a fog server box and boot your proxmox master over pxe once a week and have it dump the raw disk image to whatever storage you gave it
at least, you have the backup, right?
Put all your eggs in one basket and swap to Unraid.
You are trying to pass through your video card to Blue Iris and you have an issue with PCI passthrough.
1. Don't speed. Problem solved. 2. Restore your Proxmox and HA VM from backup. 10 minutes later, problem solved. I don't see what's so hard about any of this???
+1 for the cursing, do it more!
If you wanna clone a disk, you can use a boot ball Linux of any variety and use dd commands manually or you can use a tool like gdisk for a gui.
Wonder if that laser works on radar 😂
You def lost a drive. This is why I mirror my boot drive lol and the blue screen whea error is from an overclock that is unstable.
unless a hardware issue, eff proxmox !! xcp-ng for the win !!!
Proxmox backup server is your best friend. Easily redeploy your vm’s in minutes. Even if your proxmox instance dies.
but that doesnt actually backup proxmox right? Like, to do my Windows VM, I had to edit proxmox files. all those configs would still have to be redone right?
@@Bytemybits You will have to redo the system configs or copy those files from your current setup whenever you make a change. So I have to redo my GPU passthrough if I reinstall Proxmox but my containers and VM's are easily recreated.
Never had much luck with proxmox
Do you have a video of your setup/homesense hardware, sensor, etc? Because that looks freaking awesome :O
I had 2 of those Samsung 500g evo SSD’s and they both failed. They are bunk as f
I have been there. For the last 60 years.
This year alone has been bad hardware wise for me lol.
Did you (or are you going to) fix the MyQ Home Assistant issue. I had recalled you set this up prior and now with the API changes in MyQ, it doesn't work with Home Assistant. Is there a fix?
What model Samsung was that that ‘went South’? (Samsung has had a rash of failures for drives made in a certain timeframe between certain months of 2021 thru Oct of 2022 ( I think?)
860evo
for clones - i would do a complete clone if you're worried about it...look up how to use the "dd" command and dd your current install to an .iso image and then if you have a failure you can then use the same "dd" command to restore to a new disk. I have also gone to larger and even SMALLER disks using the dd command, though to go to a smaller disk you'd need to play with file system size first and admittedly I have done it on windows and mac, not sure i have with linux.....anyways dd is slow and kinda brute force but it just goddamn works....other than that i think you can do a bit-for-bit backup using macrium reflect if you have a windows machine somewhere
Clonezilla can clone proxmox boot drives.
Please record with dstat or iostat over a longer period of time. iotop would also be good with aggregated monitoring. Proxmox is very sensitive to RAM going south also. When a stick of RAM starts going south it can create errors and the way proxmox is designed with self healing, it can cause an endless read/write loop occurring, which has had history of taking out peoples SSDs over a very short period of time.
i was planning on trying the ram again after all is settled. but sounds like IF i get it to boot, i need to do full memtest?
This is how I live my life lol
Samsung Evo/qvo are perfect for Linux use (seem to have random death issues after year or 2 or if they have long uptime I killed 840 and 850 ssd's running for more then 30 days, the MLC based samsung pro are fine, NVme 980 and higher is TLC and may have same problems as evo, as the 980 pro is just a super nvme evo plus)
boot should be redundant, storage ideally raid6/z2
Dafuq? I have two 840 evo for at least 5 years and no issues. Maybe you use discard or let systemd trim them?
syncoid for the promxox boot backups
Backup and restoration plan, bro. For next time.
mirror your install drive so you always have a back up
plug it in go to device manger and see if you see the drive there and of you see it change the letter and then try it again it is something 18:45 wow you have any desk installed on your computer you keep that installed and it won't be long you will be hacked always uninstall it when done using it crazy boy lol be safe.
How long is it going to take people telling you for years to run actual enterprise level hardware before you just do it?
I've had two enterprise level servers... they just get gutted and replaced with things i can afford that are faster lol (until they break ofc...)
@@Bytemybitsyou don't need Enterprise hardware for your setup. But you may want to look into ansible to help configure your PVE install if and when this happens again.
Your electricity bill must be sky high…
synology does a bare metal backup for os
2 things:
1. My BI dedicated and going through Sherlock Holmes crap to troubleshoot. Going to check drives.
2. I ROFLd when I heard "c*nt hair" first time ever on YT..
Thanks for making my day!
Take this as a lesson as to why redundancy is important. Always Raid, even if it's 2 disks, it's better than one
This is a fantastic lesson in why people buy components w/ ECC memory. The filesystem corruption is likely caused by memory instability OR the consumer SATA controller. Also moving to a filesystem that runs consistency checks are part of it's normal operation like ZFS which Proxmox supports now fully.
how would ECC save a dying SSD? SSDs are not like hard drives, when they die they just die unlike HDDs which can sometimes give you errors for weeks before finally dying.
I honestly believe the SSD didn't have any problems. The bad ram stick just introduced corruption to the system and fdisk flagged the drive as unmountable to prevent further damage. However in the running theme of the channel: Jason threw money at the problem instead of properly reading the error messages.
So his SSD would not have failed if he’d been using ZFS and/or ECC RAM? :)
@@justfasial01 It likely would have caught it faster, before bad data could be read or written. Most data integrity stuff is designed to fail fast and hard rather than slow and quietly.
@@xsillycarnifex When the SSD didn't show in BIOS with the USB adapter I assume the SSD is dead, doesn't look like data corruption but you're right ECC would help avoid that.
The ssd looks about 5 years old and just out of it's warranty period, that's when you'd expect it to go bad so it makes sense it just died. ECC has it uses but this looks like just a dead ssd.
Nice thing wirh home assistant, the backup and restore is awesome. Mine backups to another share daily and since its a vcenter appliance, i also snappshot it twice daily and store snapshots on another zfs cluster
Proxmox is the reason I got into unraid in the first place. Its poorly coded installer cuts off the next button on whatever resolution the monitor next to my server uses and every time something goes wrong I have to move the computer out of my rack to a monitor where I can see said button to proceed with the install.
Quit watching after 30 seconds.
Nevermind, didn't watch everything :+
Or just stay somewhere near the speed limit?
I did this recently with Unraid and cache drive, Basically once my cache drive filled past 90%(downloading a lot of files) it became corrupt. Had to rebuild all my dockers... I did not have a backup...dumb me. Now I have a raid 1 PNY 2-1TB cache. Lesson learned. Do not use SP nvme drives anymore. That is my 3rd drive with corruption.
Autorestic back up to backblaze.