I wanted to add that even though the Proxmox installer will enforce selecting same size disks when creating a mirrored ZFS boot drive, you can still use disks with different sizes and have a mirrored ZFS boot drive. To do that you initially need to select zfs (RAID0) and select only one of your boot drives (the smaller one of the two) in the Proxmox installer ie. create stripe boot pool with a single drive. After completing your Proxmox installation you can create a partition on your second disk that matches the size of the first disk and also change the type of your boot zfs pool from stripe (RAID0) to mirror (RAID1) by adding the newly created and size matched partition on your second drive to the pool. This can be especially useful if you don't have any m.2 slots, because you use older hardware predating m.2 and you don't want to spend money on 2 small SSDs even if they are quite cheap, you can get used 150-300 GB hard drives for literally nothing (
6:27 --- 10TB writes per year (or 30GB /day) in a PVE cluster . . . insightful information. Jolly superb video 👍 Your mannerisms evoke the "ExplainingComputers" YT channel (perhaps the non-British version). I enjoy your content covering production deployments with hyper-scaled / converged / private cloud / composable (whatever ppl deign to label it next). P.s. Whereas RAID discussions are as old as time immemorial, I understand that recovering a mirror in RAID-1 is not trivial. Kind regards, neighbours and friends and happy turkeys 2022.
Thank you for putting this together. I spent hours going down the ssd wear rabbit hole and figuring out what to put on which drive - could have saved that time with yr video. yr video helped clarify a ton
super informative video! I just picked up 2 Intel optane p1600x 58gb drives for $30 each. Those little bastards are perfect for server boot drivers, they have amazing latency and IOPS and their write endurance is off the charts! Got a few more of the 118gb p1600x to help accelerate my ZFS workflow
Quality videos as always! I will be referencing this video to make people realize just how write intensive Proxmox really is. You have no idea how many production servers I have seen around running on a single SanDisk flash drive as the boot drive! Personally I try to go with two SSDs in Raid 1 where possible and as a hard prerequisite in the production environment, regardless of their operation size. That being said, since Raid is not a form of backup, I was wondering if there is a way to make actual backups of your boot drive in Proxmox or not?
Great video. Many thanks! The fact that you share you measurements on written data is really helpful. Some questions that are now bouncing in my head.... If you just use PVE as stand alone (no cluster), does the amount of written data per day come down? How much of it is logging? And... how can we move log files to other drives/server?
All four servers in my cluster boot from pairs of SATA SSDs in a ZFS mirror (what Proxmox calls RAID 1), with data via Ceph and NFS. A fifth standalone server runs a single NVMe SSD for boot and data.
More great stuff ... really!! only one comment .. but maybe will not work due to U-tube.. put all PM in one video ... thanks for all your very detailed and good videos .. I win lottery ... you win.. ;)
Would you want one super long video with all my Proxmox content? I could edit the videos together to create a long video like that if there was demand for it. I also have a playlist on my TH-cam channel that has all my Proxmox videos in one place and the videos will autoplay like how one long video would.
I have pcie cards holding ssd’s. Used to run TrueNAS from a USB stick (usable for up to 2 years before they needed to be reformatted) but switched to pcie card with ssd.
For my future Home-Server I plan thw Solidigm SSD D3-S4510 240GB for boot drive. It is a SSD with 2M MTBF + 900TB TBW - and quite cheap for that. But a version with fewer GByte is not available.
Great video. I had a USB drive fail and I'm still trying to recover my VM information. I'm getting ready to do ZFS mirror Drive like your suggesting for a boot disk. You had mentioned that you can set up another disk for proxmox to do it s temporary writes to but I haven't seen where to set that up.
You can turn a single boot drive into a mirror if your using ZFS on the boot drive and the new drive is the same size or bigger than the existing drive. There is no way of doing this in the gui. I plan on making a video on this in the future but the process is to copy the partition table, randomize the guid of the new drive, add the mirror in zfs, then set up the boot loader.
I should try this after changing to noatime but it does seem to significantly lower writes and improves performance. I’m guessing a good amount of those writes were for logging access time.
Awesome videos, thank you! Is it feasible to boot off PXE for a PVE host, especially for a small PVE cluster? I'm intrigued by the idea of redirecting logging to a log PC and having configs stored on a PXE server with storage redundancy. This sounds like an interesting approach to explore! Just subscribed, by the way :)
I haven't looked into doing PXE booting into Proxmox. I'll take a look at it and see if possible and practical, and may make a video about it. Thanks for the idea.
I was using ZFS, and ZFS when setup as a boot drive in Proxmox doesn't have any swap configured by default. I'm still looking more into what may explain where all the writes are coming from.
Three issues for your next Boot video: o How to set up booting with Root on RAIDz. o How to install ZFS Boot Menu (Nobody has a video on ZBM.) o How to recover if the wrong drive fails when root is on ZFS.
You recommend a 128GB SSD boot drive as well as a raid mirror. ZFS has got to tax those drives pretty hard. I'll probably just use a single SSD in ext4.
A year later and I still face a similar question: should I have a boot drive that is an MD array (RAID 1, NVMe)? Or a simple boot drive but RAID 1 for the data drive? Or RAID 1 on ZFS? Also I have 2*1TB Samsung 980Pro, so perhaps I could use those for boot drive and data? Or is that discouraged? File storage is much simpler IMO. I'll probably run whatever I currently have (RAID5 with Ext4 on it).
My servers won't boot to NVME but a few years ago I installed ubuntu onto a tiny ssd for /boot and everything else on the NVME. Wondering could I do the same with Proxmox and utilize three 118gb Optane NVMEs for the OS drives in my cluster and keep the SATAs just as the boot partition. In this configuration I would think my 1tb 970 Evo Data SSDs as boot would outlast the hardware.
Thank you for explanation. Can you check on any of your servers how does zfs mirror looks like? Main question: does both drives have boot record and does system can boot from any of two drives in case of failure? Practical test will be very appreciated :) thanks
The boot record is stored on boot drives with a ZFS mirrored setup. Grub and other boot loader files are stored on both drives, so both drives can be booted from. When replacing a drive, there is a guide on the Proxmox website to follow, as just replacing the drive in ZFS won't let the second drive be bootable.
Are there any speed-related issues when using flash drives as boot drives for Proxmox? I have a few mini PCs with 1TB NVMe drives but no options for disk expansion
Just found your channel and subscribed immediately. I am running Proxmox 7.3 in home-lab environment on a pc. How do I install PVE 7.3 on a 58GB Intel Optane SSD and use that drive for OS only?
To install Proxmox on the Optane SSD, boot from the Proxmox installer ISO, then select the Optane SSD as the drive you will install on. If its easy I sometimes remove all the other drives so I can't accidentally install on a drive I didn't want to. Then once the OS is installed, add your other storage options, and don't store anything else on the local drives, and the local drive will be used for boot only.
Yea a usb 2 to sata to ssd would work well. Usb drives are generally pretty reliable these days and the performance should be plenty for a boot drive. My usb 2 drives in using as a boot drive for me server now is plenty of speed and since I have 2 in raid 1 I can easily replace them as needed.
Great video, very useful for my installation ! Is there a difference between m.2 nvme at 2500mb/s (or more) in writing and ssd sata 3 up to 500mb/s max for boot proxmox ?
I haven’t done a test of usb drives to have any hard data, but most of the usb sticks I have used are of fairly low quality. I’d generally recommend a sb ssd as they should be much more reliable and faster than most usb sticks.
@@ElectronicsWizardry Thanks for getting back to me. Limited to 1 m2 and 1 sata drive in a small pc so I was thinking about usb sticks or a high endurance sd cards in usb adapter since mirror is impossible otherwise. What do you recommend ?
@@vincentgallo3351 Can't say about ElectronicsWizardry's recommendations, but personally would recommend a Kingston KC3000, a bit overkill capacity wise, but they have great tbw and can be had for around 50 usd (at least locally for me). Have a Corsair Force M510 as boot drive in my server currently, but their a bit older and harder to find at this point and endurance wise (at least at lowest capacity) are a bit worse than the Kingston. Combine it with cheap m.2 enclosure and it should last basically the whole life of the server (quite possibly even outlive the server) in this use case.
I have 2 consumer SSDs 120gb and 256gb. Is it possible to make over provision onto 256gb drive and make its capacity 128gb to make more endurance. Is it overkill?
Over-provisioning can only be done within a drive as it relys on knowledge of how much each physical block is used, and that isn't shared over the sata/nvme data. You can do something like a stripe, raid0, or jbod that would split the writes up, but I'd think its best to just keep the second drive as a replacement in case the primary one fails.
@@ElectronicsWizardry my idea was that I would have z-raid1 with two same size drives and primarily point of view was the possibility of using leftover space for increasing TBW (on controller lever of the drive) I just have these two drives on hand and wanna make raid1 with them and don't wanna to trim and lose space on second drive
@@neins Raidz1 is a single parity solution, and typically used for 3 or more drives. I think you want a mirror here so both drives have the same data, and if one drives dies you don't lose any data. You would get 128GB of space with this configuration.
@@neins With a mirror setup there would be 2 128GB mirrored chunks, one on the 128GB SSD, and the other 128GB would be 1/2 of the second SSD. The empty space on the larger SSD would be used for overprovisioning. But with SSDs these days I wouldn't worry too much about having extra overprovisioning space and endurance isn't a big issue with most SSDs and use cases.
I'm far from the best programmer di there is sgood chance I didn't do that fully right. But there is a shebang at the top that tells the system to runcthr script on python3. You should be able to download the script and then make it executable with chmod +x diskWrites.sh, and the run the script with ./diskWrites.sh. Hope this helps.
@@ElectronicsWizardry Thanks, I forgot completely to make it executable. I don't understand why, but I get "./diskWrites.sh: line 1: payload:allShortcutsEnabled:false: command not found". I checked the script, as always even though I'm not programmer at all and don't understand why that env line doesn't work. Tried running it in Proxmox out of curiosity
@@ElectronicsWizardry Oh never mind. Me stupid ass downloaded the html format from github instead of raw. Reminder to myself "always check the local file". Man I feel stupid 🤦🏻 It works just fine..
![ตีสิบเดย์ [FULL] | นางเอกหมอลำเงินล้าน "อุ๋งอิ๋ง เพชรบ้านแพง" & ไฮโซสาวสุดฮอต "ลิลลี่ เหงียน"](http://i.ytimg.com/vi/jPxQS_EgrJ4/mqdefault.jpg)
Man your videos have such valuable information, really helps with getting the most bang for the buck, you rock!
I wanted to add that even though the Proxmox installer will enforce selecting same size disks when creating a mirrored ZFS boot drive, you can still use disks with different sizes and have a mirrored ZFS boot drive.
To do that you initially need to select zfs (RAID0) and select only one of your boot drives (the smaller one of the two) in the Proxmox installer ie. create stripe boot pool with a single drive. After completing your Proxmox installation you can create a partition on your second disk that matches the size of the first disk and also change the type of your boot zfs pool from stripe (RAID0) to mirror (RAID1) by adding the newly created and size matched partition on your second drive to the pool.
This can be especially useful if you don't have any m.2 slots, because you use older hardware predating m.2 and you don't want to spend money on 2 small SSDs even if they are quite cheap, you can get used 150-300 GB hard drives for literally nothing (
You can disable cluster services if not used, they are VERY active with logging.
6:27 --- 10TB writes per year (or 30GB /day) in a PVE cluster . . . insightful information.
Jolly superb video 👍 Your mannerisms evoke the "ExplainingComputers" YT channel (perhaps the non-British version). I enjoy your content covering production deployments with hyper-scaled / converged / private cloud / composable (whatever ppl deign to label it next).
P.s. Whereas RAID discussions are as old as time immemorial, I understand that recovering a mirror in RAID-1 is not trivial.
Kind regards, neighbours and friends and happy turkeys 2022.
Just found your channel. Excellent content and a solid review! - Another sub for you sir.
Thank you for putting this together. I spent hours going down the ssd wear rabbit hole and figuring out what to put on which drive - could have saved that time with yr video. yr video helped clarify a ton
super informative video! I just picked up 2 Intel optane p1600x 58gb drives for $30 each. Those little bastards are perfect for server boot drivers, they have amazing latency and IOPS and their write endurance is off the charts! Got a few more of the 118gb p1600x to help accelerate my ZFS workflow
when BTRFS comes out of beta in late 2049 it should make a great boot drive :)
Your a Gaddamn Legend. Thank you sir!
Great content. You're killing it dude!
All you vids are just great ... in depth which few do... thank you very much.....
Thank you for the very thorough video.
Quality videos as always!
I will be referencing this video to make people realize just how write intensive Proxmox really is. You have no idea how many production servers I have seen around running on a single SanDisk flash drive as the boot drive!
Personally I try to go with two SSDs in Raid 1 where possible and as a hard prerequisite in the production environment, regardless of their operation size.
That being said, since Raid is not a form of backup, I was wondering if there is a way to make actual backups of your boot drive in Proxmox or not?
The Dell R630s/R640s internal SD cards work great for this as well.
Great, concise video. Very useful for my upcoming home build, thank you
Over provisioning helps with wear endurance on boot drives. Personally, I prefer TLC data centre SSD's with high writes endurance and MTBF
Great video. Many thanks!
The fact that you share you measurements on written data is really helpful.
Some questions that are now bouncing in my head....
If you just use PVE as stand alone (no cluster), does the amount of written data per day come down?
How much of it is logging? And... how can we move log files to other drives/server?
Good info thanks for putting this together
This a great video. Learned a lot. Thank you!
Can the logging be sent to a log server in order not to stress the boot drive? Or disabled for good?
Hi. Can you do a video on how you make ssd to run only the boot drive? And do you store VMs in it or in a different drive altogether?
All four servers in my cluster boot from pairs of SATA SSDs in a ZFS mirror (what Proxmox calls RAID 1), with data via Ceph and NFS. A fifth standalone server runs a single NVMe SSD for boot and data.
More great stuff ... really!! only one comment .. but maybe will not work due to U-tube.. put all PM in one video ... thanks for all your very detailed and good videos .. I win lottery ... you win.. ;)
Would you want one super long video with all my Proxmox content? I could edit the videos together to create a long video like that if there was demand for it. I also have a playlist on my TH-cam channel that has all my Proxmox videos in one place and the videos will autoplay like how one long video would.
@@ElectronicsWizardry That you even considered this request shows the content and quality of your character.
I like gentleman scholars!
I have pcie cards holding ssd’s.
Used to run TrueNAS from a USB stick (usable for up to 2 years before they needed to be reformatted) but switched to pcie card with ssd.
For my future Home-Server I plan thw Solidigm SSD D3-S4510 240GB for boot drive. It is a SSD with 2M MTBF + 900TB TBW - and quite cheap for that. But a version with fewer GByte is not available.
That's one thing that really messed me up starting with proxmox. Why does it split the boot drive!?! At least let me pick how it's split.
Great video. I had a USB drive fail and I'm still trying to recover my VM information. I'm getting ready to do ZFS mirror Drive like your suggesting for a boot disk. You had mentioned that you can set up another disk for proxmox to do it s temporary writes to but I haven't seen where to set that up.
You can turn a single boot drive into a mirror if your using ZFS on the boot drive and the new drive is the same size or bigger than the existing drive. There is no way of doing this in the gui. I plan on making a video on this in the future but the process is to copy the partition table, randomize the guid of the new drive, add the mirror in zfs, then set up the boot loader.
When booting a Linux box off a USB flash drive I've found that using the noatime flag in fstab makes a massive difference in longevity.
I should try this after changing to noatime but it does seem to significantly lower writes and improves performance. I’m guessing a good amount of those writes were for logging access time.
Awesome videos, thank you!
Is it feasible to boot off PXE for a PVE host, especially for a small PVE cluster? I'm intrigued by the idea of redirecting logging to a log PC and having configs stored on a PXE server with storage redundancy. This sounds like an interesting approach to explore!
Just subscribed, by the way :)
I haven't looked into doing PXE booting into Proxmox. I'll take a look at it and see if possible and practical, and may make a video about it. Thanks for the idea.
One suggestion I heard is to use a usb flash drive and turn off swap. Not sure if that solves issues with temp and logs thrashing the drive.
I was using ZFS, and ZFS when setup as a boot drive in Proxmox doesn't have any swap configured by default. I'm still looking more into what may explain where all the writes are coming from.
Three issues for your next Boot video: o How to set up booting with Root on RAIDz. o How to install ZFS Boot Menu (Nobody has a video on ZBM.) o How to recover if the wrong drive fails when root is on ZFS.
You recommend a 128GB SSD boot drive as well as a raid mirror. ZFS has got to tax those drives pretty hard. I'll probably just use a single SSD in ext4.
I had quite a few SSDs suddenly 'die' over the years. RAID1 of 2 120GB SSDs is a smart move - you will sleep better.
A year later and I still face a similar question: should I have a boot drive that is an MD array (RAID 1, NVMe)? Or a simple boot drive but RAID 1 for the data drive? Or RAID 1 on ZFS?
Also I have 2*1TB Samsung 980Pro, so perhaps I could use those for boot drive and data? Or is that discouraged?
File storage is much simpler IMO. I'll probably run whatever I currently have (RAID5 with Ext4 on it).
Btrfs also has raid1 option...
My servers won't boot to NVME but a few years ago I installed ubuntu onto a tiny ssd for /boot and everything else on the NVME. Wondering could I do the same with Proxmox and utilize three 118gb Optane NVMEs for the OS drives in my cluster and keep the SATAs just as the boot partition. In this configuration I would think my 1tb 970 Evo Data SSDs as boot would outlast the hardware.
Thank you for explanation. Can you check on any of your servers how does zfs mirror looks like? Main question: does both drives have boot record and does system can boot from any of two drives in case of failure? Practical test will be very appreciated :) thanks
The boot record is stored on boot drives with a ZFS mirrored setup. Grub and other boot loader files are stored on both drives, so both drives can be booted from. When replacing a drive, there is a guide on the Proxmox website to follow, as just replacing the drive in ZFS won't let the second drive be bootable.
Are there any speed-related issues when using flash drives as boot drives for Proxmox? I have a few mini PCs with 1TB NVMe drives but no options for disk expansion
Just found your channel and subscribed immediately. I am running Proxmox 7.3 in home-lab environment on a pc. How do I install PVE 7.3 on a 58GB Intel Optane SSD and use that drive for OS only?
To install Proxmox on the Optane SSD, boot from the Proxmox installer ISO, then select the Optane SSD as the drive you will install on. If its easy I sometimes remove all the other drives so I can't accidentally install on a drive I didn't want to. Then once the OS is installed, add your other storage options, and don't store anything else on the local drives, and the local drive will be used for boot only.
Dell BOSS Wohooo!
How about booting from a SSD connected with a sata to USB2 ?
No endurance issues and still keep the max number of drives.
Yea a usb 2 to sata to ssd would work well. Usb drives are generally pretty reliable these days and the performance should be plenty for a boot drive. My usb 2 drives in using as a boot drive for me server now is plenty of speed and since I have 2 in raid 1 I can easily replace them as needed.
Smicro sdom I wish there was 2 sockets so I could mirror
Do you have any advice on naming conventions for single and clustered volumes?
Great video, very useful for my installation !
Is there a difference between m.2 nvme at 2500mb/s (or more) in writing and ssd sata 3 up to 500mb/s max for boot proxmox ?
For boot drive use a SATA SSD vs a NVMe SSD won't make any different performance wise. The extra performance won't matter for boot uses.
Which USB drives have high endurance for running mirrored boot ?
I haven’t done a test of usb drives to have any hard data, but most of the usb sticks I have used are of fairly low quality. I’d generally recommend a sb ssd as they should be much more reliable and faster than most usb sticks.
@@ElectronicsWizardry Thanks for getting back to me. Limited to 1 m2 and 1 sata drive in a small pc so I was thinking about usb sticks or a high endurance sd cards in usb adapter since mirror is impossible otherwise. What do you recommend ?
@@vincentgallo3351 Can't say about ElectronicsWizardry's recommendations, but personally would recommend a Kingston KC3000, a bit overkill capacity wise, but they have great tbw and can be had for around 50 usd (at least locally for me). Have a Corsair Force M510 as boot drive in my server currently, but their a bit older and harder to find at this point and endurance wise (at least at lowest capacity) are a bit worse than the Kingston. Combine it with cheap m.2 enclosure and it should last basically the whole life of the server (quite possibly even outlive the server) in this use case.
@@h.b.5577 THX!
The 30gb of writes per day is really the reason to avoid zfs.
Hello, could you make a tutorial on how to spin-down hdd in Proxmox? thanks in advance!
Thanks for the idea. Adding this to list of future videos to work on.
I dont know what a promox drive is, it just appeared in my recommended. Well thanks but I cant find any usefulness in it for me
I have 2 consumer SSDs 120gb and 256gb. Is it possible to make over provision onto 256gb drive and make its capacity 128gb to make more endurance. Is it overkill?
Over-provisioning can only be done within a drive as it relys on knowledge of how much each physical block is used, and that isn't shared over the sata/nvme data. You can do something like a stripe, raid0, or jbod that would split the writes up, but I'd think its best to just keep the second drive as a replacement in case the primary one fails.
@@ElectronicsWizardry my idea was that I would have z-raid1 with two same size drives and primarily point of view was the possibility of using leftover space for increasing TBW (on controller lever of the drive)
I just have these two drives on hand and wanna make raid1 with them and don't wanna to trim and lose space on second drive
@@neins Raidz1 is a single parity solution, and typically used for 3 or more drives. I think you want a mirror here so both drives have the same data, and if one drives dies you don't lose any data. You would get 128GB of space with this configuration.
@@ElectronicsWizardry so on 256gb drive "redundant" space will be unused in my case?
@@neins With a mirror setup there would be 2 128GB mirrored chunks, one on the 128GB SSD, and the other 128GB would be 1/2 of the second SSD. The empty space on the larger SSD would be used for overprovisioning. But with SSDs these days I wouldn't worry too much about having extra overprovisioning space and endurance isn't a big issue with most SSDs and use cases.
I'm embarassed to even ask. I checked out your script, but I'm bit confused as it's .sh but apparently runs on python? Hmm. How do you launch it?
I'm far from the best programmer di there is sgood chance I didn't do that fully right.
But there is a shebang at the top that tells the system to runcthr script on python3. You should be able to download the script and then make it executable with chmod +x diskWrites.sh, and the run the script with ./diskWrites.sh. Hope this helps.
@@ElectronicsWizardry Thanks, I forgot completely to make it executable. I don't understand why, but I get "./diskWrites.sh: line 1: payload:allShortcutsEnabled:false: command not found". I checked the script, as always even though I'm not programmer at all and don't understand why that env line doesn't work. Tried running it in Proxmox out of curiosity
@@ElectronicsWizardry Oh never mind. Me stupid ass downloaded the html format from github instead of raw. Reminder to myself "always check the local file". Man I feel stupid 🤦🏻
It works just fine..
Agree on the brand. Cheap chinese brands just fail constantly. OCZ, Teamgroup, pretty much garbage
your blinking is distracting me so much
Close YOUR eyes and listen....
New sub right here, great work thank you !