Just a note on the Backup section, when you setup the backup job, you forgot to select the Storage on the synology-nas-storage location. Otherwise, great video!
It's literally been this way since I made the video and I never realized 🤦. Thank you very much for pointing that out!!! Saved me questioning why I had no storage space on my local drive LOL - thank you again! Pinned for visibility.
It stood out to me because when I stood up my Proxmox datacenter for the first time 6 months ago, I made the same mistake and filled up the main node local drive lol.
Lol, oh the network loops.... Where were you a year ago when I needed you?! That is the gem of this video! I like what you did with the active backup. Gonna try that out on my cluster.
Also follow up... Great video! Also, what you did wasn't quite niche... More like best practices for all of us Unifi users. Where you really stood out on the rest of the 900 videos on the interwebs about proxmox was the network back bone. That's what's always missing. Let me know if you need help with ceph and clustering if you get to that.
Thank you very much! Oh the frustration that network loops can cause...hope you got it all squared away. Appreciate the kind words and thank you for watching!
I also use a number of VLANs on my Proxmox setup and use the VM ID and tags too. One change to the VM ID I like to use is the addition of the VLAN in the ID. Example is if the VM is on VLAN 101 and has an IP with the last octet of 110 (192.168.101.110) then my VM ID is going to be 101110. This means that I can reuse the the 110 again for another VM in a separate VLAN. Great video!
you mentioned doing LACP which sure its good, but i'm assuming if you have SFP+ ports you likely have some intel 10gb+ networking and can make use of SR-IOV. SR-IOV will be a lot better in terms of general networking overheads than making use of bridges. this is because SR-IOV will bypass the kernel and allow you the ability to place direct portions of a network card directly into your VMs / containers. In doing so you will be able to get hardware level networking for all of them, which includes all the great offload features like TCP offload, all the hashing offloaded to the nic and a few other things im probably forgetting. another tip that i don't see mentioned often is that you can change the networking so that its references by mac and name instead. This is good because if you move the network card to a different slot, then you generally have to get console access and change the name to whatever new name it got (usually changing the numbers around). if you instead identify the device by mac and create your own name, you can migrate it around as you see fit and you don't need to get into the console to modify the networking.
I considered SR-IOV, but came to the conclusion that it just wasn't worth it for me. The performance increase would be nice, but at only 10Gb speeds, keeping things simple seemed better over the ~10% speed increase I'd likely see. Your second point is something I definitely should have done, so thanks for highlighting it!
Excellent presentation. Thank you. One thing I also do on my homelab is to have multiple Proxmox backup servers "nested" on different machines for doing backups. I actually have 3 nested Proxmox servers at the moment on different boxes. In my case, there is no need to have a dedicated hardware for the backup server - just nest it and use the horsepower and storage on less used boxes. I also have a share from a NAS like describe for doing backups as well. They are really helpful for not only doing backups but for restoring things around to different Proxmox environments for testing. The Proxmox Backup Server also does dedup and compression making a great place to keep stuff stored and ready to restore for some quick tests.
I loved and will use most, if not all the tips you mentioned especially the networking tips (bonds, etc.) within Proxmox. At 12:45 you talk about the export and content type. Does it impose any performance or overhead that you know of, to select all the types (Disk, ISO, etc.), as you did? Maybe a video idea for you. When is it OK to select all, and when is it helpful to select one type of content? For example, if you select VZDump content, does the backup run twice as fast, or does it take half the space? Thanks for the video. It gave me lots to think about.
Thanks for the video and sharing! Small issue: On point 2, creating the pool is great, but you'll still need to tell Proxmox what to store and where (Go to Datacenter, storage, select your storage and double click on it, and select in the "Content" what to store)
Thanks for the thoughts on the network loops. Definitely got me thinking. 🤔 Dang, now I need to buy me an aggregation switch for my unifi network, so I can move my 3-node mini micro Proxmox cluster out to the garage, also got to work on getting the garage better insulated - but that is another project...
The aggregation switch (non-pro version for $269) is actually pretty dang great. It's only layer-2, but if you're looking to do basic aggregation, it's hard to beat. Thanks for watching!
As an add to this, instead of making the last octet of the IP the VM number you could use the guest agent and then you have the IP listed natively. Then just let proxmox handle the machine numbering.
That was a really fine video, thanks a lot. I do some of the stuff to advise, but I realized I needed to go a litlle deeper. I do have a Proxmox cluster already, and you mentionned using a second NIC for a cluster, are you gonna do a video on this ? I will be eargerly awaiting for other videos about Proxmox Settings!!!!
Great stuff. I have Proxmox on a Minisforum MS-01 but haven't created VMs yet, so I will eventually probably just reinstall Proxmox and follow this, though I need to put in a rack for some Unifi gear first. I'm a little torn on the MS-01. On paper I think it's an amazing device for this purpose, efficient and powerful CPU, 96 GB RAM, 2x 2.5 Gbps LAN, 2x 10 Gbps SFP+, 2x 40 Gbps Thunderbolt, three NVMe slots, half-width half-height PCIe slot. But it doesn't support ECC. So if I want to build out a cluster I'm torn between trying to get another two of them in the same config, or just using the MS-01 for something else (or a standalone Proxmox) and trying to build out some 2U rack mounted nodes for Proxmox. With a setup like you're doing here with daily backups of all VMs to a NAS anyway, how necessary would you say ECC is? The storage in my MS-01 is aa ZFS mirror for a little redundancy.
So funny enough, I never had a server that DIDN'T have ECC Memory...until now. I couldn't justify the cost of ECC DDR5 memory, so short term...everything has been fine. Long term? Who knows...but hopefully it'll be fine. The backups unfortunately won't really help with an issue like that because if the error is written to disk, it'll be written to disk in the backups as well. Only way to fix it would be to notice it (which you probably won't to be honest) and roll back to a prior retained backup before it happened (if it exists). So yes, backups can help, but you'd have to notice it and roll back to a retained version without the issue.
Great vid. I've come from a VMWare background but wanted to try ProxMox, never really been a fan of Hyper-V. Quite an advanced user in some respects with a professional background in systems and networking. I've started on the firewall side of things within ProxMox and using security groups for common settings but would like to look at the SDN side :-) . That nagging repository window I have tried to fix but broke an install already, lol. Currently Got a 2 hardware node cluster at home with a quorum running on a 3rd debian server (actually running on a Synology, lol). Looking forward to some more content and subscribed 🙂 . Oh ye network loops you don't need to tell me about them, had to deal with a few at work, none created by me, lol.
That's exactly what I'll be doing too...just think that I'll be using a Raspberry Pi for quorum - still need to figure it out though. Thank you for watching!
Thanks for all the videos this year, they've helped me a lot on my Proxmox journey. I've finally got Proxmox Backup Server running as a VM on my Truenas Scale server. It's not super-fast but now I don't have to have another system running. Is your PVE host an MS01? Best wishes for 2025!
Using two bond devices is not really needed, this would be much easier by adding both enp7s0 and bond0 to the bridge after enabling STP. Linux will detect that the 2.5G and 2x10G bond both lead to the same network, and will choose to enable only the bond0 link, disabling the 2.5G link, since it has a lower speed. But as soon as the bond goes down, it will reevaluate this and enable the 2.5G link. Also setting the STP priorities is not needed, the switches and the bridge will settle into an arrangement that has no loops, since that it the entire point of Spanning Tree. And since the bridge on Proxmox is a leaf of that network, it is highly unlikely that the Unify switches will prefer the Proxmox Host over the link directly between them. But in case you pull the cables between both switches STP will ensure that the 2.5G link becomes enabled, and traffic from your PoE switch will use the Proxmox Host as an intermediary to get to the Aggregation switch, giving you time to plug the cable back in without interrupting traffic much. In essence - use bonding with LACP to do link aggregation, but prefer STP for active-backup scenarios.
Thanks for the first tip! I will try it out. In terms of STP priorities, I did indeed have network loops. The only way to resolve them was by turning on STP for the bridge, which then respected the Switches priority. Modifying the Switch priorities without turning on STP for the bridge still resulted in network loops, and turning on STP for the bridge without modifying the priority (did it to test) had network loops as well.
@@WunderTechTutorials hm.. if the Unify switches have STP enabled that should not happen - they should have either cut the 2x10G link or the 2.5G link on their end. The active-passive bond should have not have both its enslaved interfaces up at the same moment. But then again the internet does not recommend bonding bonds, and some sources have reported problems of it not switching over from the primary bond to the secondary, since the primary never truly goes down. I would be really interested to see the pattern that occured when you found it to be a loop - did you run wireshark and see duplicated packets en masse, did it just escalate to a broadcast storm? Ideally you should be able to prove the system working as you intended by pulling cables and observing the traffic, and showing the misbehavings in wireshark. I think that would even make a good video - "proving the network setup live" ;)
I haven't ever heard that bonding a bond isn't recommended. I've seen it done a lot, and never personally had any problems with it, BUT the Switch AND Hypervisor must be configured properly. If they're both not configured properly, you're going to run into network loops. Maybe that's why people have reported problems. I haven't really dug any deeper into it, and will consider a video on it, but I'm not sure anyone other than you and I would be interested in it LOL!
it can be done. I did it once and hand editing 3 files was do-able but wasn't straightforward. Also you lose the manual edits when you do an update. So was it worth it? not for me.
I don't unfortunately, but Proxmox is a hypervisor that allows you to run virtual machines and Linux Containers. Basically, you run multiple operating systems virtually on your hardware rather than having an individual operating system installed on it.
@@WunderTechTutorialsJust thought I'd let you know that even though I'm an IT professional (academia) and have a NAS at home, I also had no idea what Proxmox was about and how I might benefit from it. I guess there are other introductory videos about Proxmox available but I was wondering why you thought that there was no place for this on your channel. I don't think I'll develop the need for virtualisation, so don't worry about me; just wondering about your audience in general.
@@coolcat23 It's something I should have considered. I've done Proxmox videos in the past so speaking honestly, it's not even something that crossed my mind, but in hindsight, a 1-2 minute explanation would have been worth it. Always looking to improve so I appreciate the feedback!
Just a note on the Backup section, when you setup the backup job, you forgot to select the Storage on the synology-nas-storage location. Otherwise, great video!
It's literally been this way since I made the video and I never realized 🤦. Thank you very much for pointing that out!!! Saved me questioning why I had no storage space on my local drive LOL - thank you again! Pinned for visibility.
It stood out to me because when I stood up my Proxmox datacenter for the first time 6 months ago, I made the same mistake and filled up the main node local drive lol.
Lol, oh the network loops.... Where were you a year ago when I needed you?! That is the gem of this video! I like what you did with the active backup. Gonna try that out on my cluster.
Also follow up... Great video! Also, what you did wasn't quite niche... More like best practices for all of us Unifi users. Where you really stood out on the rest of the 900 videos on the interwebs about proxmox was the network back bone. That's what's always missing. Let me know if you need help with ceph and clustering if you get to that.
Thank you very much! Oh the frustration that network loops can cause...hope you got it all squared away. Appreciate the kind words and thank you for watching!
I also use a number of VLANs on my Proxmox setup and use the VM ID and tags too. One change to the VM ID I like to use is the addition of the VLAN in the ID. Example is if the VM is on VLAN 101 and has an IP with the last octet of 110 (192.168.101.110) then my VM ID is going to be 101110. This means that I can reuse the the 110 again for another VM in a separate VLAN.
Great video!
Love it! Thank you for watching!
What an excellent video...wow!!! the networking part with the bonds and failover...pure genius!!!
Thanks, Avi! Appreciate you watching!
you mentioned doing LACP which sure its good, but i'm assuming if you have SFP+ ports you likely have some intel 10gb+ networking and can make use of SR-IOV. SR-IOV will be a lot better in terms of general networking overheads than making use of bridges. this is because SR-IOV will bypass the kernel and allow you the ability to place direct portions of a network card directly into your VMs / containers. In doing so you will be able to get hardware level networking for all of them, which includes all the great offload features like TCP offload, all the hashing offloaded to the nic and a few other things im probably forgetting.
another tip that i don't see mentioned often is that you can change the networking so that its references by mac and name instead. This is good because if you move the network card to a different slot, then you generally have to get console access and change the name to whatever new name it got (usually changing the numbers around). if you instead identify the device by mac and create your own name, you can migrate it around as you see fit and you don't need to get into the console to modify the networking.
I considered SR-IOV, but came to the conclusion that it just wasn't worth it for me. The performance increase would be nice, but at only 10Gb speeds, keeping things simple seemed better over the ~10% speed increase I'd likely see. Your second point is something I definitely should have done, so thanks for highlighting it!
Excellent presentation. Thank you. One thing I also do on my homelab is to have multiple Proxmox backup servers "nested" on different machines for doing backups. I actually have 3 nested Proxmox servers at the moment on different boxes. In my case, there is no need to have a dedicated hardware for the backup server - just nest it and use the horsepower and storage on less used boxes. I also have a share from a NAS like describe for doing backups as well. They are really helpful for not only doing backups but for restoring things around to different Proxmox environments for testing. The Proxmox Backup Server also does dedup and compression making a great place to keep stuff stored and ready to restore for some quick tests.
I have my proxmox backup server running as a vm on my NAS. 👍🏼
You, CAN put ceph on a vlan, it doesn’t NEED a dedicated port. It doesn’t even use that much data in my testing. We used it at work for years
It's been a long time since I've watched a ProxMox video and actually learned something new...
Please show some more advanced stuff!!👍🏼
Glad to hear it helped, thanks for watching!
I loved and will use most, if not all the tips you mentioned especially the networking tips (bonds, etc.) within Proxmox. At 12:45 you talk about the export and content type. Does it impose any performance or overhead that you know of, to select all the types (Disk, ISO, etc.), as you did? Maybe a video idea for you. When is it OK to select all, and when is it helpful to select one type of content? For example, if you select VZDump content, does the backup run twice as fast, or does it take half the space? Thanks for the video. It gave me lots to think about.
Thanks for the video and sharing!
Small issue: On point 2, creating the pool is great, but you'll still need to tell Proxmox what to store and where (Go to Datacenter, storage, select your storage and double click on it, and select in the "Content" what to store)
It should set VM/Container by default, but good point regardless and something to check! Thanks for sharing and for watching!
Thanks for the thoughts on the network loops. Definitely got me thinking. 🤔 Dang, now I need to buy me an aggregation switch for my unifi network, so I can move my 3-node mini micro Proxmox cluster out to the garage, also got to work on getting the garage better insulated - but that is another project...
The aggregation switch (non-pro version for $269) is actually pretty dang great. It's only layer-2, but if you're looking to do basic aggregation, it's hard to beat. Thanks for watching!
1:11 excellent tip and way to share this
Love the Proxmox content! Thanks.
As an add to this, instead of making the last octet of the IP the VM number you could use the guest agent and then you have the IP listed natively. Then just let proxmox handle the machine numbering.
I need to check that out, thanks for sharing!
ProxMox has made me excited to self host again.
That was a really fine video, thanks a lot.
I do some of the stuff to advise, but I realized I needed to go a litlle deeper.
I do have a Proxmox cluster already, and you mentionned using a second NIC for a cluster, are you gonna do a video on this ? I will be eargerly awaiting for other videos about Proxmox Settings!!!!
Thanks! Yes, video is coming soon on that!
The helper scripts are great for beginners 😊
Great stuff. I have Proxmox on a Minisforum MS-01 but haven't created VMs yet, so I will eventually probably just reinstall Proxmox and follow this, though I need to put in a rack for some Unifi gear first.
I'm a little torn on the MS-01. On paper I think it's an amazing device for this purpose, efficient and powerful CPU, 96 GB RAM, 2x 2.5 Gbps LAN, 2x 10 Gbps SFP+, 2x 40 Gbps Thunderbolt, three NVMe slots, half-width half-height PCIe slot. But it doesn't support ECC. So if I want to build out a cluster I'm torn between trying to get another two of them in the same config, or just using the MS-01 for something else (or a standalone Proxmox) and trying to build out some 2U rack mounted nodes for Proxmox.
With a setup like you're doing here with daily backups of all VMs to a NAS anyway, how necessary would you say ECC is? The storage in my MS-01 is aa ZFS mirror for a little redundancy.
So funny enough, I never had a server that DIDN'T have ECC Memory...until now. I couldn't justify the cost of ECC DDR5 memory, so short term...everything has been fine. Long term? Who knows...but hopefully it'll be fine. The backups unfortunately won't really help with an issue like that because if the error is written to disk, it'll be written to disk in the backups as well. Only way to fix it would be to notice it (which you probably won't to be honest) and roll back to a prior retained backup before it happened (if it exists). So yes, backups can help, but you'd have to notice it and roll back to a retained version without the issue.
Great vid. I've come from a VMWare background but wanted to try ProxMox, never really been a fan of Hyper-V. Quite an advanced user in some respects with a professional background in systems and networking. I've started on the firewall side of things within ProxMox and using security groups for common settings but would like to look at the SDN side :-) . That nagging repository window I have tried to fix but broke an install already, lol. Currently Got a 2 hardware node cluster at home with a quorum running on a 3rd debian server (actually running on a Synology, lol). Looking forward to some more content and subscribed 🙂 . Oh ye network loops you don't need to tell me about them, had to deal with a few at work, none created by me, lol.
That's exactly what I'll be doing too...just think that I'll be using a Raspberry Pi for quorum - still need to figure it out though. Thank you for watching!
Excellent video thanks
Did you compare TrueNAS Baremetal vs. Proxmox Baremetal?
I did not, but I'll have a video on TrueNAS Scale soon (hopefully, within the next month or so).
I just host a small VM on my Synology that runs PBS....then I get dedup and compression which saves an order of magnitude of space.
Thanks for all the videos this year, they've helped me a lot on my Proxmox journey. I've finally got Proxmox Backup Server running as a VM on my Truenas Scale server. It's not super-fast but now I don't have to have another system running.
Is your PVE host an MS01? Best wishes for 2025!
Glad to hear they've helped - thank you for watching! No, this is in follow up to the home server I built (my last video was on that).
Using two bond devices is not really needed, this would be much easier by adding both enp7s0 and bond0 to the bridge after enabling STP. Linux will detect that the 2.5G and 2x10G bond both lead to the same network, and will choose to enable only the bond0 link, disabling the 2.5G link, since it has a lower speed. But as soon as the bond goes down, it will reevaluate this and enable the 2.5G link.
Also setting the STP priorities is not needed, the switches and the bridge will settle into an arrangement that has no loops, since that it the entire point of Spanning Tree. And since the bridge on Proxmox is a leaf of that network, it is highly unlikely that the Unify switches will prefer the Proxmox Host over the link directly between them. But in case you pull the cables between both switches STP will ensure that the 2.5G link becomes enabled, and traffic from your PoE switch will use the Proxmox Host as an intermediary to get to the Aggregation switch, giving you time to plug the cable back in without interrupting traffic much.
In essence - use bonding with LACP to do link aggregation, but prefer STP for active-backup scenarios.
Thanks for the first tip! I will try it out. In terms of STP priorities, I did indeed have network loops. The only way to resolve them was by turning on STP for the bridge, which then respected the Switches priority. Modifying the Switch priorities without turning on STP for the bridge still resulted in network loops, and turning on STP for the bridge without modifying the priority (did it to test) had network loops as well.
@@WunderTechTutorials hm.. if the Unify switches have STP enabled that should not happen - they should have either cut the 2x10G link or the 2.5G link on their end. The active-passive bond should have not have both its enslaved interfaces up at the same moment. But then again the internet does not recommend bonding bonds, and some sources have reported problems of it not switching over from the primary bond to the secondary, since the primary never truly goes down. I would be really interested to see the pattern that occured when you found it to be a loop - did you run wireshark and see duplicated packets en masse, did it just escalate to a broadcast storm?
Ideally you should be able to prove the system working as you intended by pulling cables and observing the traffic, and showing the misbehavings in wireshark. I think that would even make a good video - "proving the network setup live" ;)
I haven't ever heard that bonding a bond isn't recommended. I've seen it done a lot, and never personally had any problems with it, BUT the Switch AND Hypervisor must be configured properly. If they're both not configured properly, you're going to run into network loops. Maybe that's why people have reported problems.
I haven't really dug any deeper into it, and will consider a video on it, but I'm not sure anyone other than you and I would be interested in it LOL!
Additional comment: it will be interesting to see how to setup the bonding stuff on the free OpenWRT router.
Yes good presentation, i would like to have CPU temp on dashboard. tell me if you can, and again good job
Thanks! I believe you can, but it has to be done through the CLI (as far as I know) and it's not the most straightforward process.
it can be done. I did it once and hand editing 3 files was do-able but wasn't straightforward. Also you lose the manual edits when you do an update.
So was it worth it? not for me.
Oh wow, blindly running that script was fantastic. Highly recommend.
Great video, learned a lot, jealous of your hardware, 100% subbed now.
I number my machines by the last bit of the IP address so for example 10.10.10.10 the machine number is 1010, 10.10.20.10 would be 2010 and so on
Do you have a, "What is Proxmox and why would you use it?" video?
I don't unfortunately, but Proxmox is a hypervisor that allows you to run virtual machines and Linux Containers. Basically, you run multiple operating systems virtually on your hardware rather than having an individual operating system installed on it.
@WunderTechTutorials Than you. I don't have a known need today. Perhaps that will change.
@@WunderTechTutorialsJust thought I'd let you know that even though I'm an IT professional (academia) and have a NAS at home, I also had no idea what Proxmox was about and how I might benefit from it. I guess there are other introductory videos about Proxmox available but I was wondering why you thought that there was no place for this on your channel. I don't think I'll develop the need for virtualisation, so don't worry about me; just wondering about your audience in general.
@@coolcat23 It's something I should have considered. I've done Proxmox videos in the past so speaking honestly, it's not even something that crossed my mind, but in hindsight, a 1-2 minute explanation would have been worth it. Always looking to improve so I appreciate the feedback!