Congratulations on the amount of detail in the comparison. I'm virtualizing a small server environment, all Windows and I understood after this video that XCP-NG is the ideal option for this project. Thanks!
Our previous datacenter admin preferred Proxmox, which got me into it. He retired and we went with another admin to manage our colo'd servers who is all about xcp-ng but its been a bumpy road during the transition. Personally I'm still running proxmox on my homelab cluster with ceph.
I really like proxmox lxc containers. I don't even have any vms now. Transfered everything to lxc containers. Super small, quick to spin up, and usually only 200mb of memory. I can probably run 50 of them if i wanted to. Whereas a few vms and my system starts to have problems.
Great video man. I think overall I'll give XCP-NG a go. What I want is stability, easy LAG/LACP and VLAN network configuration, as well as scalability with local storage live migrations. Thanks for the guidance 👍
Homelab? Someone should define it more clearly. Had multiple server and also multi-hundred TB of storage at HOME, yet they called it homelab???? They should called it HOME DATA CENTER.
ProxMox is my choice of VM support. I've tried Microsoft Hyper-V (sad..I know..lol) and VMWare. The reason I like Proxmox is because it's built around free and open source. Plus the fact you can tinker around inside Linux for customizations that you can't do with VMWare. Hell, you can install ProxMox on a standard Debian machine without using their ISO. It's that flexible.
Thanks for sharing! Flexibility is through the roof with ProxMox, no matter if it's production or not (like with my example about the media center machine :D ).
Thanks for sharing. I tried using XCP-ng for the business I work for. We’re on Xen 6.5 and I needed a way forward. Sadly the iSCSI implementation in XCP was so unstable for us that we went another route, to Hyper-V actually. Never considered Proxmox or any Linux type virtualisation engine as I neede to get up and running quickly. Now, a year later I am revising the topic once more. I’m looking for a way forward virtualising for my own business. Although, I run my heaviest applications on bare metal as I have not encountered any virtualisation environment that is quick enough. Interesting video. Thank you again.
One main thing that alot of people dont talk about and as a IT guy myself its important. Is that Xcp-ng is Red Hat Linux RPM based and Proxmox is DEB based. Both have certain hardware compatibility strengths and weaknesses to keep in mind when thinking about future upgrading.
Really? Not more comments on this vid? Huh, this is a good video. More people watching this video. I have not used proxmox. XCP-NG has been working well for both windows and Linux / Unix VMs. Docker Containers I installed on a Debian VM. Proxmox you can just install on hypervisor the containers, not sure how that functions in practice but maybe less tax on CPU that way? I don't have performance issue with my dockers on Debian VM, on XCP-NG. :)
You lost me when you said I was clustering across a WAN link with Proxmox. That a no no with any product - just because you can, doesn't mean you should.
Thanks for your comment. I had to try it, especially after ProxMox team switched corosync to unicast mode. And I made that comment specifically for the people who are thinking of clustering over the WAN :)
Ярослав, использование corosync вне единого домена коллизий официально не поддерживается и не рекомендуется - так что все-таки это не минус а скорее фича.
@@dkorzhevin я это понимаю) но людям нужно было рассказать, меня уже не раз об этом спрашивали) ну и плюс хотел потестить это дело после внедрения юникаста в коросинк командой проксмокса)
Thanks, one of the reasons to pick XCP-NG (Googled couple of manuals, XCP-NG seemed easier to do). As well as Nested Virtualization. I wonder if there are performance tests on Nested Virtualization and passthrough stuff.
Interesting video, thanks for the comparison. I wonder if you've done any performance testing between the two platforms, and if you've seen any difference between them? In my experience XCP-ng was significantly slower than Proxmox. For example, in CPU video transcoding, database performance, network transfer speeds on my 10 gig homelab network, local SSD storage read/write performance, and general usage in the console (e.g. opening applications, maximising/minimising windows, loading browser pages). I don't know if this is a genuine performance difference in the underlying hypervisor/architecture, or if it was something unique to my server (which doesn't have any exotic or esoteric hardware, just a Dell R730 with common hardware) and home network. I'd be interested to hear your thoughts or experience on the topic. Thanks!
With ProxMox you can allow KVM to pass some CPU supported flags to the VM, like AES instructions and such. So for graphical environments it should be faster, as well as for transcoding types of workloads etc. I am not at all surprised that ProxMox is way faster in your case :) But that's a very specific use case, and I couldn't possibly think of them all.
@@GatewayITTutorials and what about the hardware compatibility? Both works better on intel? Do I have any chance to think about using a Ryzen? What about the gpu pass though? Decoding, encyption or most of regular tasks?
Proxmox is a little ahead in terms of new hardware support, because system uses newer kernel. CPU-heavy workloads are better handled by Proxmox too, due to a performance difference between Xen and KVM.
ProxMox is much better at running Linux VMs, XCP-Ng is better at running Windows or BSD. But it will also depend on your environment, for example there is better driver support for network cards, that are not Intel, on XCP-Ng. If you are after a raw performance metrics, the general rule of thumb is: more Linux guests - ProxMox (KVM), more Windows or BSD guests - XCP-Ng (XEN). Hope that makes sense.
Since XCP-NG requires XenOrchestra for all of the features you've basically loved, how about including Proxmox Backup Server? That'll fix your snapshot and backup woes. Also, you can easily convert a VMDK or any other format to raw format in Proxmox by using qemu-img convert -f and convert the disk to QCOW2. Lack of OVA support does suck and networking is absolutely horrid in Proxmox, you are right. I'm not sure why adding/removing nodes to and from the cluster repeatedly would be a normal instance, btw. The fact I have to install Xen Orchestra and compile my own install from source in order to use a web UI for XCP-Ng was a huge hell naw.
Love your comment, very detailed :) 1) There are no automatic snapshots on ProxMox, with Backup Server or without it. I am using ZFS replication or GlusterFS async writes for the DR, so I don't care about the Backup Server at all, I hope that makes sense :) 2) Yes, it sucks to compile XO OSE every time there is an update to it, but I think that info is in the video. 3) Networking is not horrible on ProxMox, especially if you install ifupdown2. With that package, it's not required to restart the node every time network config changes. But yes, there is a much steeper learning curve if you'd want to do some advanced networking on ProxMox. 4) Re clustering: you are underestimating fast growing environments, or use cases where you need to move machines between datacenters. Another use case is project tiny-mini-micro (covered by STH), where you have many small nodes, which you need to reinstall sometimes, add new ones or remove a lot of out-of-date equipment. 5) I think I said few things about VMDK2RAW or VMDK2QCOW conversion, and in my opinion, there should be an option included in the WebUI for this purpose. I love CLI, it's very efficient for bigger tasks, but sometimes it is easier to just press a button to import the virtual image. These are just my humble opinions after all, don't take them too seriously :)
@@GatewayITTutorials for automatic snapshotting, just install debian package "zfs-auto-snapshot" on the proxmox host. (and additionally you could install cockpit and the "cockpit-zfs-manager/zfs /usr/share/cockpit" to get a beautiful snapshotting managment solution directly on your host) I can not see the developers ever implementing a "native" snapshotting-function in proxmox, because this type of snapshot function is just stupid to implement on the hypervisor level. Because if you snapshot a, say 10TB fileserver, it takes hours/days to delete a snapshot and going back to a single vmdk/vhd file. Even vmware and netapp had a (Storage)native snapshotting implementation like 5 years ago.
Heeeeeey, quick question: Why would you keep a 10TB file server as a VM o_0 Both ESX and NetApp are targeted at enterprise: they are big enough to dictate the market, not other way around, so I don't look at them as a good example of anything. ZFS snapshots and replication is the best thing I've seen that you can do for your backups: - Snapshots will be identical across all systems, so disaster recovery is a breeze - ZFS can fit nearly unlimited number of snapshots - Replication happens over SSH, no need to worry about any other nonsense software or unencrypted protocols - No need to worry about a shared storage: just use local storage with replication Obviously this is not a solution for everyone, but it's as simple and reliable as it gets
I'd say if it is a small enough place where you are expecting to just have 1 hypervisor, use proxmox. If you are planning on expanding and may potentially later see yourself using 2 or more hypervisors and would have a separate storage server and may be replicating VM's across each other and anything larger, then XCP-ng is your choice. I personally use a lot of proxmox for businesses I manage.
CEPH is cool, but it is designed for bigger deployments, with at least 3 nodes (but ideally 5, to give you the flexibility to loose 2 in case of disaster).
It's bhyve, not bee hive) but that's a funny way of spelling it) you made my day bro) bhyve is a FreeBSD hypervisor, kinda new in this space, but very ambitious. I am planning to do a video on it in the next couple of weeks.
I am not getting where you were going with your comment (I've got no means to insult you). It was either a pun or you are taking this a bit too seriously :)
That's absolutely alright, I can't control how other people perceive me (nor do I want to control that), and I always say what I think is right (and what's "right" is always subjective to the person saying it). Your first comment on the other hand, came across (at least to me) as some sort of mockery, but again that's subjective :)
I use a variety of hardware: Few laptops, few Intel NUCs, 2 iMacs (2011 models) and 3x HP G8 1U servers connected to TrueNAS :) Then, depending on the software, I'll deploy it to the appropriate cluster.
I wished that there was a virtualisation system/platform that combined the best of both worlds. I am currently using Oracle VirtualBox on a Windows Host and I think Chris Titus did a video showing how much faster a VM can be using qemu, except that qemu only runs in Linux and it isn't as easy nor straightforward (i.e. "dumb" point-and-click to deploy). I am using an Intel Core i7-6700K right now, and I've got somewhere between 8-10 VMs running off that system right now (plus the host). So, it would be nice if I can run those VMs faster vs. the speed that they're currently running at. I might fire up two of my older workstations, and one on, throw XCP-ng and Xen Orchestra on it and on the other, throw Promox VE on it and see which one works better in the end.
With Proxmox, you set which network you want for corosync. The reasoning why proxmox clustering stabbed you in the back was user error, thus, you cannot blame the technology as bad.
I know there is a dedicated link option, and/or redundant link/links for corosync, but that's not the point) The point is that I had to explain: clustering over WAN is a no go. I've seen people do it too many times in my consulting practice, hope you understand.
Thanks for the video! At th-cam.com/video/Aqhkd4-S66c/w-d-xo.html you said that proxmox is missing encryption at rest - but you can have your zpool under cryptsetup / luks. You can also use native zfs encryption (but I guess that's fairly recent addition).
As for the ProxMox: Crypt/luks is very inflexible when it comes to dynamic password change, or when you want to use a key instead of password. ZFS dataset encryption (on the root pool) only works under UEFI systems, if your system does not support UEFI, thought luck, then create a separate pool in order to use ZFS dataset encryption. If you will still go ahead and encrypt one of the datasets on the root pool (under BIOS system) then your system won't boot anymore (trust me, I tried it :D). That's a Grub2 limitation, which is unlikely to be fixed any time soon. But with Bhyve and ZFS I can load the password/key at any time via a short SSH command, or number of other ways. It always comes down to usability :) We still use ProxMox on a number of our systems, due to some feature limitations on Bhyve. You gotta use a right tool for the job, and both ProxMox and Bhyve are just tools.
Thanks for the comparison. Unless I missed it (me multi-tasking), maybe you can compare ease of updating XCP-ng vs Proxmox. I like the ease to copy/clone VM from one XCP server to another. I'm currently playing around with Proxmox, but figuring out how to import VM from XCP-ng was trial and error. I have been using an older Xen/XCP-ng (and Center) for my Win development VMs, but have issues with running two Win VMs at same time on a i7-4790 via RDP (yes an old processor!). Normally I only have one running, unless I need to troubleshoot an issue with my other software on the other VM. Between slowing down and then one dropping RDP connection, I decided to try out Proxmox on an i3, which would run both (slow) but didn't loose RDP connections. Now experimenting with Proxmox on a DL380 g8 dual-cpu (no SMART info passes thru from raid card) and it is plenty fast with no problems running multiple Win VMs. Will install latest XCP-ng on DL380 next, as I've been using an older version to see if the issue is gone; if so, I most likely stick with XCP-ng.
ProxMox is much easier to update and jump between major versions, all of it can be done with a simple APT command, where with XCP-Ng you need fresh ISOs every time there is a major update. ProxMox will always be a bit more snappier, or faster if one can say that, because it's based on KVM, which is one of the fastest hypervisors out there.
Proxmox has automatic backups and also automatic snapshots th-cam.com/video/1CT6uh-fasg/w-d-xo.html th-cam.com/video/GoZaMgEgrHw/w-d-xo.html The import/export hassle of PM stinks, but the ZFS support looks like it trumps that. Wait, it's just the root xcp-ng that can't be on ZFS, but still may be lacking some ZFS gui abilities? th-cam.com/video/XLQp_jI5vNs/w-d-xo.html
For homelab, my pick would be XCP-NG because of these: - OVA import/export (for people that migrate from other homelabs) so it's more openness. VMDK/OVA are still out there to download if you need something like Hacintosh or exchange with other homelabbers. - Nested Virtualization seems better working (Proxmox still had issue topics as of 2019 requiring some manual interventions, while I need Windows VM with Docker in it, so it requires Nested-V). I believe it should be just a checkbox. - Passthrough seems easier (current manuals as of 2022), as well as commitment to get it done for various vGPUs working and so on. Those are main hiccups. I'm migrating from ESXi 6.7 because of introduced core limitation (5.5 had no core limitation, but had Nested-V problem so I had to upgrade). ESXi 7.0 eliminates native Realtek support, free cores aren't growing. Time for change. And during change, I'll continue to have both hypervisors. Proxmox feels a bit like jail. May be comfortable one, but who knows if you'll get out of there if you need to. XCP-NG seems like innocent "I can change you anytime" which will lead to... not changing probably. I may comment later if I found any issues with XCP-NG on my main hiccup points.
First encounter with XCP-NG - Import works through XO. OVF/OVA export is basically non-existent (UPD, if you build XOA from May 2022 OVA exists). Xen Orchestra (at least the default one) doesn't have the ability - XVA export only. XCP-NG Center doesn't properly import or export at any format, but UI seems to support OVA/OVF if it was working in the first place. (XCP-NG 8.2.1 + Center 20.04.01.33)
Nested-V is also became a problem. With Nested-V enabled, previously built VMWare VMDK doesn't boot (Windows BSODs). It runs under ESXi 6.7 and Windows 10 / Hyper-V, but no success on XCP-NG.
I tried everything in XOA or XCP-NG Center can do to modify VM settings in different ways: no luck in getting Nested-V VM even started. It doesn't even boot (in a way that different Hard Disk driver or different BIOS would help to load). However, after installing Proxmox I was able to run Docker in Windows / Nested-V in about 4 hours. I guess that's it. My UI Import/Export sympaties are with XCP-NG, but unfortunately, Proxmox worked so I'll go with that.
@@clockware I had no problems installing headless Ubuntu as a VM and running Docker. Years ago I turned on Nested-V for running jails in TrueNAS core and it basically worked. I abandoned the jails for Docker and Portainer though, a much more widely used method of containerization, more support, works better. In your other comment you mention importing VMs: Build the latest Xen Orchestra from sources and try the import menu item. I avoid XCP-ng Center because its actually kind of buggy and dangerous. Xen Orchestra is safe.
... "you can not import a VM from VirtualBox ..." in Proxmox. I would consider this just wrong. Install Proxmox on a cheap Optane with NVME to USB adapter. Done.
But there is no import function in ProxMox, I don't know what you are trying to say here... The only option you have is to manually unpack the OVA archive, convert virtual drive to RAW or qcow formats, then rename that file to something already present in the system and move it to the right place by hand, which is unnecessary complicated. And then the sentence about the Optane and a USB adapter just doesn't make any sense. Not to be rude, but try to re-read what you've written there, you might have missed something.
@@GatewayITTutorials Well, first of all you can make yourself a simple script to convert and import other image formats of virtual machines. Thats super easy and a sys-admin like you shouldnt be bothered by it. I wouldnt take that as disadvantage. Quite the opposite. At the same time you re-configure the image to your needs. Xnp-ng seems to not be that flexible. You were saying Proxmox isn’t suitable to be installed on a USB drive in terms of wearing out USB drives quickly due to the extensive logging. You are quite wrong. There are a lots of people, even dc operators, who are running cheap 16gb Optane drives as os drive with Proxmox installed and connect them to NVMe to USB connectors, which then are connected to internal usb connectors. this is a cheap all in all 40 US$ solution and very reliable. The setup is fast enough for the OS, the Optane drives are bootable, they have a 300+TB endurance which is high enough for heavy logging. So, set yourself aside and research a little bit before you produce a vid with such ‘questionable’ statements and then give very aggressive replies.
This video is targeted at people who don't know how to make scripts, and just want to experiment with the platform. My comment wasn't angry, I was genuinely trying to understand what you have to say, and now I do, thank you for the clarification. The point about USB drives: Did you hear me say - you can't use USB to sata adapter and install ProxMox onto the SSD? I assume the answer is no. What I was talking about is a cheap USB thumb drive, short and compact, so in case someone accidentally hits it, it wouldn't brake the board or itself into pieces. I used external drives to install ProxMox myself, but due to a nature of the system (many random reads and writes to the FS) you are somewhat limited with your drive choices. I love ProxMox, it's a great product, but many homelab users don't bother to write scripts for the migration, don't bother to read the docs, and so on. That's why I had to bring up the points I brought up in the video. Most of these will never be applicable to enterprise or DC, but nonetheless they have their use cases.
Bro, I know my audience, they are brilliant people, and I am not trying to put anyone down, quite the opposite. Not everyone who will watch this video is professional in IT industry. Not everyone can or should write scripts, or think about automation among other things. That's why they are comparing, listening to different opinions, learning new things. I really do understand what you were trying to say, but hear me out too: sometimes people can have different opinions, and that's completely normal. For an average person, it's imperative to have that "Import" function in the GUI, I know it, I've been there just few years ago. If you disagree, that's fine, it just means we have different views.
Congratulations on the amount of detail in the comparison. I'm virtualizing a small server environment, all Windows and I understood after this video that XCP-NG is the ideal option for this project. Thanks!
This was really thorough addressing the exact question I'm working through. Thanks
Thank you for also providing a blog I can read! Very much appreciated.
Our previous datacenter admin preferred Proxmox, which got me into it. He retired and we went with another admin to manage our colo'd servers who is all about xcp-ng but its been a bumpy road during the transition. Personally I'm still running proxmox on my homelab cluster with ceph.
Excellent video! Thought I am left with more doubts than before. LOL... Great job!
I really like proxmox lxc containers. I don't even have any vms now. Transfered everything to lxc containers. Super small, quick to spin up, and usually only 200mb of memory. I can probably run 50 of them if i wanted to. Whereas a few vms and my system starts to have problems.
Great video man. I think overall I'll give XCP-NG a go. What I want is stability, easy LAG/LACP and VLAN network configuration, as well as scalability with local storage live migrations. Thanks for the guidance 👍
Proxmox is rock-solid and has many customizable functions (mostly thru CLI) that make it irreplaceable.
Homelab?
Someone should define it more clearly.
Had multiple server and also multi-hundred TB of storage at HOME, yet they called it homelab????
They should called it HOME DATA CENTER.
Superb content! thank you for posting this!
Great video ! I've installed XCP-NG some days ago and it works really well (I use only linux VM). Thank you !
ProxMox is my choice of VM support. I've tried Microsoft Hyper-V (sad..I know..lol) and VMWare. The reason I like Proxmox is because it's built around free and open source. Plus the fact you can tinker around inside Linux for customizations that you can't do with VMWare. Hell, you can install ProxMox on a standard Debian machine without using their ISO. It's that flexible.
Thanks for sharing! Flexibility is through the roof with ProxMox, no matter if it's production or not (like with my example about the media center machine :D ).
Thanks for sharing. I tried using XCP-ng for the business I work for. We’re on Xen 6.5 and I needed a way forward. Sadly the iSCSI implementation in XCP was so unstable for us that we went another route, to Hyper-V actually. Never considered Proxmox or any Linux type virtualisation engine as I neede to get up and running quickly. Now, a year later I am revising the topic once more. I’m looking for a way forward virtualising for my own business. Although, I run my heaviest applications on bare metal as I have not encountered any virtualisation environment that is quick enough. Interesting video. Thank you again.
iSCSI in TrueNAS core allegedly only works when given absurd amounts of RAM. I gave up on it and use NFS in TrueNAS Core. No problems since then.
Thank you for the comparison. Very useful
One main thing that alot of people dont talk about and as a IT guy myself its important. Is that Xcp-ng is Red Hat Linux RPM based and Proxmox is DEB based. Both have certain hardware compatibility strengths and weaknesses to keep in mind when thinking about future upgrading.
Great video! I really learned a lot.
I definitely want a video on bhyve. I haven't heard of it till now.
Really? Not more comments on this vid? Huh, this is a good video. More people watching this video. I have not used proxmox. XCP-NG has been working well for both windows and Linux / Unix VMs. Docker Containers I installed on a Debian VM. Proxmox you can just install on hypervisor the containers, not sure how that functions in practice but maybe less tax on CPU that way? I don't have performance issue with my dockers on Debian VM, on XCP-NG. :)
Hell yeah. I have a truenas and I always had issues with VM...
You lost me when you said I was clustering across a WAN link with Proxmox. That a no no with any product - just because you can, doesn't mean you should.
Thanks for your comment.
I had to try it, especially after ProxMox team switched corosync to unicast mode.
And I made that comment specifically for the people who are thinking of clustering over the WAN :)
Ярослав, использование corosync вне единого домена коллизий официально не поддерживается и не рекомендуется - так что все-таки это не минус а скорее фича.
@@dkorzhevin я это понимаю) но людям нужно было рассказать, меня уже не раз об этом спрашивали) ну и плюс хотел потестить это дело после внедрения юникаста в коросинк командой проксмокса)
Great comparison! I also did not find that xcp-ng can PCI Passthrough with consumer GPU.
Yes, that's true, only pro GPUs are supported.
I use Proxmox and I was able to passthrough the NUC iGPU. My use case is Emby transcoding.
XCP-ng can passthrough GPU. I had successs with AMD GPU and Intel i7 6700 iGPU. Very straight forward.
Thanks, one of the reasons to pick XCP-NG (Googled couple of manuals, XCP-NG seemed easier to do). As well as Nested Virtualization. I wonder if there are performance tests on Nested Virtualization and passthrough stuff.
Interesting video, thanks for the comparison. I wonder if you've done any performance testing between the two platforms, and if you've seen any difference between them? In my experience XCP-ng was significantly slower than Proxmox. For example, in CPU video transcoding, database performance, network transfer speeds on my 10 gig homelab network, local SSD storage read/write performance, and general usage in the console (e.g. opening applications, maximising/minimising windows, loading browser pages).
I don't know if this is a genuine performance difference in the underlying hypervisor/architecture, or if it was something unique to my server (which doesn't have any exotic or esoteric hardware, just a Dell R730 with common hardware) and home network. I'd be interested to hear your thoughts or experience on the topic. Thanks!
With ProxMox you can allow KVM to pass some CPU supported flags to the VM, like AES instructions and such. So for graphical environments it should be faster, as well as for transcoding types of workloads etc.
I am not at all surprised that ProxMox is way faster in your case :)
But that's a very specific use case, and I couldn't possibly think of them all.
@@GatewayITTutorials That makes sense, thanks for the explanation.
@@GatewayITTutorials and what about the hardware compatibility? Both works better on intel? Do I have any chance to think about using a Ryzen? What about the gpu pass though? Decoding, encyption or most of regular tasks?
Proxmox is a little ahead in terms of new hardware support, because system uses newer kernel.
CPU-heavy workloads are better handled by Proxmox too, due to a performance difference between Xen and KVM.
I know I'm a little late but would you mind including time stamps for your videos?
Thank you for the suggestion, I'll start adding them shortly.
@@GatewayITTutorials thanks so much
What do you think about performance of vms, wich is better on benchmarks?
ProxMox is much better at running Linux VMs, XCP-Ng is better at running Windows or BSD. But it will also depend on your environment, for example there is better driver support for network cards, that are not Intel, on XCP-Ng.
If you are after a raw performance metrics, the general rule of thumb is: more Linux guests - ProxMox (KVM), more Windows or BSD guests - XCP-Ng (XEN).
Hope that makes sense.
Since XCP-NG requires XenOrchestra for all of the features you've basically loved, how about including Proxmox Backup Server? That'll fix your snapshot and backup woes. Also, you can easily convert a VMDK or any other format to raw format in Proxmox by using qemu-img convert -f and convert the disk to QCOW2. Lack of OVA support does suck and networking is absolutely horrid in Proxmox, you are right. I'm not sure why adding/removing nodes to and from the cluster repeatedly would be a normal instance, btw. The fact I have to install Xen Orchestra and compile my own install from source in order to use a web UI for XCP-Ng was a huge hell naw.
Love your comment, very detailed :)
1) There are no automatic snapshots on ProxMox, with Backup Server or without it. I am using ZFS replication or GlusterFS async writes for the DR, so I don't care about the Backup Server at all, I hope that makes sense :)
2) Yes, it sucks to compile XO OSE every time there is an update to it, but I think that info is in the video.
3) Networking is not horrible on ProxMox, especially if you install ifupdown2. With that package, it's not required to restart the node every time network config changes. But yes, there is a much steeper learning curve if you'd want to do some advanced networking on ProxMox.
4) Re clustering: you are underestimating fast growing environments, or use cases where you need to move machines between datacenters. Another use case is project tiny-mini-micro (covered by STH), where you have many small nodes, which you need to reinstall sometimes, add new ones or remove a lot of out-of-date equipment.
5) I think I said few things about VMDK2RAW or VMDK2QCOW conversion, and in my opinion, there should be an option included in the WebUI for this purpose. I love CLI, it's very efficient for bigger tasks, but sometimes it is easier to just press a button to import the virtual image.
These are just my humble opinions after all, don't take them too seriously :)
@@GatewayITTutorials for automatic snapshotting, just install debian package "zfs-auto-snapshot" on the proxmox host.
(and additionally you could install cockpit and the "cockpit-zfs-manager/zfs /usr/share/cockpit" to get a beautiful snapshotting managment solution directly on your host)
I can not see the developers ever implementing a "native" snapshotting-function in proxmox,
because this type of snapshot function is just stupid to implement on the hypervisor level.
Because if you snapshot a, say 10TB fileserver, it takes hours/days to delete a snapshot and going back to a single vmdk/vhd file.
Even vmware and netapp had a (Storage)native snapshotting implementation like 5 years ago.
Heeeeeey, quick question:
Why would you keep a 10TB file server as a VM o_0
Both ESX and NetApp are targeted at enterprise: they are big enough to dictate the market, not other way around, so I don't look at them as a good example of anything.
ZFS snapshots and replication is the best thing I've seen that you can do for your backups:
- Snapshots will be identical across all systems, so disaster recovery is a breeze
- ZFS can fit nearly unlimited number of snapshots
- Replication happens over SSH, no need to worry about any other nonsense software or unencrypted protocols
- No need to worry about a shared storage: just use local storage with replication
Obviously this is not a solution for everyone, but it's as simple and reliable as it gets
Amazing video. Liked and subscribed
ProxMox
Backup using ProxMox BAckup Server is... well next level :)
Using ProxMox now for hosting a FreeNAS build, but I've got a box I'm building to try XCP-NG to mainly use for home labs. Just not happy with ESXi.
I'd say if it is a small enough place where you are expecting to just have 1 hypervisor, use proxmox.
If you are planning on expanding and may potentially later see yourself using 2 or more hypervisors and would have a separate storage server and may be replicating VM's across each other and anything larger, then XCP-ng is your choice.
I personally use a lot of proxmox for businesses I manage.
Thank you for your input. I'm deciding which software to run for my homelab. Your comment is valuable.
The logs can be moved on XCP-NG. I’ve had two usb drives die but it was over the course of years.was simple to fix
I recommend CEPH starting with 2 disks in one node. That means... almost always.
CEPH is cool, but it is designed for bigger deployments, with at least 3 nodes (but ideally 5, to give you the flexibility to loose 2 in case of disaster).
Did You tried Rocky OS (centOS)?
What's bee hive in freebsd you talked about?
It's bhyve, not bee hive) but that's a funny way of spelling it) you made my day bro)
bhyve is a FreeBSD hypervisor, kinda new in this space, but very ambitious. I am planning to do a video on it in the next couple of weeks.
bhyve.org/
r/nopunintended
I am not getting where you were going with your comment (I've got no means to insult you). It was either a pun or you are taking this a bit too seriously :)
That's absolutely alright, I can't control how other people perceive me (nor do I want to control that), and I always say what I think is right (and what's "right" is always subjective to the person saying it).
Your first comment on the other hand, came across (at least to me) as some sort of mockery, but again that's subjective :)
you can live migratve VMS on local storage, since proxmox 6.x so even at the time of release of this video it was possible
Not if they are located on top of ZFS pool, but other than that it's possible, yes.
Are you using an old laptop as server?
I use a variety of hardware:
Few laptops, few Intel NUCs, 2 iMacs (2011 models) and 3x HP G8 1U servers connected to TrueNAS :)
Then, depending on the software, I'll deploy it to the appropriate cluster.
I wished that there was a virtualisation system/platform that combined the best of both worlds.
I am currently using Oracle VirtualBox on a Windows Host and I think Chris Titus did a video showing how much faster a VM can be using qemu, except that qemu only runs in Linux and it isn't as easy nor straightforward (i.e. "dumb" point-and-click to deploy).
I am using an Intel Core i7-6700K right now, and I've got somewhere between 8-10 VMs running off that system right now (plus the host).
So, it would be nice if I can run those VMs faster vs. the speed that they're currently running at.
I might fire up two of my older workstations, and one on, throw XCP-ng and Xen Orchestra on it and on the other, throw Promox VE on it and see which one works better in the end.
With Proxmox, you set which network you want for corosync. The reasoning why proxmox clustering stabbed you in the back was user error, thus, you cannot blame the technology as bad.
I know there is a dedicated link option, and/or redundant link/links for corosync, but that's not the point) The point is that I had to explain: clustering over WAN is a no go. I've seen people do it too many times in my consulting practice, hope you understand.
Thanks for this!
Great video
Thanks for the video!
At th-cam.com/video/Aqhkd4-S66c/w-d-xo.html you said that proxmox is missing encryption at rest - but you can have your zpool under cryptsetup / luks. You can also use native zfs encryption (but I guess that's fairly recent addition).
As for the ProxMox:
Crypt/luks is very inflexible when it comes to dynamic password change, or when you want to use a key instead of password.
ZFS dataset encryption (on the root pool) only works under UEFI systems, if your system does not support UEFI, thought luck, then create a separate pool in order to use ZFS dataset encryption.
If you will still go ahead and encrypt one of the datasets on the root pool (under BIOS system) then your system won't boot anymore (trust me, I tried it :D). That's a Grub2 limitation, which is unlikely to be fixed any time soon.
But with Bhyve and ZFS I can load the password/key at any time via a short SSH command, or number of other ways. It always comes down to usability :)
We still use ProxMox on a number of our systems, due to some feature limitations on Bhyve. You gotta use a right tool for the job, and both ProxMox and Bhyve are just tools.
Thanks for the comparison. Unless I missed it (me multi-tasking), maybe you can compare ease of updating XCP-ng vs Proxmox. I like the ease to copy/clone VM from one XCP server to another.
I'm currently playing around with Proxmox, but figuring out how to import VM from XCP-ng was trial and error. I have been using an older Xen/XCP-ng (and Center) for my Win development VMs, but have issues with running two Win VMs at same time on a i7-4790 via RDP (yes an old processor!). Normally I only have one running, unless I need to troubleshoot an issue with my other software on the other VM. Between slowing down and then one dropping RDP connection, I decided to try out Proxmox on an i3, which would run both (slow) but didn't loose RDP connections. Now experimenting with Proxmox on a DL380 g8 dual-cpu (no SMART info passes thru from raid card) and it is plenty fast with no problems running multiple Win VMs. Will install latest XCP-ng on DL380 next, as I've been using an older version to see if the issue is gone; if so, I most likely stick with XCP-ng.
ProxMox is much easier to update and jump between major versions, all of it can be done with a simple APT command, where with XCP-Ng you need fresh ISOs every time there is a major update.
ProxMox will always be a bit more snappier, or faster if one can say that, because it's based on KVM, which is one of the fastest hypervisors out there.
I use vSphere
I find Esxi incredibly difficult to import Vmdks but love the UI
Good video!
Proxmox has automatic backups and also automatic snapshots th-cam.com/video/1CT6uh-fasg/w-d-xo.html th-cam.com/video/GoZaMgEgrHw/w-d-xo.html
The import/export hassle of PM stinks, but the ZFS support looks like it trumps that. Wait, it's just the root xcp-ng that can't be on ZFS, but still may be lacking some ZFS gui abilities? th-cam.com/video/XLQp_jI5vNs/w-d-xo.html
For homelab, my pick would be XCP-NG because of these:
- OVA import/export (for people that migrate from other homelabs) so it's more openness. VMDK/OVA are still out there to download if you need something like Hacintosh or exchange with other homelabbers.
- Nested Virtualization seems better working (Proxmox still had issue topics as of 2019 requiring some manual interventions, while I need Windows VM with Docker in it, so it requires Nested-V). I believe it should be just a checkbox.
- Passthrough seems easier (current manuals as of 2022), as well as commitment to get it done for various vGPUs working and so on.
Those are main hiccups.
I'm migrating from ESXi 6.7 because of introduced core limitation (5.5 had no core limitation, but had Nested-V problem so I had to upgrade). ESXi 7.0 eliminates native Realtek support, free cores aren't growing. Time for change. And during change, I'll continue to have both hypervisors.
Proxmox feels a bit like jail. May be comfortable one, but who knows if you'll get out of there if you need to. XCP-NG seems like innocent "I can change you anytime" which will lead to... not changing probably. I may comment later if I found any issues with XCP-NG on my main hiccup points.
First encounter with XCP-NG - Import works through XO. OVF/OVA export is basically non-existent (UPD, if you build XOA from May 2022 OVA exists). Xen Orchestra (at least the default one) doesn't have the ability - XVA export only. XCP-NG Center doesn't properly import or export at any format, but UI seems to support OVA/OVF if it was working in the first place. (XCP-NG 8.2.1 + Center 20.04.01.33)
Nested-V is also became a problem. With Nested-V enabled, previously built VMWare VMDK doesn't boot (Windows BSODs).
It runs under ESXi 6.7 and Windows 10 / Hyper-V, but no success on XCP-NG.
I tried everything in XOA or XCP-NG Center can do to modify VM settings in different ways: no luck in getting Nested-V VM even started. It doesn't even boot (in a way that different Hard Disk driver or different BIOS would help to load).
However, after installing Proxmox I was able to run Docker in Windows / Nested-V in about 4 hours. I guess that's it. My UI Import/Export sympaties are with XCP-NG, but unfortunately, Proxmox worked so I'll go with that.
@@clockware I had no problems installing headless Ubuntu as a VM and running Docker. Years ago I turned on Nested-V for running jails in TrueNAS core and it basically worked. I abandoned the jails for Docker and Portainer though, a much more widely used method of containerization, more support, works better.
In your other comment you mention importing VMs: Build the latest Xen Orchestra from sources and try the import menu item. I avoid XCP-ng Center because its actually kind of buggy and dangerous. Xen Orchestra is safe.
I use proxmox
... "you can not import a VM from VirtualBox ..." in Proxmox. I would consider this just wrong. Install Proxmox on a cheap Optane with NVME to USB adapter. Done.
But there is no import function in ProxMox, I don't know what you are trying to say here...
The only option you have is to manually unpack the OVA archive, convert virtual drive to RAW or qcow formats, then rename that file to something already present in the system and move it to the right place by hand, which is unnecessary complicated.
And then the sentence about the Optane and a USB adapter just doesn't make any sense. Not to be rude, but try to re-read what you've written there, you might have missed something.
@@GatewayITTutorials Well, first of all you can make yourself a simple script to convert and import other image formats of virtual machines. Thats super easy and a sys-admin like you shouldnt be bothered by it. I wouldnt take that as disadvantage. Quite the opposite. At the same time you re-configure the image to your needs. Xnp-ng seems to not be that flexible.
You were saying Proxmox isn’t suitable to be installed on a USB drive in terms of wearing out USB drives quickly due to the extensive logging. You are quite wrong. There are a lots of people, even dc operators, who are running cheap 16gb Optane drives as os drive with Proxmox installed and connect them to NVMe to USB connectors, which then are connected to internal usb connectors. this is a cheap all in all 40 US$ solution and very reliable. The setup is fast enough for the OS, the Optane drives are bootable, they have a 300+TB endurance which is high enough for heavy logging.
So, set yourself aside and research a little bit before you produce a vid with such ‘questionable’ statements and then give very aggressive replies.
This video is targeted at people who don't know how to make scripts, and just want to experiment with the platform.
My comment wasn't angry, I was genuinely trying to understand what you have to say, and now I do, thank you for the clarification.
The point about USB drives:
Did you hear me say - you can't use USB to sata adapter and install ProxMox onto the SSD? I assume the answer is no. What I was talking about is a cheap USB thumb drive, short and compact, so in case someone accidentally hits it, it wouldn't brake the board or itself into pieces.
I used external drives to install ProxMox myself, but due to a nature of the system (many random reads and writes to the FS) you are somewhat limited with your drive choices.
I love ProxMox, it's a great product, but many homelab users don't bother to write scripts for the migration, don't bother to read the docs, and so on. That's why I had to bring up the points I brought up in the video. Most of these will never be applicable to enterprise or DC, but nonetheless they have their use cases.
@@GatewayITTutorials I believe you totally underestimate the abilities of your audience. But ok, ....
Bro, I know my audience, they are brilliant people, and I am not trying to put anyone down, quite the opposite. Not everyone who will watch this video is professional in IT industry. Not everyone can or should write scripts, or think about automation among other things.
That's why they are comparing, listening to different opinions, learning new things.
I really do understand what you were trying to say, but hear me out too: sometimes people can have different opinions, and that's completely normal.
For an average person, it's imperative to have that "Import" function in the GUI, I know it, I've been there just few years ago. If you disagree, that's fine, it just means we have different views.
Great comparison and discussion. I had been considering using XCP-NG over Proxmox VE. This helped me make the right decision 🦾. Thanks 🙏