Hi, this is pretty great! We set a plain KVM with Ceph on AlmaLinux in our production and it's been over a year now. Now we're setting up the Promox HCI on DR infra. Long Live KVM!
@@CodetheThings We only have the standard support option. The most troublesome loads where old unsupported windows servers. We started over a year ago so we built an ansible driven migration engine for low-downtime migrations. Proxmox has matured quite a lot lately and with the vmware exodus I hope they invest resources into R&D. Mostly I think vmware admins are too comfortable to learn KVM/linux and discard it as a non-enterprise platform. Its not vmware (yet) but its far from just being a homelab solution.
@@tactoad That's awesome! I have seen a flurry of updates and releases from Proxmox and other vendors since the VMware purchase. Right now, trying to sort through what is real and what is hype has been time consuming. That said, people are definitely making the switch to something and that's a pleasant surprise. I figured we would've seen a lot of 'upset' folks who never end up migrating off of VMware.
@@CodetheThings Well, we spent evaluating options for vmware for over 3 years. We pretty much tried everything under the sun (hyper-v, nutanix, RHV, xcp-ng etc) but settled with Proxmox. Being a MSP is a competitive market and cutting licensing costs was a big incentive for us. Dodging the Broadcom debacle was just icing on the cake ;) The only way to evaluate solutions is to test them in your environment. If you only go by features on marketing material be ready for disappointment. For me personally these are exciting times as the virtualization market was stale for so long until now. Feel free to reach out to me if you want details from our evaluations.
@@CodetheThings So i wrote a lenghty reply that disappeared. Anyway...we tried everything under the sun so feel free to reach to me if you want our notes from the hypervisor evaluations.
Subbed. My VMUG runs out in September, need to start exploring alternatives. I might miss vSAN and Tanzu in my homelab but my thoughts about Broadcom mirror yours.
Morpheus has the ability to deploy K8s clusters (MKS) or import. I've also created my own Ansible to deploy a custom cluster layout. That said, finding the best option all around is a lot more heavy lifting than I expected for this summer :D
This is the first time i've heard of Morpheus. I'll have to dig into it a little more to see if it might be worth doing some testing. I've been using proxmox for the past few years and like it a lot, but it does have some shortcomings. I'm really curious about your thoughts when it comes to ceph performance, because I tested a 3 node ceph cluster on proxmox recently (using consumer nvme) and found abysmal performance compared to just using the nvme alone and then using the migration synchronization features found in proxmox to handle migration (it's far from the smooth ceph experience but it's significantly more performant when things are up and running). I hope you tell us your findings in a future video :D
For CEPH to perform well on nvme needs more tuning then outlined in the proxmox docs. you need a fast cluster network for the replication. Also depending on the size of the disks you might want to partition them over several OSDs to maximise IOPS. Using consumer disks for anything than lab is not recommended.
NVMe is massively overkill until you have min 25Gbps networking. My home lab gets a decent budget, but not that nice :-) Also, 100% consumer shouldn't be used outside of a lab. I already hit some pretty low bottlenecks utilizing consumer disks and I'm not running large scale databases or anything.
(copying from another reply) Initial testing I wasn't overly sold on proxmox. I think it'll be difficult to see adoption at the enterprise level. Things like DRS and overlay networks are still not in production and would be requirements for most folks. Also, I'm not sure what Proxmox's enterprise support models look like, but the purpose of my home lab is leisure and enterprise training and testing. Anything I run should be used in some scale with customers I work with. Morpheus is already an enterprise-sold solution, now with added features like MVM.
Very interesting. I've been disappointed with xcp-ng having a limitation with the number of vlans it will support. Any reason for preferring ubuntu over parent disro debian?
There's a community license which lets you run it in your lab for free. The best link I can find for that at the moment is this (driving atm): support.morpheusdata.com/s/article/How-to-install-Morpheus-with-a-free-community-license?language=en_US For larger scale they have enterprise licensing. Think vRA/vRO/vRealize/Aria/Whatever Broadcom names it today.
Morpheus has a free community license. There are 'maximums' but in most home lab environments those aren't thresholds you would hit. After that it goes into ELAs. This is the easiest link that talks about the community stuff and links to their hub to register: support.morpheusdata.com/s/article/How-to-install-Morpheus-with-a-free-community-license?language=en_US
I used to be a Windows shop but now I'm mostly Linux VMS or a nested k8s cluster. I do install so many various apps for testing I think lxc would be hard to efficiently make use of my hardware.
@CodetheThings it sounds like a nice setup you have. I'm using Proxmox with CEPH, so it's a slightly different animal. I like the speed of LXC containers.
Initial testing I wasn't overly sold on proxmox. I think it'll be difficult to see adoption at the enterprise level. Things like DRS and overlay networks are still not in production and would be requirements for most folks. Also, I'm not sure what Proxmox's enterprise support models look like, but the purpose of my home lab is leisure and enterprise training and testing. Anything I run should be used in some scale with customers I work with. Morpheus is already an enterprise-sold solution, now with added features like MVM.
@@roccociccone597 There is a community edition which I run in my lab. (support.morpheusdata.com/s/article/How-to-install-Morpheus-with-a-free-community-license?language=en_US) There is a closed beta right now of MVM that should be open in a few weeks. The other features are nice to have in a lab either way. Automation/Orchestration is always nice!
I am using proxmox but now i am moving towards openstack because of celph so my proxmox and openstack can work together. Just for curiousity and leaening no hate for anyone except vmware because i was unable to passthrough usb keyboard :(
Currently Morpheus configures with the default replication configuration. I could choose the non-HCI layout and build ceph manually, or destroy and configure as needed. Part of my testing will be different configuration/optimizations and failure testing of ceph.
Everything I can! I setup Steam gaming servers for my friends on it, use it to spin up just about any hypervisor/app my customers use for training, and media / editing storage.
I'm going to circle back with XCP-NG. I''ll probably start on nested first and then go from there. I've never had hands on with it, but my customers are bringing it up more often so I'll need to look at it asap. Is that what you are currently running?
@@CodetheThings At home, not anymore. I got rid of all enterprise grade servers and moved to micro PC's(USFF). I had it before on one server just for testing and on one of my old jobs, run it in prod. From my perspective - it's the closest thing that feels and behaves like VMware. Plus it has good compatibility for VMware as well.
I love ProxMox it is not quite as good as VMware ESXI but it is free and ESXI is not free anymore. Although I think ProxMox may be almost as good I never had any trouble with it
I think what Broadcom/VMware forgot is the free/VMUG is why they had such a strong following. It was easy to load up and utilize in small environments, which enabled constant training/expertise/growth.
Last I read free ESXi in any capacity has been discontinued (as well as VMUG subscriptions). They did announce Fusion was free for use, but not really solving the dedicated hosts.
@@CodetheThings from their website. The free version is available for non-commercial, personal and home use. We also encourage students and non-profit organizations to benefit from this offering.
Hey, libvirt is essentially a nice wrapper for KVM & QEMU. You could even run Oracle VM Server for x86 it's based on OpenSource Xen. Ditch RHEL as well and move to other variants like Rocky, or Suse or Oracle Linux, Debian, hell even OpenIndiana/Illumos (OpenSolaris), *bsd. It's time to fight against these damn profit mongers and patent/license trolls.
Why would you go through all of that work when you can put all of your servers in a proxmox cluster and run truenas in a vm and have everything easy to access and make vms in proxmox so you can run unbuntu or what ever your running in virtual machines
I'm not overly sold on proxmox. Also, anything I run needs to be enterprise supportable at scale as my lab is used for home and professional training. That said I'm still doing discovery on what's going to be the best route, ease of support and access to documentation, etc. And yes, I agree having to spin up all of that at the moment is a lot, but being a beta release currently I'd expect ease of deployment a focus in the future.
Because Proxmox is NOT an appliance and as such, it’s not enterprise ready. Having to modify the grub boot loader by hand to enable pass through!? An appliance is a device/service that you can get back up running with just one configuration file.
@@CodetheThingsyou think a lot like me as I do use .0 and .3 networks for admin and access. Using TrueNAS Scale after moving from ESXi. I have a secondary server that mimics the main server, for testing or in case the main stops working. So do not need a cluster, but will check that Morpheus community version.
@@pepeshopping nice! There is a non-HCI layout that could utilize n+x hosts and bring your own storage solution. Right now , support outside of Ceph is NFS only which works just as well. I believe by the open beta the plans are iscsi, cif support as well (which is a few weeks away).
I know there are several converter options. That's part of my tests with the different hypervisors. I have a lot of data that needs to move off of vmdk.
I won't ever turn down more nodes :-) Technically whatever I move to I could flex up to 6, but my hosts are (3) R740XD and (3) R620 so a bit of a diff in hardware. Any other key settings to look into adjusting in ceph? I have ceph dashboard running but haven't messed with it too much yet.
There are for sure a lot of options out there. Luckily I work in the general space of compute and have been hands on with a lot of the VMware alternatives. Finding a good mix of capabilities, ease of management, and cost is going to be the trick.
That's way out of their league. Other things they do are out in space, but enterprise/server/professional systems just isn't what they do 24x7 for a living.
Stay with VMware my man, still way ahead of any other hypervisor out there and is not going anywhere. Might even get better. Just like any transition it will take time to iron out something comfortable for both Broadcom and end users.
@@pepeshopping yet they end up hopping from one solution to the next because they all lack of a certain feature. But you are right on that for basic tasks they all work. But in my professional opinion VMware still top dog on virtualization and other things.
I agree they have been a dominant force. However, I believe that Broadcom has shown their nearsightedness and did a really poor job retaining customer favor with drastic pricing changes and software EOL. I'll be curious on how it all plays out, but I think if anything VMware/Broadcom showed all of your eggs in one basket is a bad idea.
Given Broadcom's history with acquisitions the last thing they are interested in is finding a middle ground. They will milk large VMware customers for all they can because they know it would be very time consuming and expensive for them to move to another platform. I'd love to be wrong about that but so far everything is pointing in that direction.
Hi, this is pretty great!
We set a plain KVM with Ceph on AlmaLinux in our production and it's been over a year now. Now we're setting up the Promox HCI on DR infra.
Long Live KVM!
How large is your infra? Curious at what scale alternative hypervisors are seeing.
Morpheus, what an interesting choice for a home lab.
We aready migrated 2K VMs from vsphere to Proxmox, however this seems like a very nice kvm managent solution.
Oh wow! How are the enterprise support options for Proxmox? Any workloads that you couldn't move?
@@CodetheThings We only have the standard support option. The most troublesome loads where old unsupported windows servers. We started over a year ago so we built an ansible driven migration engine for low-downtime migrations.
Proxmox has matured quite a lot lately and with the vmware exodus I hope they invest resources into R&D. Mostly I think vmware admins are too comfortable to learn KVM/linux and discard it as a non-enterprise platform. Its not vmware (yet) but its far from just being a homelab solution.
@@tactoad That's awesome! I have seen a flurry of updates and releases from Proxmox and other vendors since the VMware purchase. Right now, trying to sort through what is real and what is hype has been time consuming.
That said, people are definitely making the switch to something and that's a pleasant surprise. I figured we would've seen a lot of 'upset' folks who never end up migrating off of VMware.
@@CodetheThings Well, we spent evaluating options for vmware for over 3 years. We pretty much tried everything under the sun (hyper-v, nutanix, RHV, xcp-ng etc) but settled with Proxmox. Being a MSP is a competitive market and cutting licensing costs was a big incentive for us. Dodging the Broadcom debacle was just icing on the cake ;)
The only way to evaluate solutions is to test them in your environment. If you only go by features on marketing material be ready for disappointment. For me personally these are exciting times as the virtualization market was stale for so long until now.
Feel free to reach out to me if you want details from our evaluations.
@@CodetheThings So i wrote a lenghty reply that disappeared. Anyway...we tried everything under the sun so feel free to reach to me if you want our notes from the hypervisor evaluations.
Subbed. My VMUG runs out in September, need to start exploring alternatives. I might miss vSAN and Tanzu in my homelab but my thoughts about Broadcom mirror yours.
Morpheus has the ability to deploy K8s clusters (MKS) or import. I've also created my own Ansible to deploy a custom cluster layout. That said, finding the best option all around is a lot more heavy lifting than I expected for this summer :D
This is the first time i've heard of Morpheus. I'll have to dig into it a little more to see if it might be worth doing some testing. I've been using proxmox for the past few years and like it a lot, but it does have some shortcomings.
I'm really curious about your thoughts when it comes to ceph performance, because I tested a 3 node ceph cluster on proxmox recently (using consumer nvme) and found abysmal performance compared to just using the nvme alone and then using the migration synchronization features found in proxmox to handle migration (it's far from the smooth ceph experience but it's significantly more performant when things are up and running). I hope you tell us your findings in a future video :D
For CEPH to perform well on nvme needs more tuning then outlined in the proxmox docs. you need a fast cluster network for the replication. Also depending on the size of the disks you might want to partition them over several OSDs to maximise IOPS.
Using consumer disks for anything than lab is not recommended.
NVMe is massively overkill until you have min 25Gbps networking. My home lab gets a decent budget, but not that nice :-)
Also, 100% consumer shouldn't be used outside of a lab. I already hit some pretty low bottlenecks utilizing consumer disks and I'm not running large scale databases or anything.
Why not proxmox?
🤢
@@davydotcom ?
(copying from another reply)
Initial testing I wasn't overly sold on proxmox. I think it'll be difficult to see adoption at the enterprise level. Things like DRS and overlay networks are still not in production and would be requirements for most folks.
Also, I'm not sure what Proxmox's enterprise support models look like, but the purpose of my home lab is leisure and enterprise training and testing. Anything I run should be used in some scale with customers I work with. Morpheus is already an enterprise-sold solution, now with added features like MVM.
@@CodetheThings hmm I see🤔
Very interesting.
I've been disappointed with xcp-ng having a limitation with the number of vlans it will support.
Any reason for preferring ubuntu over parent disro debian?
Pretty much any customer I have uses Ubuntu or RHEL, so from that standpoint Ubuntu seemed to be the best go ahead.
@@CodetheThings I wish I did not have to say the same about Microsoft Windows 😉🤣👍
This is really cool! I haven't heard of Morpheus before this video? is it a paid software and if so, how much is the cost for the license?
There's a community license which lets you run it in your lab for free. The best link I can find for that at the moment is this (driving atm):
support.morpheusdata.com/s/article/How-to-install-Morpheus-with-a-free-community-license?language=en_US
For larger scale they have enterprise licensing. Think vRA/vRO/vRealize/Aria/Whatever Broadcom names it today.
Morpheus looks pretty sweet. Wonder if it's close to being a decent vmware alternative. Might have to check it out.
How accessible is morpheus for the Home lab user? What does licensing look like compared to VMUG?
Morpheus has a free community license. There are 'maximums' but in most home lab environments those aren't thresholds you would hit. After that it goes into ELAs. This is the easiest link that talks about the community stuff and links to their hub to register:
support.morpheusdata.com/s/article/How-to-install-Morpheus-with-a-free-community-license?language=en_US
Do you have lots of linux VMs? If yes, LXC for containers is worth considering.
I used to be a Windows shop but now I'm mostly Linux VMS or a nested k8s cluster. I do install so many various apps for testing I think lxc would be hard to efficiently make use of my hardware.
@CodetheThings it sounds like a nice setup you have. I'm using Proxmox with CEPH, so it's a slightly different animal. I like the speed of LXC containers.
Is there a reason for why you didn't use Proxmox?
Initial testing I wasn't overly sold on proxmox. I think it'll be difficult to see adoption at the enterprise level. Things like DRS and overlay networks are still not in production and would be requirements for most folks.
Also, I'm not sure what Proxmox's enterprise support models look like, but the purpose of my home lab is leisure and enterprise training and testing. Anything I run should be used in some scale with customers I work with. Morpheus is already an enterprise-sold solution, now with added features like MVM.
@@CodetheThings I see. I didn’t know about Morpheus, is there a free version?
@@roccociccone597 There is a community edition which I run in my lab. (support.morpheusdata.com/s/article/How-to-install-Morpheus-with-a-free-community-license?language=en_US) There is a closed beta right now of MVM that should be open in a few weeks. The other features are nice to have in a lab either way. Automation/Orchestration is always nice!
I am using proxmox but now i am moving towards openstack because of celph so my proxmox and openstack can work together. Just for curiousity and leaening no hate for anyone except vmware because i was unable to passthrough usb keyboard :(
For Ceph, do you use replication or erasure coding ?
Currently Morpheus configures with the default replication configuration. I could choose the non-HCI layout and build ceph manually, or destroy and configure as needed.
Part of my testing will be different configuration/optimizations and failure testing of ceph.
what kind of homelab has this kind of setup, what are you doing ?
Everything I can! I setup Steam gaming servers for my friends on it, use it to spin up just about any hypervisor/app my customers use for training, and media / editing storage.
@@CodetheThings Thats pretty neat
Was XCP-NG & Xen Orchestra an option?
I'm going to circle back with XCP-NG. I''ll probably start on nested first and then go from there. I've never had hands on with it, but my customers are bringing it up more often so I'll need to look at it asap.
Is that what you are currently running?
@@CodetheThings At home, not anymore. I got rid of all enterprise grade servers and moved to micro PC's(USFF). I had it before on one server just for testing and on one of my old jobs, run it in prod. From my perspective - it's the closest thing that feels and behaves like VMware. Plus it has good compatibility for VMware as well.
the only issue with ubuntu server 2404 setting up the auto updates on the server side not easy and outdated info too would do do a video on that
I love ProxMox it is not quite as good as VMware ESXI but it is free and ESXI is not free anymore. Although I think ProxMox may be almost as good I never had any trouble with it
I think what Broadcom/VMware forgot is the free/VMUG is why they had such a strong following. It was easy to load up and utilize in small environments, which enabled constant training/expertise/growth.
Are you doing this as residential or commercial? If residential, you should know that VMWare is now free for personnel use.
Last I read free ESXi in any capacity has been discontinued (as well as VMUG subscriptions). They did announce Fusion was free for use, but not really solving the dedicated hosts.
@@CodetheThings from their website.
The free version is available for non-commercial, personal and home use. We also encourage students and non-profit organizations to benefit from this offering.
Hey, libvirt is essentially a nice wrapper for KVM & QEMU. You could even run Oracle VM Server for x86 it's based on OpenSource Xen. Ditch RHEL as well and move to other variants like Rocky, or Suse or Oracle Linux, Debian, hell even OpenIndiana/Illumos (OpenSolaris), *bsd. It's time to fight against these damn profit mongers and patent/license trolls.
Nebula?
Why would you go through all of that work when you can put all of your servers in a proxmox cluster and run truenas in a vm and have everything easy to access and make vms in proxmox so you can run unbuntu or what ever your running in virtual machines
I'm not overly sold on proxmox. Also, anything I run needs to be enterprise supportable at scale as my lab is used for home and professional training. That said I'm still doing discovery on what's going to be the best route, ease of support and access to documentation, etc.
And yes, I agree having to spin up all of that at the moment is a lot, but being a beta release currently I'd expect ease of deployment a focus in the future.
Because Proxmox is NOT an appliance and as such, it’s not enterprise ready.
Having to modify the grub boot loader by hand to enable pass through!?
An appliance is a device/service that you can get back up running with just one configuration file.
@@CodetheThingsyou think a lot like me as I do use .0 and .3 networks for admin and access.
Using TrueNAS Scale after moving from ESXi.
I have a secondary server that mimics the main server, for testing or in case the main stops working.
So do not need a cluster, but will check that Morpheus community version.
@@pepeshopping nice! There is a non-HCI layout that could utilize n+x hosts and bring your own storage solution. Right now , support outside of Ceph is NFS only which works just as well. I believe by the open beta the plans are iscsi, cif support as well (which is a few weeks away).
Love this :) we all kicked ESXi
Proxmox using the VMWhere import
er
I know there are several converter options. That's part of my tests with the different hypervisors. I have a lot of data that needs to move off of vmdk.
You'd normally want more nodes with ceph. Maybe look at i.e. 3 quad node boxes (they dont need to be maxed out at all!) for the next iteration.
I won't ever turn down more nodes :-) Technically whatever I move to I could flex up to 6, but my hosts are (3) R740XD and (3) R620 so a bit of a diff in hardware.
Any other key settings to look into adjusting in ceph? I have ceph dashboard running but haven't messed with it too much yet.
You can always send a question about this to Linus Tech Tips and see what advice they give. You are running Linux so there should be many ways to go.
There are for sure a lot of options out there. Luckily I work in the general space of compute and have been hands on with a lot of the VMware alternatives. Finding a good mix of capabilities, ease of management, and cost is going to be the trick.
I have never followed or enjoyed LTT.
That's way out of their league. Other things they do are out in space, but enterprise/server/professional systems just isn't what they do 24x7 for a living.
Stay with VMware my man, still way ahead of any other hypervisor out there and is not going anywhere. Might even get better. Just like any transition it will take time to iron out something comfortable for both Broadcom and end users.
“Way ahead” means nothing.
List the details.
For most users, other hypervisors “way behind” do everything they need.
@@pepeshopping yet they end up hopping from one solution to the next because they all lack of a certain feature. But you are right on that for basic tasks they all work. But in my professional opinion VMware still top dog on virtualization and other things.
I agree they have been a dominant force. However, I believe that Broadcom has shown their nearsightedness and did a really poor job retaining customer favor with drastic pricing changes and software EOL.
I'll be curious on how it all plays out, but I think if anything VMware/Broadcom showed all of your eggs in one basket is a bad idea.
Given Broadcom's history with acquisitions the last thing they are interested in is finding a middle ground. They will milk large VMware customers for all they can because they know it would be very time consuming and expensive for them to move to another platform. I'd love to be wrong about that but so far everything is pointing in that direction.
@@nadtz they were being sued in Europe and pulled back a bit, but we’ll see what happens.
To me, watching this is way better than porn!
proxmox all day!