I had 3 old laptops, why not make a cluster on them, and put some vms on the cluster. I setup some shared storage with Ceph and go over all the issues I ran into
I easily got ProxMox running on a new laptop computer after a week of trying to install ProxMox from the same USB on a old Windows HP compatible system that just did not want to boot the ProxMox ico file from the USB. Just hung at GRUB Welcome screen. This video probably saved a few hundred dudes 30+ hours and who knows how much cash. Your awesome
His vids watched sequentially will lesson this ol' boomer's learning curve big time! You sir are a true credit to your craft!. I've done that marathon dance lol. .Messed up grub broken mbr legacy or eufi...... 'what? Wait ! ... how did secure boot get reenabled again ... Oh Damn I didn't unplug and drain down (or did it too quickly..smh) grrr.. lol Now at least 3 healthy lthis decade sff or hp800 g3 minis Sooo much tweaking can be done. Load balance all your traffic severed locally with very low wattages. wow. AND there's still millions of 'em out there. crank down a small board dual or quad port sata connection an nvme or two .... rotating Native or Network Avail. Storage,
Thanks for making these videos. I'm new to the channel and really liking the content so far. Appreciate you walking through the process and challenges. Its a great way to learn. I have quite a bit of old hardware and this gets me to thinking to play around with this stuff just to experiment.
I've always steered away from ceph as it scared me haha! Having shared storage on a separate, dedicated box (or boxes, NAS/SAN etc) always seemed like a better idea to me. Cool vid in any case, thanks. 😊
i played with ceph in my homeLAN (not lab) and have decided against it due to a number of reasons. 3x is needed minimal for Ceph to operate with default configs. the default config requires 2 nodes to "verify" each write. so in a 3 node cluster, 1 node can go offline. however, when there is only 1 node remaining, *Ceph cannot obtain the required 2 nodes verification and blocks the write.* you can edit the Ceph config to allow only 1 verify if need be (and risk all of the auto-repair Ceph goodness out there when a 2nd disk fails!); however, that would not be a reliable data system and you could get conflicting data when the other node rejoins - thinking *it* was the master of the data. also, Ceph's fsycn ack and PVE's corosync requires very low latency - something a saturated 1 Gbps (e.g. VM migration) greatly makes impossible. so you had two serious issues with Ceph. lastly, even a 3 node Ceph cluster is highly not recommended. because once your cluster gets in such a shape as you saw, it's pretty much impossible to bring back to normal due to the fsync conflicts of two sets of data that are not equal when both thought they were the master. the point here is: *Never let your Ceph get into such an error state, because it's pretty much unrecoverable*. So how do you avoid that with a requirement for 2x verify? You run a LOT more nodes! Ceph staging/dev clusters are limited to 5x nodes dead minimal (because the servers can go offline in a 'safe' state if you loose two nodes that 2 of the 3 copies of data are on). Production? Absolute minimal is 9 nodes. Ceph's real power comes from running the dozens, if not 100s of servers as hyperconverged at Petabyte scales. Performance from IOPS only really starts to get to the standard SSD levels - at a queue depth of say 4 vms writing at the same time! - when you are at 6 or 7 nodes (and this was on very fast SAS3 SSDs, separating WAL and all the bluestor tweaks). a lot of people tend to forget queue depths... and let me tell you, given enough time, corosync will start to fall on its face as well over such a setup. corosync really needs its own dedicated 1 Gbps link with as few hops as possible to be happy - with a 5x node cluster. what do I run at home? in the end, I decided against HyperConvergence in general mainly because of the heat generation from three to five nodes I'd need to run, and the performance numbers (even with enterprise SSDs!) wasn't up to my requirements. i did setup a single-node Ceph server. But in the end, I couldn't get the IOPS where I wanted it. oh, and tweaking Ceph is only done via "de-compiling", "building", and then "compiling" the config file back. there's zero GUI for it.
love this idea - to better this setup you want i7 refurbs with nvme and faster networks - cheap nodes - why is this so interesting for smb build 4 for quorum and then test for scaling with load balancing - biz wants cheap nodes but they can just spin up vms if they want to scale temp - the faster io and networks will work well- nvme scales - biz wants iops
Would be interesting to revisit the idea with three properly working laptops and CEPH. I have three identical HP Elitedesks with CEPH and works flawlessly. I have not made extensive tests with different VMs and how they survive the HA. I know my Ubuntu VM works and that is good enough for me.
1:09 To solve this problem, why not create a virtual machine on both devices? That way, if one laptop fails, Proxmox (I assume) thinks that 2 PCs (the host computer and the VM running on it) vote the same thing. I guess you could install a Linux VM on both to save resources. You would (possibly) still have high availability without the need for a third system (a third physical device). I haven't confirmed any of this myself, it's just a theory, but I see no reason why it can't be tried.
I tried many times and failed to build a cluster of old computers. But I succeeded in doing the opposite without virtualization, running 3 users at the same time on 1 windows os. software: ASTER
I’m just learning about home networking, and I have a few mini PCs that I’m trying to see what I can use them for. Is there any point it setting up a proxmox cluster for a home network?
Nope you need to make a cluster to move VMs between nodes easily. You could without a cluster setup the same storage share with backups and backup and restore VMs, but it will result in more downtime.
1. Does it increases speed? 2. How did u connect those two laptops pls show. I have 20 old dual core PCs in my lab , how can I parallely use these processor. Thank you
I haven’t worked too much with torrents, but there are similar concepts being used in software like lizardfs. The data is split into chunks and then a level of redundancy is set for files and extra copies are made if one is lost. Distributed storage though is normally a pretty bad experience on 1gbe as the network latency will significantly limit up performance.
@@jirehla-ab1671 I don't think there is any way to do that easily. You may be able to do it in qemu, but it would be emulating those instructions and much slower than traditional virtualization.
Nice vid. I'm building a similar PM cluster. However, I'm unable to get one of the machines to join the cluster. QQ: Do all the nodes in a cluster need to be on the same version of Proxmox? That's not the case in this home.
Yea that should be possible. I'd love to see what the project looks like when finished. A few things that might help you use the boards better is faster networking as the 1gbe on most laptops is slow for clusters, and more ram which helps for caching and other uses.
@@Svaraijt A switch normally connects to an existing ethernet port, so it likely won't help here. But you can get faster USB 2.5 and 5GBE network cards, and if thunderbolt is a option that should be able to do 10GBe speeds or faster.
i mean you could but i wouldn't suggest it. even if you can technically pool together the resources of multiple PCs to work on the same task in parallel, Game engines are notorious for not benefiting from simply adding more cores to split the workload. Usually adding anything beyond 8 cores has deminishing returns in terms of gaming, and thats first assuming the Game engine can even make use of all the cores. it isnt uncommon at all for games to offload most of the work on to the cores unevenly, they will pick and choose which cores to use and leave the rest barely used. This is why people do not build gaming rigs with Xeon and Eypc server CPUs even if they have tons more compute power than something like a Core i7 or i9 these CPUs will achieve greater FPS even having only a fraction of the cores, because Games typically respond better to just faster clock speeds, than just being able to split the workload. And in a cluster all this data has to also travel through ethernet, even with 10gbe networking between each PC that's nearly 300 times slower than how fast data transfers within a single modern system. So there is a ton of latency when it comes to rendering, it's not a smooth experience. A channel called NetworkChuck was able to cluster a bunch of raspberry Pi4s and play minecraft on them, sort of. As for the second part I guess it depends on what you're rendering.
In this video I think I only used wired connections. I may go over WiFi on proxmox in a future video. Look up guides for WiFi in Debian for steps that would also work in proxmox.
There isn’t a way to make it ack like one faster computer but if you have multiple VMs you can make a cluster and split the VMs and containers between the systems.
It may be technically possible but I’d say that hardware is too slow for a cluster. I’d probably set it up with the file share on the host and no vms but it really depends on the exact software setup you're using.
Core 2 Duo laptops? Nah. Don't revive them. Don't use Proxmox on them. It's not power efficient. Proxmox will crash on it, especially early model that adapt 64 bit.
I forget exactly where (Google will have the answer) but there are 3 lines in the config file you need to change to disable the laptop from turning off when the lid is closed. I did it on my systems.
If you don't have virtualization your mostly limited to containers. You can run vms via emulation, but that will be slow and if thebcou doesn't support virtualization is probably pretty slow cpu.
Thanks for the info, you talk about interesting topics....gets a little hard to watch all the way through. constructive criticism - focus on keeping your head level. - select words that you're able to say clearly.
You are seriously underrated brother. ❤️ Keep going you will reach great heights for sure. 🔥
Completely agree. Just found him and subed.
same. You got the tech chops, keep it up
his not underrated, people intersted in this stuff are low, ive been in industry for a decade and now i just started looking for this material
I easily got ProxMox running on a new laptop computer after a week of trying to install ProxMox from the same USB on a old Windows HP compatible system that just did not want to boot the ProxMox ico file from the USB. Just hung at GRUB Welcome screen. This video probably saved a few hundred dudes 30+ hours and who knows how much cash. Your awesome
His vids watched sequentially will lesson this ol' boomer's learning curve big time!
You sir are a true credit to your craft!.
I've done that marathon dance lol. .Messed up grub broken mbr legacy or eufi......
'what? Wait ! ... how did secure boot get reenabled again ... Oh Damn I didn't unplug and drain down (or did it too quickly..smh) grrr.. lol
Now at least 3 healthy lthis decade sff or hp800 g3 minis
Sooo much tweaking can be done. Load balance all your traffic severed locally with very low wattages. wow.
AND there's still millions of 'em out there. crank down a small board dual or quad port sata connection
an nvme or two .... rotating Native or Network Avail. Storage,
You explain things really well. I really like your videos. Great job!
Learned a lot and appreciate all the honesty and sharing of failure in your video. Well done and thanks you!
Awesome! This is how IT works: not always smooth, and not making a fuzz about it.
Hey mate Ive come across your videos a couple of times now, you seem incredibly knowledgable. Keep up the great work.
Thanks for making these videos. I'm new to the channel and really liking the content so far. Appreciate you walking through the process and challenges. Its a great way to learn. I have quite a bit of old hardware and this gets me to thinking to play around with this stuff just to experiment.
I've always steered away from ceph as it scared me haha! Having shared storage on a separate, dedicated box (or boxes, NAS/SAN etc) always seemed like a better idea to me.
Cool vid in any case, thanks. 😊
i played with ceph in my homeLAN (not lab) and have decided against it due to a number of reasons.
3x is needed minimal for Ceph to operate with default configs. the default config requires 2 nodes to "verify" each write. so in a 3 node cluster, 1 node can go offline. however, when there is only 1 node remaining, *Ceph cannot obtain the required 2 nodes verification and blocks the write.* you can edit the Ceph config to allow only 1 verify if need be (and risk all of the auto-repair Ceph goodness out there when a 2nd disk fails!); however, that would not be a reliable data system and you could get conflicting data when the other node rejoins - thinking *it* was the master of the data.
also, Ceph's fsycn ack and PVE's corosync requires very low latency - something a saturated 1 Gbps (e.g. VM migration) greatly makes impossible. so you had two serious issues with Ceph.
lastly, even a 3 node Ceph cluster is highly not recommended. because once your cluster gets in such a shape as you saw, it's pretty much impossible to bring back to normal due to the fsync conflicts of two sets of data that are not equal when both thought they were the master. the point here is: *Never let your Ceph get into such an error state, because it's pretty much unrecoverable*. So how do you avoid that with a requirement for 2x verify? You run a LOT more nodes! Ceph staging/dev clusters are limited to 5x nodes dead minimal (because the servers can go offline in a 'safe' state if you loose two nodes that 2 of the 3 copies of data are on). Production? Absolute minimal is 9 nodes.
Ceph's real power comes from running the dozens, if not 100s of servers as hyperconverged at Petabyte scales. Performance from IOPS only really starts to get to the standard SSD levels - at a queue depth of say 4 vms writing at the same time! - when you are at 6 or 7 nodes (and this was on very fast SAS3 SSDs, separating WAL and all the bluestor tweaks). a lot of people tend to forget queue depths...
and let me tell you, given enough time, corosync will start to fall on its face as well over such a setup. corosync really needs its own dedicated 1 Gbps link with as few hops as possible to be happy - with a 5x node cluster.
what do I run at home? in the end, I decided against HyperConvergence in general mainly because of the heat generation from three to five nodes I'd need to run, and the performance numbers (even with enterprise SSDs!) wasn't up to my requirements. i did setup a single-node Ceph server. But in the end, I couldn't get the IOPS where I wanted it.
oh, and tweaking Ceph is only done via "de-compiling", "building", and then "compiling" the config file back. there's zero GUI for it.
This is an awesome video.
From what I understand, that third "vote" is commonly called a "witness"
Hahaha you killed me with "I've got got a windows 10 doing what windows 10 likes to do: updating"
love this idea - to better this setup you want i7 refurbs with nvme and faster networks - cheap nodes - why is this so interesting for smb build 4 for quorum and then test for scaling with load balancing - biz wants cheap nodes but they can just spin up vms if they want to scale temp - the faster io and networks will work well- nvme scales - biz wants iops
This is a good video 👍
edit: Since the laptops were low cost, would be *great* to see this same video but updated with, say, 9 nodes.
Great video buddie!! Regards from Argentina 👋
Keep going!! 🙌💪🫡
So humble.
Thank you.
can these be bundled into one SUPERCOMPUTER? with like 500 ghz and 300 gb of ram?
Would be interesting to revisit the idea with three properly working laptops and CEPH. I have three identical HP Elitedesks with CEPH and works flawlessly. I have not made extensive tests with different VMs and how they survive the HA. I know my Ubuntu VM works and that is good enough for me.
1:09 To solve this problem, why not create a virtual machine on both devices? That way, if one laptop fails, Proxmox (I assume) thinks that 2 PCs (the host computer and the VM running on it) vote the same thing. I guess you could install a Linux VM on both to save resources. You would (possibly) still have high availability without the need for a third system (a third physical device). I haven't confirmed any of this myself, it's just a theory, but I see no reason why it can't be tried.
I tried many times and failed to build a cluster of old computers. But I succeeded in doing the opposite without virtualization, running 3 users at the same time on 1 windows os. software: ASTER
I’m just learning about home networking, and I have a few mini PCs that I’m trying to see what I can use them for. Is there any point it setting up a proxmox cluster for a home network?
I have a windows 10 VM that does what it loves to do! 🤣 it's updating. ❤️
What about make a cheap FreeNAS or TrueNAS with ZFS and multiple SSDs as shared storage for the ProxMox laptop cluster ??
Thx for the video.
Question - if I setup a second PVE server but will not create a cluster, will I have ability to migrate/copy VMs ?
Nope you need to make a cluster to move VMs between nodes easily. You could without a cluster setup the same storage share with backups and backup and restore VMs, but it will result in more downtime.
1. Does it increases speed?
2. How did u connect those two laptops pls show.
I have 20 old dual core PCs in my lab , how can I parallely use these processor.
Thank you
Thank you very much for this video!
Thanks! Really good info
Question: Would shared storage as a torrent for peer redundency across each laptop work better in this low-budget situation?
I haven’t worked too much with torrents, but there are similar concepts being used in software like lizardfs. The data is split into chunks and then a level of redundancy is set for files and extra copies are made if one is lost. Distributed storage though is normally a pretty bad experience on 1gbe as the network latency will significantly limit up performance.
@@ElectronicsWizardryin qrmu lvm is it possible to rmulate 10th gen intel cpu on an 8th gen host model?
@@jirehla-ab1671 I don't think there is any way to do that easily. You may be able to do it in qemu, but it would be emulating those instructions and much slower than traditional virtualization.
Awesome video!
Nice vid. I'm building a similar PM cluster. However, I'm unable to get one of the machines to join the cluster. QQ: Do all the nodes in a cluster need to be on the same version of Proxmox? That's not the case in this home.
Its suggested to have all the systems on the save version if possible, so Id upgrade the last system if possible.
So I have this idea of buying couple g15 motherboards and putting them into cluster and having a decent monster...can I do that?
Yea that should be possible. I'd love to see what the project looks like when finished. A few things that might help you use the boards better is faster networking as the 1gbe on most laptops is slow for clusters, and more ram which helps for caching and other uses.
@@ElectronicsWizardry what bandwidth you recommend? Can a switch solve onboard slow Ethernet ports?
@@Svaraijt A switch normally connects to an existing ethernet port, so it likely won't help here. But you can get faster USB 2.5 and 5GBE network cards, and if thunderbolt is a option that should be able to do 10GBe speeds or faster.
Great video!
u got a beautiful jawline ong
Hello sir ,can i make cluster for gaming or rendering?
i mean you could but i wouldn't suggest it. even if you can technically pool together the resources of multiple PCs to work on the same task in parallel, Game engines are notorious for not benefiting from simply adding more cores to split the workload. Usually adding anything beyond 8 cores has deminishing returns in terms of gaming, and thats first assuming the Game engine can even make use of all the cores. it isnt uncommon at all for games to offload most of the work on to the cores unevenly, they will pick and choose which cores to use and leave the rest barely used. This is why people do not build gaming rigs with Xeon and Eypc server CPUs even if they have tons more compute power than something like a Core i7 or i9 these CPUs will achieve greater FPS even having only a fraction of the cores, because Games typically respond better to just faster clock speeds, than just being able to split the workload.
And in a cluster all this data has to also travel through ethernet, even with 10gbe networking between each PC that's nearly 300 times slower than how fast data transfers within a single modern system. So there is a ton of latency when it comes to rendering, it's not a smooth experience. A channel called NetworkChuck was able to cluster a bunch of raspberry Pi4s and play minecraft on them, sort of.
As for the second part I guess it depends on what you're rendering.
Can I run COMSOL using cluster system on 3 laptops?
very interesting, thanks!
A cluster of high availability laptops, who knew lol
I have a bunch of older laptops but this doesn't really seem worth it
How did you get the wifi to work on the laptops? i cant seem to get it to connect to my network.
In this video I think I only used wired connections. I may go over WiFi on proxmox in a future video. Look up guides for WiFi in Debian for steps that would also work in proxmox.
@@ElectronicsWizardry wow thanks for the reply on an old video! I can't get network with wifi or USB ethernet. So I look forward to the video!
i have an old laptop and a PC, can i connect the two together to combine their power, if so how?
There isn’t a way to make it ack like one faster computer but if you have multiple VMs you can make a cluster and split the VMs and containers between the systems.
@@ElectronicsWizardry oh alr thanks for answering
Also can it work with different laptop have different versions
Could you use more than 3?
Yup. I don't believe there is limit to the number of nodes in a proxmox cluster.
I have 4 core2duo pc with only @4gb each. Is it enough for running proxmox cluster with 2 win10 vm and file sharing
It may be technically possible but I’d say that hardware is too slow for a cluster. I’d probably set it up with the file share on the host and no vms but it really depends on the exact software setup you're using.
can I create cluster for proxmox with running machines?
To made a cluster all the additional modes should have no vms on them. So you may need to backup vms, add the system to the cluster then restore vms
@@ElectronicsWizardry what about the main node?
The main node can have VMs and they will stay on a cluster when other nodes are added
I am going to try to do this with one server and 20 Chromebook’s and one dell oplix 9020 for a speed nas it was hard to get proxmox on the Chromebooks
Update?
@@NickSolo420 it works
@@1destroyer100its good? Or nah
@@ElaraArale yea proxmox is great and it runs great
06:34
windows 10 do what windows 10 love to do
windows 10: blue screan
Core 2 Duo laptops?
Nah.
Don't revive them. Don't use Proxmox on them.
It's not power efficient. Proxmox will crash on it, especially early model that adapt 64 bit.
How do i get proxmox to stay on when laptop lid is closed??
I forget exactly where (Google will have the answer) but there are 3 lines in the config file you need to change to disable the laptop from turning off when the lid is closed. I did it on my systems.
why is this dude is staring at the ceiling the whole time
is there a spider
a cluster of spiders
What if it doesnt have virtualization
If you don't have virtualization your mostly limited to containers. You can run vms via emulation, but that will be slow and if thebcou doesn't support virtualization is probably pretty slow cpu.
how i create any vm ip private akses whit one ip public
If he is having trouble with this, all I think is DONT TRY THIS AT HOME
You Sir are a raging psychopath, very good explained and set up :D
Hi,
Can I use a laptop cluster for crypto mining
Thanks for the info, you talk about interesting topics....gets a little hard to watch all the way through.
constructive criticism
- focus on keeping your head level.
- select words that you're able to say clearly.
Just being picky, but when you're talking, you're looking at something else - a little disorienting. Love your content, though. Brand new subscriber.
Seriously?
why are you showing your face more than your screen😢