It's pretty cool, but what do you use for persistent data storage? Longhorn or NFS. I find that the storage part is too often set aside when this is the most important thing. The distributed storage is top, except that on raspberry with a single ethernet it is normally not enough. So I wouldn't be interested to know what you're putting in place
Completely agree. Im homelab you need to think about computing power as well as electricity bills, especially in middle Europe. I just kicked longhorn out because it failed on idle power consumption as well as with rebuild time on segmentation issues. Resiliency is not everything.
@@guillaumearchambault905 I also agree 100% persistent data storage seems to be ignored often in video’s and guides; without it a Kubernetes cluster is nothing but a bunch of clustered standalone nodes!
@@tomekem3473 for now openebs, but I'm still testing few things. Longhorn is about to release 1.7, Ill check it again if it can handle idle state correctly, this was broken about v1.4.
What an amazing video! You inspired me to finally get my own home-lab after thinking about it on and off for a couple of years, so thank you for being such a great inspiration.
Thank Lempa for putting so much effort into creating and sharing this video with everyone. This video provides a wealth of information for Raspberry Pi cluster project users and offers strong support for Waveshare POE HAT customers in their practical applications. I believe Waveshare should sponsor you with free products. If you have any ideas about incorporating other new Waveshare products in your future videos, I will certainly provide you with the greatest support!
@@christianlempa Thank you for letting us know! We believe your independent educational videos will provide our enthusiasts with increasingly objective and valuable information! We all really need you!
Thank you for your sharing. You are using our 52pi N04 expansion board, your idea is fantastic. I also want to try to build a similar environment on our Rack mate T1 and give K8S a try, it looks very promising! You are absolutely right, everyone should learn Linux!
Great video, i installed my cluster without kube vip because i was installing it via my own install script instead of k3sup and it wasnt easy, will need to give this a shot
Very nice. I like the idea and simplicity of building your cluster on bare metal. Feels more approachable then starting with a hypervisor. I hope you take the same approach with storage and keep the cluster completely self-contained so you could take that RackNex unit with you and have a portable cluster. Are you looking at file systems like Gluster or Ceph?
Great video. May I ask what advantage does running k3s on three Pis have over running on Proxmox? If it is power efficiency, wouldn't Proxmox on Mini PCs be better? While I was looking at building a k3s cluster, getting a pi, the hats and ssd all combined became more expensive and more finicky than getting a used Lenovo/HP/Dell mini PCs. I ended up going with Mini PCs + Proxmox and that is very power efficient and is a lot more flexible than the PIs
Thank you! :) It was just a fun experiment to tinker, and I wanted to have an environment that is a bit more power efficient than my bigger proxmox server, so I can shut it down when I don't need it and still run my prod services on my kube cluster
I think that you don't need to use kube-vip if you have opted for the cilium cni instead of flannel, since that is quite rich in feature when it comes to l2, service mesh, kube-proxy replacement and more. Maybe perfect for a follow up video to make that switch ^_^ other than that thanks, a great video as usual!
Great video, Christian! One question, if the kube-vip address is virtual then how do you reserve it on your dhcp server so it doesn't get allocated to another machine in your network?
You can use different ip ranges for dynamic and static ip addresses. Most dhcp servers have settings to exclude specific IPs from the dynamic range as well
Hi Christian, any chance of doing an install howto for teleport (community edition) for a homelab, I feel this is more advanced than I would like, thank you, love your videos very insightful
This is nice but unfortunately some images do not run on the raspi architecture. I do not remember by hard which was it in my case exactle I believe timescaleDB.
Hey Cristian, great video! Wanted to try out Kubernetes myself but never really get the hang on it but I am eager to try it out with mini PCs I have lying around. :D One side question, which is not dedicated to the video itself: In some other videos you uploaded, I saw a metal plate you screwed on the wall and hung your (for example) Zima Board onto it - I never found such a metal plate; what's this thing called, where did you get it? You did not mention it anywhere on your Kit page, either. Keep up the great work, you helped me and my projects many times! Best.
Hello! Great video. I love K3s too. Are your Fitwok Nvme still running ok? My Pi5 completely destroyed my SD card after running fro 15 days with Ubuntu server and K3s and I'm thinking of moving my single node k3s to nvme instead of wasting money with a better SD.
I attempted this last year but docs werent great may have to try again. Im still unsure what youre using for storage. I need something thats not gonna go down
only .kube/config was updated on your workstation/laptop but not only your kubectl communicate with kube api, another nodes also communicate with kube api, what still on all another confs ip of first node. 27:09 you add second node using ip of first node, not vip I bet, if you turnoff first node, you will got problems with your cluster
wow, that's really good raspberry pi setup on steroids! Compared to my cluster your's is super awesome! I own 4 raspberry pi's on my cluster, but they are 4'th gen version. I'm using uctronics rackmount for raspberry pi. It was only reasonable rack solution at that time ;-) Every node is installed on SD 64GB cards. I've built it 2 or 3 years ago. What i can share with you Christian, consider adding one or two amd64 vm's to this cluster. Not every app is working well on arm. Kubernetes with proper nodeSelector configuration will do the job, when you'll encounter some problems with ARM architecture ;-) Many thanks for tip with kube-vip solution. It's one of the crucial missing pieces in my homelab setup.
Trying to find with kube-vip can/does it do any kind of health checks of the endpoints? For example, with keepalived and haproxy (or nginx as a load balancer) and configure health checks to know if x failures occur remove the load from the lb unless y successes occur? From what I've seen so far with kube-vip that if say the k3s on a master died but is still on, would the request then just fail if that node has the vip? or would it move to another node? or if the vip is on node2, but k3s svc on node1 is dead would traffic get sent there?
Hey Christian, love your work and all the great content you make. 1 nitpicky thing though, the font on those diagrams in the beginning and elsewhere in the video is horrendous. Completely unreachable without zooming in (using mobile to watch this). Obviously, nothing overly important just a suggestion to use a standard font in the future. PLEASE keep making this content, it is a huge motivation to me and others to explore areas outside our respective fields and there is always great stuff to try out. Thank you for all the hard work!
So I am 90% there, I tried with k3sup and just plain old k3s docs and `--disable servicelb` doesn't seem to work because the first time I create a deployment (nginx) and expose it with an lb, I get 6 pods beginning with `svclb-nginx -` all running in my cluster with an ip address of the host node.
The Raspberry Pi uses arm, so some applications and services cannot be deployed... You have to write your own yaml /docker compose file for deployment, which is a bit troublesome. Thank you boss for sharing. By the way, why didn’t you consider using Argon NEO? 🤣
Keep in mind that is the control plane, worker nodes (agents) could be deployed on other pi's (with same or different os), x86 or x64 physical hardware (with various os'es), and/or proxmox vm's. Then using labels (for example on the nodes) could then define which apps would go where based on the definition in the yaml (by os, platform or other label) . Also, if it's your own app(s) then you could do multi platform builds.
I still do not truly don't understand why ppl are choosing PI5 instead of mini PC (for eg on N100) ... mini PC have much faster CPU ... more options about memory ... more options about storage ... nower days you can connect nvme without any problems ... where you need some extra super puper adapter to connect any hdd or ssd to PI5
For some applications small size, low power draw and the already integrated IO options for cameras and other stuff are important enough to make the RPi a favorable choice. There is a reason why Raspberry Pi's are produced (and sold) in fairly large numbers, i.e. it's not just hobbyists who buy this stuff. In the past at least, price was also a major reason why the RPI was so popular. Obviously, there are also many other applications where a RPI is not the best solution (like you said).
@@itssoaztek4592 the key is here "for some" ... if we speak about regular serving like kuba ... its not about RPi ... yes we can up some kind of "hello world" web site but we still will have problems with storage, cooling and etc. Many years I tried to serve Home Assistant on RPi and use the same RPi to host GUI on external touch screen ... an it was awful time ... Ive finally switched to mini PC from GMK on Intel cpu and it works much better and flawlessly if we compare it with RPi. I also have some exp with multiple mini PC from Asrock to test cluster and it was muuuch better than RPi setup
This is short story how to start with tiny pi and convert it into big and expensive, problematic build still limited to just 1Gbit :/ You are able to stick 16x pi on 2u case. You could use RK3588 alternatives to get about 2x more cpu power, 2.5x faster ethernet as well as 4x faster nvme (or 10Gbit ethernet) and up to 4x more RAM. Same with any n100 board. This is expensive lesson. Raspberry lost its fame for beeing affordable and powerfull choice. Have You mentioned final price for unit/whole build? Have You measured power consumption for all units on idle and full load?
The main goal was a bare metal kubernetes and play around with rpi. Sure you might find some better options but as this is the first project of this kind, I wanted to go with something that is well supported and where you find enough content and help
I would be interested in IPv6 Dual Stack and how to expose a service using IPv6 Loadbalancer. Can kube-vip do that? Not sure if it is a Talos problem, but I failed to get IPv6 communication between nodes working using ULAs. I used the default Flannel CNI.
@@christianlempa There is definitely a shortage of IPv6 content out there and more content on it would definitely help! I changed the CNI to Cilium and then IPv6 with ULAs started working. I could also replace MetalLB with Cilium IP Pools and BGP control plane.
It's pretty cool, but what do you use for persistent data storage? Longhorn or NFS. I find that the storage part is too often set aside when this is the most important thing. The distributed storage is top, except that on raspberry with a single ethernet it is normally not enough. So I wouldn't be interested to know what you're putting in place
Completely agree.
Im homelab you need to think about computing power as well as electricity bills, especially in middle Europe.
I just kicked longhorn out because it failed on idle power consumption as well as with rebuild time on segmentation issues. Resiliency is not everything.
@@guillaumearchambault905 I also agree 100% persistent data storage seems to be ignored often in video’s and guides; without it a Kubernetes cluster is nothing but a bunch of clustered standalone nodes!
@dmcrkrk And what have you used instead of Longhorn?
@@tomekem3473 for now openebs, but I'm still testing few things. Longhorn is about to release 1.7, Ill check it again if it can handle idle state correctly, this was broken about v1.4.
Im currently testing longhorn and give it a try again! Follow up video is coming at some point ;)
What an amazing video! You inspired me to finally get my own home-lab after thinking about it on and off for a couple of years, so thank you for being such a great inspiration.
Oh, thank you so much for the kind words! And welcome on board :D
That's probably one of the most amazing videos which I have viewed so far! Thanks a lot Chris for explaining so many things in detail!
Thank you so much for the kind words 🙏
Thank Lempa for putting so much effort into creating and sharing this video with everyone. This video provides a wealth of information for Raspberry Pi cluster project users and offers strong support for Waveshare POE HAT customers in their practical applications. I believe Waveshare should sponsor you with free products. If you have any ideas about incorporating other new Waveshare products in your future videos, I will certainly provide you with the greatest support!
Thank you so much 😊 but I’m fine to pay for stuff when I got a use for it, so it’s okay for me
@@christianlempa Thank you for letting us know! We believe your independent educational videos will provide our enthusiasts with increasingly objective and valuable information! We all really need you!
@@Waveshare_Ruan thank you! :)
Racknex is awesome. Thank u for this info!
It really is!
They even have solutions for the new Mac mini.
Got a racknex mount for the Vigor 165 DSL Modem. Was the only one but its great
Nice!
Thank you for your sharing. You are using our 52pi N04 expansion board, your idea is fantastic. I also want to try to build a similar environment on our Rack mate T1 and give K8S a try, it looks very promising! You are absolutely right, everyone should learn Linux!
Thank you 😊 great feedback!
Hi, thanks for the tutorial, little tips from my side to disable traefik + servicelb you can use --no-extras. it work every time
Oh I have to try that! Thank you 🙏
@@christianlempa Another things the --no-extras should be add on the join command as mention on the github.
Instead, metal lb can also provide a virtual IP for the load balancer claster k8s
For me CNIs worked very well too with onPrem Setups. Very interesting tutorial and so well made!!
3:33 oh my goodness. This is a MASSIVE improvement to the k3s documentation since the last time I deployed a net-new cluster.
True!
Great video, i installed my cluster without kube vip because i was installing it via my own install script instead of k3sup and it wasnt easy, will need to give this a shot
Nice! Hope the video helps :)
Very nice. I like the idea and simplicity of building your cluster on bare metal. Feels more approachable then starting with a hypervisor. I hope you take the same approach with storage and keep the cluster completely self-contained so you could take that RackNex unit with you and have a portable cluster. Are you looking at file systems like Gluster or Ceph?
Im currently looking at longhorn again, maybe it’s working better this time! :)
Great video. May I ask what advantage does running k3s on three Pis have over running on Proxmox? If it is power efficiency, wouldn't Proxmox on Mini PCs be better?
While I was looking at building a k3s cluster, getting a pi, the hats and ssd all combined became more expensive and more finicky than getting a used Lenovo/HP/Dell mini PCs.
I ended up going with Mini PCs + Proxmox and that is very power efficient and is a lot more flexible than the PIs
Thank you! :) It was just a fun experiment to tinker, and I wanted to have an environment that is a bit more power efficient than my bigger proxmox server, so I can shut it down when I don't need it and still run my prod services on my kube cluster
I think that you don't need to use kube-vip if you have opted for the cilium cni instead of flannel, since that is quite rich in feature when it comes to l2, service mesh, kube-proxy replacement and more. Maybe perfect for a follow up video to make that switch ^_^ other than that thanks, a great video as usual!
Thanks! I don’t have much experience with cilium but I agree it would be another interesting topic!
thanks mate, you always fantastic .
So nice of you :)
Great video, Christian! One question, if the kube-vip address is virtual then how do you reserve it on your dhcp server so it doesn't get allocated to another machine in your network?
You can use different ip ranges for dynamic and static ip addresses. Most dhcp servers have settings to exclude specific IPs from the dynamic range as well
thank you been waiting for this.
🥰🙏
Are you sure you bought the sbc-213 model? Because it says sbc-202 on your front panel?
Hi Christian, any chance of doing an install howto for teleport (community edition) for a homelab, I feel this is more advanced than I would like, thank you, love your videos very insightful
Maybe this video helps: Installing Teleport + Traefik (Letsencrypt TLS certs)
th-cam.com/video/NzSdNoR-JPo/w-d-xo.html :)
This is nice but unfortunately some images do not run on the raspi architecture. I do not remember by hard which was it in my case exactle I believe timescaleDB.
Hey Cristian, great video! Wanted to try out Kubernetes myself but never really get the hang on it but I am eager to try it out with mini PCs I have lying around. :D
One side question, which is not dedicated to the video itself: In some other videos you uploaded, I saw a metal plate you screwed on the wall and hung your (for example) Zima Board onto it - I never found such a metal plate; what's this thing called, where did you get it? You did not mention it anywhere on your Kit page, either.
Keep up the great work, you helped me and my projects many times!
Best.
Hey thank you! Good luck with your project :) honestly I don’t remember that plate anymore it was just a random piece I found on Amazon :D
@@christianlempa That is sad to hear. ;D Thanks for the quick answer. :)
Hello! Great video. I love K3s too. Are your Fitwok Nvme still running ok? My Pi5 completely destroyed my SD card after running fro 15 days with Ubuntu server and K3s and I'm thinking of moving my single node k3s to nvme instead of wasting money with a better SD.
So far it's running fine :)
I attempted this last year but docs werent great may have to try again. Im still unsure what youre using for storage. I need something thats not gonna go down
Do you mean kube-vip? or k3sup?
k3s has a yellow background so you can install it in work and you boss thinks you're install ESXi?
Can you post where you got the PCIe cables? Definitely interested in doing something like this.
They’re on my kit page: christianlempa.de/kit
only .kube/config was updated on your workstation/laptop
but not only your kubectl communicate with kube api, another nodes also communicate with kube api, what still on all another confs ip of first node.
27:09 you add second node using ip of first node, not vip
I bet, if you turnoff first node, you will got problems with your cluster
Hmm that’s right! Thanks for the heads-up! I might need to change the config later at some point
wow, that's really good raspberry pi setup on steroids! Compared to my cluster your's is super awesome! I own 4 raspberry pi's on my cluster, but they are 4'th gen version. I'm using uctronics rackmount for raspberry pi. It was only reasonable rack solution at that time ;-) Every node is installed on SD 64GB cards. I've built it 2 or 3 years ago. What i can share with you Christian, consider adding one or two amd64 vm's to this cluster. Not every app is working well on arm. Kubernetes with proper nodeSelector configuration will do the job, when you'll encounter some problems with ARM architecture ;-)
Many thanks for tip with kube-vip solution. It's one of the crucial missing pieces in my homelab setup.
Thank you so much! Also great tip, I will think about that :)
Trying to find with kube-vip can/does it do any kind of health checks of the endpoints? For example, with keepalived and haproxy (or nginx as a load balancer) and configure health checks to know if x failures occur remove the load from the lb unless y successes occur? From what I've seen so far with kube-vip that if say the k3s on a master died but is still on, would the request then just fail if that node has the vip? or would it move to another node? or if the vip is on node2, but k3s svc on node1 is dead would traffic get sent there?
I think does but you might need to check in the docs how exactly that’s working
Great content! It would be nice if you could review some technologies like OKD, is the community edition of Openshift.
Thank you 🙏
Hey Christian, love your work and all the great content you make. 1 nitpicky thing though, the font on those diagrams in the beginning and elsewhere in the video is horrendous. Completely unreachable without zooming in (using mobile to watch this). Obviously, nothing overly important just a suggestion to use a standard font in the future. PLEASE keep making this content, it is a huge motivation to me and others to explore areas outside our respective fields and there is always great stuff to try out. Thank you for all the hard work!
Thank you! I’ll keep this in mind for the next one 😊☝️
So I am 90% there, I tried with k3sup and just plain old k3s docs and `--disable servicelb` doesn't seem to work because the first time I create a deployment (nginx) and expose it with an lb, I get 6 pods beginning with `svclb-nginx -` all running in my cluster with an ip address of the host node.
Maybe we can help you on the discord, we have a help forum :)
How do you renew the certificate when its failing on you, ie after one year?
I just download it from the k3s kubeconfig directory.
Why not use servicelb shipped with k3s?
If you need virtual address instead of fixed port numbers then You need something like this or metallb.
Great video
Thanks!
The Raspberry Pi uses arm, so some applications and services cannot be deployed...
You have to write your own yaml /docker compose file for deployment, which is a bit troublesome.
Thank you boss for sharing.
By the way, why didn’t you consider using Argon NEO? 🤣
Keep in mind that is the control plane, worker nodes (agents) could be deployed on other pi's (with same or different os), x86 or x64 physical hardware (with various os'es), and/or proxmox vm's. Then using labels (for example on the nodes) could then define which apps would go where based on the definition in the yaml (by os, platform or other label) . Also, if it's your own app(s) then you could do multi platform builds.
Its such a long time ago, I have encountered an application that I could not run in my K3S arm64 cluster. Basically support for arm is great now
ARM is not problematic now. Risc-V is coming, and still you can mix all of them together.
does gitlab support arm64 processos?
Good question, I hope it does, but if it doesn't, I could add more x86 nodes to the cluster, we'll see :)
R.I.P. srv-prod-3... you've done well. xD
😂🤫
On your videos I am hearing a clicking sound and some static. Love your content btw. :)
Thanks! But I don't know what you mean by clicking or static?
Damn, that's a lot of work with the hardware... But, it could be useful. Thanks for the detailed video!
Thank you so much 🥰
Hi! Is it possible to verify my account in Christian discord without phone? Discord still does not send me verification sms. Tried 5 times.
Unfortunately not, that hold spam and scam away from the channel
Talos OS en Talos Omni, a Video about that would be cool anyway!
Maybe somewhere next year I’ll give it a try again :)
I still do not truly don't understand why ppl are choosing PI5 instead of mini PC (for eg on N100) ... mini PC have much faster CPU ... more options about memory ... more options about storage ... nower days you can connect nvme without any problems ... where you need some extra super puper adapter to connect any hdd or ssd to PI5
For some applications small size, low power draw and the already integrated IO options for cameras and other stuff are important enough to make the RPi a favorable choice. There is a reason why Raspberry Pi's are produced (and sold) in fairly large numbers, i.e. it's not just hobbyists who buy this stuff. In the past at least, price was also a major reason why the RPI was so popular. Obviously, there are also many other applications where a RPI is not the best solution (like you said).
@@itssoaztek4592 the key is here "for some" ... if we speak about regular serving like kuba ... its not about RPi ... yes we can up some kind of "hello world" web site but we still will have problems with storage, cooling and etc. Many years I tried to serve Home Assistant on RPi and use the same RPi to host GUI on external touch screen ... an it was awful time ... Ive finally switched to mini PC from GMK on Intel cpu and it works much better and flawlessly if we compare it with RPi. I also have some exp with multiple mini PC from Asrock to test cluster and it was muuuch better than RPi setup
Can you name a "mini PC" which runs with PoE?
@@christophschug9295Minisforum S100, Intel n100 based miniPC designed to run off of PoE
@@christophschug9295 Some mini pcs can have several power outputs via the same main unit
Use Talos 😉
Agree
Talos is just next level, talos over iPXE :)
@@weirdscience-o1i There is nothing more straightforward building an K8s cluster than with Talos Omni!
@@allards Amen to that 🥳
Talos is amazing
First?
Second :)
This is short story how to start with tiny pi and convert it into big and expensive, problematic build still limited to just 1Gbit :/
You are able to stick 16x pi on 2u case. You could use RK3588 alternatives to get about 2x more cpu power, 2.5x faster ethernet as well as 4x faster nvme (or 10Gbit ethernet) and up to 4x more RAM. Same with any n100 board.
This is expensive lesson. Raspberry lost its fame for beeing affordable and powerfull choice. Have You mentioned final price for unit/whole build? Have You measured power consumption for all units on idle and full load?
I was thinking nearly the same thing.
The main goal was a bare metal kubernetes and play around with rpi. Sure you might find some better options but as this is the first project of this kind, I wanted to go with something that is well supported and where you find enough content and help
I would be interested in IPv6 Dual Stack and how to expose a service using IPv6 Loadbalancer. Can kube-vip do that? Not sure if it is a Talos problem, but I failed to get IPv6 communication between nodes working using ULAs. I used the default Flannel CNI.
No idea! IPv6 is a topic I’d like to tackle next year
@@christianlempa There is definitely a shortage of IPv6 content out there and more content on it would definitely help!
I changed the CNI to Cilium and then IPv6 with ULAs started working. I could also replace MetalLB with Cilium IP Pools and BGP control plane.