K3s + Rancher + Longhorn is a great setup. I was able to make a test K3s cluster watching your videos and Jim's Garage. I have learned so much from your videos, they are so well done.
Excellent video, now I know better how Longhorn works. K8 is so complicated and over engineered that I think my vSphere install is 100 times easier to work with even with vSAN and NSX :)
Absolutely! :D But it's fascinating to learn, and once you get through the painful beginning, you start to appreciate the benefits! Thank you so much for the feedback!
Hey Christian, since I've had many problems with longhorn, mainly perfomance and kubernetes upgrade (evicting volumes to other nodes) related, I found the drbd operator called piraeus. In my opinion it is vastly superior and makes use of proven tech like drbd for replication and either lvm or zfs for the backend storage. I hope you can give it a try and maybe compare the two. Piraeus could really use more attention as it is pretty easy to use as well and much faster.
Really love the idea of Longhorn as it's so easy to setup. But for me i ran into quota issues. That the volumes were out of space even though they were not. Apprantly when working with WAL files like prometheus uses the Used diskspace on disk and in longhorn differs by quite a lot due to the "Cleanup" Not replicating to the disk usage in longhorn. Which for me was a 2 GB metrics file that took up a 50GB Longhorn PV. and Prometheus crashed due to "Not enough diskspace. Now using Rook-Ceph for that reason.
@burads yeah I've had a number of other issues as well that I did not mention. Like problems with volume replication lagging behind and replicas failing at times. Also on bigger volumes draining nodes is nearly impossible and takes ages and resyncing failed replicas always resulted in the full volume having to be copied again, which is really annoying.
Hi Christian, I just (force-) uninstalled Longhorn from my cluster. It consumes way too much CPU resources heating my office which is kind of nice for the season. But honestly: I only provisioned a single volume which was not actively used and my CPUs showed an astonishing workload.
Longhorn requires a couple of system containers for all the functions to work which is kind of a big overhead ( even for just a single volume 😁) but it is great 👍
Great video! I have been using longhorn on my RKE2 cluster for a while. The main thing I am struggling to do is an entire cluster restore. When I do the cluster restore it creates new volumes before I can even restore the backups, so I end up having to stop the multiple services I have and do it like you showed in the video. This isn't difficult but in a DR scenario I am looking for speed and simplicity while minimizing places for human error. With restoring the etcd database, the PVC claim information should remain the same. Perhaps longhorn doesn't have a built in method for this. Any advice would be appreciated!
So, in a Kubernetes cluster, is it possible to deploy a highly available PostgreSQL service with three instances using Longhorn for volume replication and without using database replication at the PostgreSQL level? Thanks a lot for the video!
You are the best ! Perfect guide ! One question about Longhorn: We have 5 nodes (1 manger, 4 worker). Our longhorn.storageClass.settings.replicasCount = 1. Start Postgres on node = 3. All is okay. In longhorn UI - volume-replica = node-2, node-3. Okay. How can we move replica from node-2 -> node-4 (For example) ?
Thank you so much! Good question, I haven't done it myself, but from my understanding you can increase the number of replicas of that volume, so it gets replicated on node-4, and after that reduce the number again and delete it from node-2.
Hey Christian, what does Longhorn use in backend for storage? Is it ISCSI, NFS, NBD, ... ? It seems like it creates three big files, one on each node, exposes it over ISCSI, creates three-way raid1 md device, creates filesystem on top of it and then mounts it on host that hosts the container.
Hey! Great question, from my understanding, it is leveraging its own built-in mechanisms rather than traditional protocols like iSCSI or NFS, but I'm not sure.
What if I have a single-node cluster? I'd like it to be as simple as can be, but have e.g. ZFS volumes or datasets be created in the ZFS storage on my Proxmox host. I am trying my luck with proxmox-csi-plugin, but I don't think I am happy with it...
So what if you also have proxmox ceph and running kubernetes? I like to apply both best practices for HA and have some redundancy. What if you run K8s on top of Promox with ceph and longhorn..? Doesn't that sound very very redundant? What is your architecture advice here?
That's a good question, and honestly, I'm not sure what I would do here. I think it doesn't make much sense to replicate the volumes when the whole disk is replicated in Ceph. I need some time to think about it and play around with Ceph.
Used Longhorn quite a bit in a homelab, and had/have nothing but a problems with it, in any scenario that involves more than few GB of data. Too lazy to switch to something like OpenEBS, CEPH, DirectPV or Piraeus at the moment though 🙂 Basically, anything usable rather than Longhorn.
@@christianlempa it could be, but the fact remains that it handles those situations poorly. Perhaps also my lack of in-depth knowledge about the system plays a role. Never the less, both their docs, and their resilience could be better 😀
Under the hood its the same tech. K8s is a management layer on top for very large distributed apps, cool tech. But most businesses don't ever need it. Wait a few years until the videos come with the title "you probably don't need k8s" after the hype dies down (which is already happening). Edit; not to write off the tutorial christian, very thorough and well explained! If you think or want to work in mega scale services, this is valuable.
K3s + Rancher + Longhorn is a great setup.
I was able to make a test K3s cluster watching your videos and Jim's Garage.
I have learned so much from your videos, they are so well done.
That is so nice! Thank you so much for the feedback :)
Excellent video, now I know better how Longhorn works.
K8 is so complicated and over engineered that I think my vSphere install is 100 times easier to work with even with vSAN and NSX :)
Absolutely! :D But it's fascinating to learn, and once you get through the painful beginning, you start to appreciate the benefits! Thank you so much for the feedback!
excellent tutorial as always.. looking forward to the upcoming traefik and certmanager tutorial
Thank you so much! 😊 it’s almost done so I’m gonna expect it to publish next week
@@christianlempa I can't wait. ;)))
This is great! Thanks for taking the time to make such valuable and comprehensive content 👍
Glad you enjoyed it!
Hey Christian,
since I've had many problems with longhorn, mainly perfomance and kubernetes upgrade (evicting volumes to other nodes) related, I found the drbd operator called piraeus. In my opinion it is vastly superior and makes use of proven tech like drbd for replication and either lvm or zfs for the backend storage. I hope you can give it a try and maybe compare the two. Piraeus could really use more attention as it is pretty easy to use as well and much faster.
Really love the idea of Longhorn as it's so easy to setup. But for me i ran into quota issues. That the volumes were out of space even though they were not. Apprantly when working with WAL files like prometheus uses the Used diskspace on disk and in longhorn differs by quite a lot due to the "Cleanup" Not replicating to the disk usage in longhorn. Which for me was a 2 GB metrics file that took up a 50GB Longhorn PV. and Prometheus crashed due to "Not enough diskspace.
Now using Rook-Ceph for that reason.
@burads yeah I've had a number of other issues as well that I did not mention. Like problems with volume replication lagging behind and replicas failing at times. Also on bigger volumes draining nodes is nearly impossible and takes ages and resyncing failed replicas always resulted in the full volume having to be copied again, which is really annoying.
But maybe they fixed a couple of those issues. Though performance is always going to be bad because of the architecture
Really loved the video. I recently deployed longhorn in my cluster and this video made me realize how many mistakes I made. 😅
Thank you so much! :)
Hey @Christian, love your videos, always super informative and entertaining!
Thank you so much! Glad you enjoy them :)
Hi Christian, I just (force-) uninstalled Longhorn from my cluster. It consumes way too much CPU resources heating my office which is kind of nice for the season. But honestly: I only provisioned a single volume which was not actively used and my CPUs showed an astonishing workload.
Longhorn requires a couple of system containers for all the functions to work which is kind of a big overhead ( even for just a single volume 😁) but it is great 👍
Excellent tutorial, wonderful explanation as always
Thank you so much! 😍
Great Video, especially for the backup / restore part and the example with a helm chart!
Thank you! :)
I would be interested in a video about using Velero for backups in combination with Longhorn!
Great vid :)
Thanks! Never heard of it, but I will check it out ;)
Using Longhorn in kubernetes since 4 years doing better in every release
Nice!
12:00 install add-on on Chrome or safari? You mean firefox or a firefox fork, right?
No firefox sucks ;)
I am super surprised that longhorn is running without any issue on raspberry PIs 🤔 In the past I get so many cluster issues due to low resources.
Maybe that was because of a slower Rpi or disk? I'm using RPi5 with 8GB and an NVME Hat.
@@christianlempa Probably. Glad to hear this work well. I hope you will share if you start facing some issues with the cluster.
@@MrToup Of course, I'll do! :)
Great video! I have been using longhorn on my RKE2 cluster for a while. The main thing I am struggling to do is an entire cluster restore. When I do the cluster restore it creates new volumes before I can even restore the backups, so I end up having to stop the multiple services I have and do it like you showed in the video. This isn't difficult but in a DR scenario I am looking for speed and simplicity while minimizing places for human error. With restoring the etcd database, the PVC claim information should remain the same. Perhaps longhorn doesn't have a built in method for this. Any advice would be appreciated!
Thank you so much! I haven't tested this yet, but that's something I have to look at at some point
So, in a Kubernetes cluster, is it possible to deploy a highly available PostgreSQL service with three instances using Longhorn for volume replication and without using database replication at the PostgreSQL level?
Thanks a lot for the video!
Yes it is possible. There’s even a Postgres provider that does all the complicated setup for you, but I haven’t tried it out yet
You are the best ! Perfect guide ! One question about Longhorn: We have 5 nodes (1 manger, 4 worker). Our longhorn.storageClass.settings.replicasCount = 1. Start Postgres on node = 3. All is okay. In longhorn UI - volume-replica = node-2, node-3. Okay. How can we move replica from node-2 -> node-4 (For example) ?
Thank you so much! Good question, I haven't done it myself, but from my understanding you can increase the number of replicas of that volume, so it gets replicated on node-4, and after that reduce the number again and delete it from node-2.
Hey Christian, what does Longhorn use in backend for storage? Is it ISCSI, NFS, NBD, ... ? It seems like it creates three big files, one on each node, exposes it over ISCSI, creates three-way raid1 md device, creates filesystem on top of it and then mounts it on host that hosts the container.
Hey! Great question, from my understanding, it is leveraging its own built-in mechanisms rather than traditional protocols like iSCSI or NFS, but I'm not sure.
Hello Christian I was just wondering what is the terminal client that you are using? Thanks
I'm using Warp, you also find a video on my channel about it :)
i like rook ceph, but longhorn seems nice too
Nice, maybe I'll check it out at some point.
What's the terminal app your are using?
Linstor vs DRBD from Linbit is also very powerful open source storage solution for not only kubernetes clusters.
What if I have a single-node cluster? I'd like it to be as simple as can be, but have e.g. ZFS volumes or datasets be created in the ZFS storage on my Proxmox host.
I am trying my luck with proxmox-csi-plugin, but I don't think I am happy with it...
Hm good question, I'm not sure if it works great for a single node cluster, it's recommended anyway to run a minimum of 3 nodes in Kubernetes.
So what if you also have proxmox ceph and running kubernetes? I like to apply both best practices for HA and have some redundancy. What if you run K8s on top of Promox with ceph and longhorn..? Doesn't that sound very very redundant? What is your architecture advice here?
Also backup remotely. .. 321 backup rule
That's a good question, and honestly, I'm not sure what I would do here. I think it doesn't make much sense to replicate the volumes when the whole disk is replicated in Ceph. I need some time to think about it and play around with Ceph.
Used Longhorn quite a bit in a homelab, and had/have nothing but a problems with it, in any scenario that involves more than few GB of data. Too lazy to switch to something like OpenEBS, CEPH, DirectPV or Piraeus at the moment though 🙂 Basically, anything usable rather than Longhorn.
Maybe the network connection and the disks are not fast enough to replicate that amount of data.
@@christianlempa it could be, but the fact remains that it handles those situations poorly. Perhaps also my lack of in-depth knowledge about the system plays a role. Never the less, both their docs, and their resilience could be better 😀
@@ivantomica I highly agree, they don't say much about the architecture details, how they do replication, etc.
excellent tutoriial! keep up the good work :)
May I ask what's the confguration of your shell environment ?
Thank you so much! There's a video about it: th-cam.com/video/NfggT5enF4o/w-d-xo.html
Do a tutorial about your terminal please.
There is: th-cam.com/video/NfggT5enF4o/w-d-xo.html
How do you delete longhorn when it's stuck deleting lol
It depends on which resources are stuck, maybe let's follow up on Discord ;)
Longhorn requires installation of components in the nodes. This makes Longhorn a complete non starter for me.
Gigabytes ✖
G B bites ✔
I hope I did it right :D
Isn't longhorn windows vista? Is this name even legal?
I don’t think it’s a tm :D
No, it is this big bovine creature. 😅
@@FredrikRambrisunderrated reply 🤣
Docker still better
For easy home setups. Yes
For fun and/or HA setups. No
Use whatever works for you tho
Under the hood its the same tech. K8s is a management layer on top for very large distributed apps, cool tech. But most businesses don't ever need it. Wait a few years until the videos come with the title "you probably don't need k8s" after the hype dies down (which is already happening). Edit; not to write off the tutorial christian, very thorough and well explained! If you think or want to work in mega scale services, this is valuable.
For small deployment yeah, wait until you need to manage multiple instances and manage system upgrades. Basic docker engine has limited capabilities