Hey Clemenko, thanks for the detailed explanation! I really like the combination of the written and video tutorial, makes it very easy to follow along.
Clemenko, thank you for everything you are doing for our DevOps platform. Your tutorials and github repos are really helpful. Your explanations and step-by-step posts are top notch. Thank you once again! :)
Hey Andy! Thanks for the video - I was able to setup a single node Rancher cluster in my home lab (RHEL-9, 256GB NVME, 8 core, 64GB RAM) and all of your steps worked as well. Now I have to determine how to add the additional space (4TB SSD that was not attached) so I don't use up my OS/Rancher drive space. Thanks again for the clear and concise video.
@@clemenko Hi, can you make a continuation video to this one. Appreciate it very much if you can make a video to show how to add additional Master Nodes to the cluster in order to make it HA? Also can you make more videos on Neuvector?
@@lawrenceneo2294 continuation might be possible. I have to think about the best steps. I can make a video on Neuvector. Is there anything specific about NV that you want to see?
2:56 please correct me if I'm wrong, but that echo "token: bootstrapAllTheThings" > /etc/rancher/rke2/config.yaml seems like a potentially bad idea since it may override already existing config.yaml. If installer doesn't create config.yaml now, it may in later versions or if user passed some enviroment variables or command line arguments to it.
Great video with clear steps for handling rke2 and longhorn. Hi Clemenko , have you used hpe csi driver instead of longhorn before? Looking for a video with hpe csi driver utilization as well. Any resource reference from anyone is much appreciated. Thank you.
Hey Clemenko! Thanks to your videos, I am enjoying K8 now :) You got a new subscriber here. May God bless you and may you get many more subscribers soon! One quick question, your account name is very catchy, you are a firefighter too ?
Thanks for the kind words. Yes I am a volunteer firefighter/EMT. Been one for over 20 years. I love the balance of working on computers/k8s during the day and then working on a fire engine at night. It is all about balance!
thank you Thanks for the upgrade and the wonderful guide. Can you in the future explain how to install apps, manually. You explain so clearly that everything seems easy.
@@clemenko Thanks for the answer. Useful things, for example, home-assistant, pihole, the whole purpose in the end is to enable some kind of backup if one of the computers falls, there is always a backup. Thank you for your wonderful work.
Thanks for the answer. Useful things, for example, home-assistant, pihole, the whole purpose in the end is to enable some kind of backup if one of the computers falls, there is always a backup. Thank you for your wonderful work.
@avidarks so I found a really good article for pihole : chriskirby.net/highly-available-pi-hole-setup-in-kubernetes-with-secure-dns-over-https-doh/ I would follow it. I don't think I can do any better.
Any reason you went with NeuVector? I've never heard of this tool, the only similar one is Wazuh, which by the looks of it seems to be doing the same things as Vector. Thanks for the updated guide!
The real reason is that NeuVector is owned by SUSE, who also owns Rancher and the other tools. Also NeuVector is a security observability tool. I have never heard of Wazuh before. Looks interesting. I'll take a look at it. Have you deployed it to k8s?
No worries. Already made a video on that. 5 Ways to Install Applications on Kubernetes, Rancher and RKE2 th-cam.com/video/ZqfMzxj98XI/w-d-xo.html Let me know if this video doesn't work for your use case.
Hey bro. Great vid. I’ve been binging all your vids and I noticed one thing. You clear your throat every couple of minutes. And it’s loud. Try and take a lozenge or something before filming. I got you on headphones and you blow my ears every time. It’s probably an unconscious thing. Thanks.
@@clemenko Sorry bro. I reread what I wrote and I must've come off like an ass. You are very thorough and your vids rock. I've been able to get docker down and feel this is the next step up. I've subbed with notifications and hope to see more vids soon.
with regards to slip, does that mean that your suse harvester is open to public internet? what if I want to create a few rke2 clusters managed with rancher in my private network not exposing an endpoint to public internet? (Home lab)
Nope. Sslip.io is just DNS. Did you want all the clusters on your home lab? Or did you want something remote? Or did you want some things in the cloud? Take a look at th-cam.com/video/L7TSawtl97w/w-d-xo.html
@@danirdd92 If it is all home lab then there should be no problems. SSlip.io is just dns. SO I can tell you my nas is on 192.168.1.2 in my house. Since there is no way for you to get there then it is not a security concern. :D Does this help?
Your welcome. Fedora CoreOS might be a little tough. fedoraproject.org/coreos/download?stream=stable#arches is that the one you are talking about? Hope about a video where I figure it out? Would that help?
Hi, until now I had a cluster using talos linux. and I'm looking at the capabilities of rancher and the solution in general, it seems to me a lot of things are automated. Anyway, I would like to ask, what is this good for please? 1) in the sense of this cluster of three nodes is only educational, or is it a cluster that serves purely for the rancher. and I'm going to use the rancher on this cluster to manage another cluster? 2) what is NeuVector good for, what are its benefits in production? 3) if I have 2-3 physical servers that until now had proxmox and talos linux in them. and I install harvester on them, how good is the eco system, what are the benefits? how well does it work all together?. 4) how does the longhorn choose what disk to use for data? I didn't see any settings there, will it use the first available disk that is not the system disk? 5) how is the code as infrastructure architecture from the connection with rancher, e.g. for example we used to use ArgoCD --> can it be fully integrated into the eco system of rancher, can it be used normally on the rancher cluster? is there any way to force changes made inside rancher to be automatically overwritten on the argocd gitOps repository? thank you in advance for the answers. and I want to thank you for the video, it was very nice
Hi. I will try and answer these. 1. Yes this is educational. And yes, you can use this cluster JUST for rancher to manage other clusters, or manage the cluster itself. We see both use cases. 2. NeuVector is good for Security Observability. Proactively blocking bad connections and processes. 3. Harvester would be a replacement for proxmox. The real difference is that under the hood Harvester is running kubernetes to deploy virtual machines. Proxmox does not. I would probably sitck with proxmox for now. At some point maybe get an single node to play with harverster. 4. Longhorn uses `/var/lib/longhorn` on the host. This means you do not need to create or add additional disks to the OS. 5. For GitOPS you can use Ranchers built in tool Fleet. OR you can install ArgoCD on the cluster and deploy apps with helm. You have complete control to deploy how you want. Hope this helps. thanks for the kind words.
@@clemenko Thank you for your answers. I've gotten to the point where I'm going to re-deploy the cluster. i was just using talos-linux, rook-ceph, argocd inside proxmox. (In talos, there is not much security to deal with, there is practically nothing much you can do inside the system. rook-ceph is extremely sensitive to any deviations.) For this reason we are considering to load a harvester on a new SSD, and try to create a new cluster automatically via rancher. and I'm a bit worried about the management of the individual operating system nodes (in talos linux I don't have to do anything for individual nodes) and I'm also considering if switching from rook-ceph to longhorn makes sense, if it will bring me something positive. my current cluster has a lot of problems with IO delay, and that's the reason to try the new stack technology and see if it's the same
I looked at talos. It is very cool. My customers are pretty much mandated to use RHEL/Rocky. Plus the API setup process is not ideal in certain air gap situations. I think you should test harvester out. The big difference between proxmox and harvester is the kubernetes piece where you can run apps right on harvester. Because rook-ceph need extra volumes makes it a bit heavier than longhorn. The nice thing about CSIs is that you can have more than one running. So that should be easy to test. And don't forget, the closer you get to bare metal the more performance you will have.
@@clemenko I'm already trying it out on my server, it's quite interesting. i have one last question. how do i know what linux distributions are ideal for creating a kubernetes cluster via rancher. i've tried rocky for example, but i wanted to ask what is the most minimal and secure linux distribution? (I still have this delusion from talos linux)
@@arwwarr7578 that is a good question. different companies have different mandates. I know my teams use both Ubuntu and RHel. Talos is great if you are connected to the internet and other systems. The API is cool. But for new people looking at kubernetes it is a bit of a learning curve. There have been others like it in the past, rancheros, k30s, coreos. and they all failed because of adoption. I say stick with what you know and are comfortable with. RKE2 installs easily on Ubuntu and Rhel today. If Talos makes sense stick with it.
man, I've been banging my head against my desk all day trying to get this to work. I think there might be something wrong with this setup on ubuntu 24.04.1LTS After I got everything set up on the main VM, CPU usage spiked and basically froze the entire VM. After a few minutes, it settled down to 60% usage, but any kubectl commands I tried to run timed out. (this was on a vm with 8cpu cores and 10 gigs of ram). I spun up a new VM 4 times trying to troubleshoot and figure out what was going on, but never was successful. Just spun up a 22.04.5LTS VM and it doesn't seem to be having the same problem... so far.
workers have each been trying to start the rke2-agent service, and is failing with a "Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry" error And now the CPU on the main VM is spiking again. stopping and starting the rke2-agent service on the worker gives me a timeout error: "failed to retrieve configuration from server: Get \"127.0.0.1:6444/v1-rke2/config\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)""
Wow, there is a lot to unpack here. A. One of my co-workers says to stay with 22.04 instead of 24.04. B. I have never seen a cpu spike. C. If you are re-using nodes you can run rke2-uninstall.sh and it will clean the node out. Even easier is to spin up a new node. Crazy Idea. what if we got on a call together troubleshooted your environment?
@@clemenko I would love the help, that would be great, thanks! I'm slammed for the next few days, and the hard drive on my main workstation just died, so it'll be a little bit before I'm back up and running again. What's the best way for me to reach out once I'm ready? Edit: just saw you have your email address in your profile, I'll shoot you an email hopefully in the next week.
NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized Any inputs on how to get this error resolved, the Status is NotReady when I check the nodes.
Are all the software firewalls off? - Yes. 2. Are the nodes (vms) able to talk to each other? - Yes. 3. What does the networking look like between the machines? - The Network is set to Bridged Adapter option . I tried for NAT Network and manually set the IP, but when I do so the SSH doesnt work. Thats why opted for Bridged Adapter, I generally work on this ---
This is such a good video, really helped me as a newbie to understand each tool and how to get it running - thanks!
Glad it was helpful!
Hey Clemenko, thanks for the detailed explanation! I really like the combination of the written and video tutorial, makes it very easy to follow along.
Awesome, thank you!
Thanks for this !
Helped me setup the RKE2 cluster.
I'm glad it helped!
Clemenko, thank you for everything you are doing for our DevOps platform. Your tutorials and github repos are really helpful. Your explanations and step-by-step posts are top notch. Thank you once again! :)
You're very welcome!
Hey Andy!
Thanks for the video - I was able to setup a single node Rancher cluster in my home lab (RHEL-9, 256GB NVME, 8 core, 64GB RAM) and all of your steps worked as well.
Now I have to determine how to add the additional space (4TB SSD that was not attached) so I don't use up my OS/Rancher drive space.
Thanks again for the clear and concise video.
I'm glad it all worked out! A 4TB SSD is going to be very handy for your cluster.
Well explained!!! By far the best explanation I have ever seen.
Thanks. I am glad you liked it. What video should I make next?
@@clemenko Hi, can you make a continuation video to this one. Appreciate it very much if you can make a video to show how to add additional Master Nodes to the cluster in order to make it HA? Also can you make more videos on Neuvector?
@@lawrenceneo2294 continuation might be possible. I have to think about the best steps. I can make a video on Neuvector. Is there anything specific about NV that you want to see?
Love your videos! So helpful!!
Thank you so much!
This is incredible! Thank you
You're so welcome!
Its really awesome! Thank you!
Glad you like it!
this is super helpful! Please do it with let's encrypt certs! Thanks for your work!
Interesting. You want to encrypt the app across the tailnet?
2:56 please correct me if I'm wrong, but that
echo "token: bootstrapAllTheThings" > /etc/rancher/rke2/config.yaml
seems like a potentially bad idea since it may override already existing config.yaml. If installer doesn't create config.yaml now, it may in later versions or if user passed some enviroment variables or command line arguments to it.
On an install config.yaml does not get generated. It is safe to update it and restart the daemon.
Great video with clear steps for handling rke2 and longhorn. Hi Clemenko , have you used hpe csi driver instead of longhorn before? Looking for a video with hpe csi driver utilization as well. Any resource reference from anyone is much appreciated. Thank you.
@ericneba970 I have not. Let me see if I can find the install docs. Do you have a link?
I found scod.hpedev.io/partners/suse_rancher/index.html. Testing now.
@ericneba970 What did you make me do?
Rancher HPE CSI Driver Install had issues! We fixed it. | th-cam.com/video/DcTSIlcGAPo/w-d-xo.html
10:16 I am getting "503 Service Temporarily Unavailable", please advice anyone. I am new to rancher and kubes . Thanks !
it loaded after i took a break to get my breakfast ...looks like it was waiting for m to have my breakfast first :D
Can't skip breakfast. ;) Let me know if you run into any other issues.
@@clemenko thank you it's all running smoothly now. Really enjoyed learning new stuff thank you for this video 😇
Thank you 🙏🏼
You’re welcome 😊
Hey Clemenko! Thanks to your videos, I am enjoying K8 now :) You got a new subscriber here. May God bless you and may you get many more subscribers soon! One quick question, your account name is very catchy, you are a firefighter too ?
Thanks for the kind words. Yes I am a volunteer firefighter/EMT. Been one for over 20 years. I love the balance of working on computers/k8s during the day and then working on a fire engine at night. It is all about balance!
@@clemenko that's wonderful, 😇
thank you
Thanks for the upgrade and the wonderful guide.
Can you in the future explain how to install apps, manually.
You explain so clearly that everything seems easy.
Absolutely. What apps are you interested in seeing?
@@clemenko Thanks for the answer.
Useful things, for example, home-assistant, pihole, the whole purpose in the end is to enable some kind of backup if one of the computers falls, there is always a backup.
Thank you for your wonderful work.
Thanks for the answer.
Useful things, for example, home-assistant, pihole, the whole purpose in the end is to enable some kind of backup if one of the computers falls, there is always a backup.
Thank you for your wonderful work.
Let me see what I can find on pihole. I actually run adguard on docker inside my synology nas at home. But it should be easy enough.
@avidarks so I found a really good article for pihole : chriskirby.net/highly-available-pi-hole-setup-in-kubernetes-with-secure-dns-over-https-doh/ I would follow it. I don't think I can do any better.
Any reason you went with NeuVector? I've never heard of this tool, the only similar one is Wazuh, which by the looks of it seems to be doing the same things as Vector. Thanks for the updated guide!
The real reason is that NeuVector is owned by SUSE, who also owns Rancher and the other tools. Also NeuVector is a security observability tool. I have never heard of Wazuh before. Looks interesting. I'll take a look at it. Have you deployed it to k8s?
@@clemenko Yes, it works on pretty much everything and it's also open source. Try it out!
How to build a container from the rancher ?
@clemenko
Sorry, I mean deploy :)
No worries. Already made a video on that. 5 Ways to Install Applications on Kubernetes, Rancher and RKE2 th-cam.com/video/ZqfMzxj98XI/w-d-xo.html Let me know if this video doesn't work for your use case.
Hey bro. Great vid. I’ve been binging all your vids and I noticed one thing. You clear your throat every couple of minutes. And it’s loud. Try and take a lozenge or something before filming. I got you on headphones and you blow my ears every time. It’s probably an unconscious thing. Thanks.
Thanks. Honestly didn't realize I cleared my throat that much. I will work on it. At least I will drop the audio levels if I do. Hope that helps.
@@clemenko Sorry bro. I reread what I wrote and I must've come off like an ass. You are very thorough and your vids rock. I've been able to get docker down and feel this is the next step up. I've subbed with notifications and hope to see more vids soon.
@@josemercado1674 no worries. All feedback is good feedback. Hope you are liking the videos. Let me know if you have any video ideas.
I clear my throat too often due to acid reflux
@@ucfeconknight i think i was recovering from a cold when I shot this video.
with regards to slip, does that mean that your suse harvester is open to public internet? what if I want to create a few rke2 clusters managed with rancher in my private network not exposing an endpoint to public internet? (Home lab)
Nope. Sslip.io is just DNS. Did you want all the clusters on your home lab? Or did you want something remote? Or did you want some things in the cloud? Take a look at th-cam.com/video/L7TSawtl97w/w-d-xo.html
@clemenko multi cluster all home lab no port forwarding my home router to my cluster for security concerns
@@danirdd92 If it is all home lab then there should be no problems. SSlip.io is just dns. SO I can tell you my nas is on 192.168.1.2 in my house. Since there is no way for you to get there then it is not a security concern. :D Does this help?
Hello, thanks for the great content can you make a video about RKE2 on fedora coreos adding rancher & monitoring & longhorn , thanks
Your welcome. Fedora CoreOS might be a little tough. fedoraproject.org/coreos/download?stream=stable#arches is that the one you are talking about?
Hope about a video where I figure it out? Would that help?
How to uninstall and remove everything so that I can re initiate the exact same setup ?
There should be a command "rke-uninstall.sh" that will remove everything. If possible I would delete the VMs and start over.
Hi, until now I had a cluster using talos linux. and I'm looking at the capabilities of rancher and the solution in general, it seems to me a lot of things are automated.
Anyway, I would like to ask, what is this good for please?
1) in the sense of this cluster of three nodes is only educational, or is it a cluster that serves purely for the rancher. and I'm going to use the rancher on this cluster to manage another cluster?
2) what is NeuVector good for, what are its benefits in production?
3) if I have 2-3 physical servers that until now had proxmox and talos linux in them. and I install harvester on them, how good is the eco system, what are the benefits? how well does it work all together?.
4) how does the longhorn choose what disk to use for data? I didn't see any settings there, will it use the first available disk that is not the system disk?
5) how is the code as infrastructure architecture from the connection with rancher, e.g. for example we used to use ArgoCD --> can it be fully integrated into the eco system of rancher, can it be used normally on the rancher cluster? is there any way to force changes made inside rancher to be automatically overwritten on the argocd gitOps repository?
thank you in advance for the answers. and I want to thank you for the video, it was very nice
Hi. I will try and answer these.
1. Yes this is educational. And yes, you can use this cluster JUST for rancher to manage other clusters, or manage the cluster itself. We see both use cases.
2. NeuVector is good for Security Observability. Proactively blocking bad connections and processes.
3. Harvester would be a replacement for proxmox. The real difference is that under the hood Harvester is running kubernetes to deploy virtual machines. Proxmox does not. I would probably sitck with proxmox for now. At some point maybe get an single node to play with harverster.
4. Longhorn uses `/var/lib/longhorn` on the host. This means you do not need to create or add additional disks to the OS.
5. For GitOPS you can use Ranchers built in tool Fleet. OR you can install ArgoCD on the cluster and deploy apps with helm. You have complete control to deploy how you want.
Hope this helps.
thanks for the kind words.
@@clemenko Thank you for your answers.
I've gotten to the point where I'm going to re-deploy the cluster. i was just using talos-linux, rook-ceph, argocd inside proxmox. (In talos, there is not much security to deal with, there is practically nothing much you can do inside the system. rook-ceph is extremely sensitive to any deviations.)
For this reason we are considering to load a harvester on a new SSD, and try to create a new cluster automatically via rancher.
and I'm a bit worried about the management of the individual operating system nodes (in talos linux I don't have to do anything for individual nodes)
and I'm also considering if switching from rook-ceph to longhorn makes sense, if it will bring me something positive. my current cluster has a lot of problems with IO delay, and that's the reason to try the new stack technology and see if it's the same
I looked at talos. It is very cool. My customers are pretty much mandated to use RHEL/Rocky. Plus the API setup process is not ideal in certain air gap situations. I think you should test harvester out. The big difference between proxmox and harvester is the kubernetes piece where you can run apps right on harvester.
Because rook-ceph need extra volumes makes it a bit heavier than longhorn. The nice thing about CSIs is that you can have more than one running. So that should be easy to test.
And don't forget, the closer you get to bare metal the more performance you will have.
@@clemenko I'm already trying it out on my server, it's quite interesting. i have one last question. how do i know what linux distributions are ideal for creating a kubernetes cluster via rancher. i've tried rocky for example, but i wanted to ask what is the most minimal and secure linux distribution? (I still have this delusion from talos linux)
@@arwwarr7578 that is a good question. different companies have different mandates. I know my teams use both Ubuntu and RHel. Talos is great if you are connected to the internet and other systems. The API is cool. But for new people looking at kubernetes it is a bit of a learning curve. There have been others like it in the past, rancheros, k30s, coreos. and they all failed because of adoption. I say stick with what you know and are comfortable with. RKE2 installs easily on Ubuntu and Rhel today. If Talos makes sense stick with it.
man, I've been banging my head against my desk all day trying to get this to work. I think there might be something wrong with this setup on ubuntu 24.04.1LTS
After I got everything set up on the main VM, CPU usage spiked and basically froze the entire VM. After a few minutes, it settled down to 60% usage, but any kubectl commands I tried to run timed out. (this was on a vm with 8cpu cores and 10 gigs of ram). I spun up a new VM 4 times trying to troubleshoot and figure out what was going on, but never was successful.
Just spun up a 22.04.5LTS VM and it doesn't seem to be having the same problem... so far.
workers have each been trying to start the rke2-agent service, and is failing with a "Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry" error
And now the CPU on the main VM is spiking again.
stopping and starting the rke2-agent service on the worker gives me a timeout error: "failed to retrieve configuration from server: Get \"127.0.0.1:6444/v1-rke2/config\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)""
restarting the main VM, and the rke2-server service logs this when trying to start: "unable to verify local node password: hash does not match"
Wow, there is a lot to unpack here.
A. One of my co-workers says to stay with 22.04 instead of 24.04.
B. I have never seen a cpu spike.
C. If you are re-using nodes you can run rke2-uninstall.sh and it will clean the node out. Even easier is to spin up a new node.
Crazy Idea. what if we got on a call together troubleshooted your environment?
@@clemenko I would love the help, that would be great, thanks! I'm slammed for the next few days, and the hard drive on my main workstation just died, so it'll be a little bit before I'm back up and running again. What's the best way for me to reach out once I'm ready?
Edit: just saw you have your email address in your profile, I'll shoot you an email hopefully in the next week.
@@jrucker2004 Good luck with the drive replacement.
NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Any inputs on how to get this error resolved, the Status is NotReady when I check the nodes.
where are you getting that message?
@@clemenko when I check on - kubectl get node command it shows not ready . So I checked and got it from kubectl describe command
Are all the software firewalls off? Are the nodes (vms) able to talk to each other? What does the networking look like between the machines?
Are all the software firewalls off? - Yes. 2. Are the nodes (vms) able to talk to each other? - Yes. 3. What does the networking look like between the machines? - The Network is set to Bridged Adapter option . I tried for NAT Network and manually set the IP, but when I do so the SSH doesnt work. Thats why opted for Bridged Adapter, I generally work on this ---
@@thanushashetty-b8w are the vms on your laptop? It would appear there is a networking issue. Do you have access to a cloud provider?
Kuhlman Brook
who dis? lol