Great jobs! Thanks for your valuable and useful suggestions. Tomorrow I will try to follow your tutorial in our environment. You drawn away clouds from my brain. :-)
It is desirable if you can also detailed your ha proxy setup and also how to prepare your private CA. Hopeful you can refresh this video very soon with these little suggestions (and also the latest OpenShift version!). Thanks!
Here is an example of my ha-proxy configuration github.com/ocpdude/vmware-upi-install/blob/main/haproxy/example-haproxy.cfg For the private CA, there are lots of examples Internet wide, my lab firewall using OpenSSL to generate the certs which I install throughout the lab.
Hello; Would it be possible to record similar session on how to create an Openshift single-node cluster on ESXi server? There are some posts in the Internet, but despite deploying every step mentioned, I have not been able to build a working cluster. I have access to an ESXi server in our office but don't have vCenter. Thanks.
@@OCPdude Thanks. Yes that's what I need. I created a Centos minimal and used KVM to create a single OCP VM where it is both a master and worker node at the same time. Some say the official documents don't support single-node cluster, but I found several blog posts about creating such a cluster for learning lab (I followed some of them, but didn't manage to make it work at the end, though).
@@adamfreemotion8849 Hi, I haven't recorded the video yet - but if you want the steps to install a single node OCP lab... I've posted a script on GitHub : github.com/ocpdude/crc-vm-build I built this using CentOS 7.9 Minimal, on ESXi 6.7 U3... I'll try to get the video up sometime this week.
@@OCPdude Thank you again. I'm excitingly waiting for this video :) Just one question; the link you shared uses Code Ready Container. Is it suitable for OKD admin? somewhere I read CRC focuses more on developer side of the openshift than administrator. I'm not developer.
Thank you so much for detailed explanation, I'm new to openshift. I could see couple of ways of openshift installation. My question is companies use IPI way of installing openshift in their infrastructure or they go with UPI only in most cases ?
@@OCPdude Thanks for the response. I could see in Redhat openshift documents for all the latest versions (4.x) are with UPI way of installation, this method replaced Ansible playbooks in 4.x or we still use playbooks in the recent versions?
I am trying to configure OKD latest version using VMware sphere 6.5 U1 customised the ISO with static ip and ignition file download on the boot loader itself and OVF file setting diskEnableUUID set to true then I packed OVF and ISO to OVA. imported into sphere. The problem I am facing is 1. On the bootstrap node there is some issue on the Kubelet.service, I have to edit the service and set this env Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice" 2. Then the bootstrap is up, but on worker node it booted up successfully, meanwhile after sometime it got into another issue runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? I have no idea how to resolve this
Great clarity; thanks much! Can you please give some pointers on how to tweak install-config.yaml to install OCP 4.6.1 on two vSphere clusters under single vCentre (so that high availability can be achieved)?
You will achieve HA in your ocp/kubernetes cluster by installing it as described. The managers provide scheduling failover (ha), and the workers will respond by receiving new jobs from failed nodes. Your applications should be designed in such a way to address fail overs. To get to the point of your question, you may install a second instance of ocp using a different cluster name/dns/folder.... generally speaking, I would recommend sub-domains if you want to keep the same parent domain. ie, .openshift.domain.com & .openshift.domain.com. Make sense?
You base64 encoded your append-bootstrap.yaml, master.yaml and worker.yaml, and then entered them in the VM Options>Advanced as outlined here: th-cam.com/video/6TvyHBdHhes/w-d-xo.html and it's not fetching the bootstrap.yaml from your http server? Is that what you're saying?
Only the ESXi Hypervisor is free of charge. However, you can install OCP on ESXi without vCenter, or coming soon... (today?) I'm publishing a video on Code Ready Containers - it's a single master/worker OCP deployment for dev/test. You may find this interesting, stay tuned.
@@NathLeeUK I'd recommend making sure you can resolve forward and reverse lookups on the hostname, and if you can, could you email an example of your value or a screen shot? You can find my email address or slack from the about page.
@@OCPdude FATAL failed to fetch Master Machines: failed to load asset "Install Config": failed to unmarshal install-config.yaml: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field InstallConfig.pullSecret of type string any idea on this issue
Unlike open source Kubernetes, OpenShift platform relies on CoreOS, therefore you must install it from the install scripts. There are 3 options, 1. a single node (master/worker), 2. IPI where a lot of the build is automated for you and 3. UPI where you have the ability to customize your build/deployment. There is also CodeReady Containers (CRC) which is great for small lab environments. In this link, please see "Datacenter or Local" for these options: console.redhat.com/openshift/create/cloud
Great tutorial. Thank you. You mentioned it is possible to add more compute nodes with 99_openshift-cluster-api_worker-machineset-0.yaml later. Would you be so kind and provide some instructions.
I just run a nginx service to serve up the bootstrap.ign file... this could be from a separate vm or hosted from your install machine. You’ll then copy the file over and make sure read permissions are granted.
@@karanpatani84 That's correct. Your DNS for *.apps.cluster.domain.com and api.cluster.domain.com should point to your load balancer, and then just create port LB rules to distribute to your master/worker(ingress) nodes. I went ahead and put an example on GitHub for you: github.com/ocpdude/vmware-upi-install/blob/main/haproxy/example-haproxy.cfg
@@OCPdude so we tried setting up the load balancer similar to the one you uploaded on github. we seem to have multiple issues. 1. It seems the ignition files aren't picking up correct SSH keys added to yaml file. 2. The bootstrap VM only comes up with dynamic IP instead of static like we want. 3. The bootstrap VM doesn't come up the first time with base64 encoding string added to it. It would only come up once we remove the string. 4. Once the bootstrap VM comes online the second time with base64 string, we are unable to ssh to it. And keep on getting Permission denied error. Please help as we are stuck here for quite sometime now. We have created the ignition multiple times now due to cert expiry.
@@karanpatani84 I should first mention that I am not Red Hat support and I'm only trying to help out here. That all understood, this sounds like you have a formatting error in your install-config.yaml file... additionally, you may not have your static ip set correctly in vmware advance options. The load balancer hasn't even come in to play yet since the bootstrap node isn't online. Please email me your install-config.yaml (remove any confidential info - like your domain and ssh_key) and I'll review it. Also, here's the video for properly setting your static IP addresses: th-cam.com/video/anroxnT7pyM/w-d-xo.html
Can you explain how you tell the installer that you have external Load Balancer? I have an external load balancer, but when the bootstrap start it always assign the api ip adress to himself and provision an internal loadbalancer
Sounds like your doing an IPI install and not this method. Under UPI, you'd have an external load balancer, same as me... just make sure all your forward and reverse DNS is setup correctly. If you're using DHCP (default) then make sure your MAC addresses are set properly, or set your IP addresses manually (th-cam.com/video/anroxnT7pyM/w-d-xo.html)
@@OCPdude I followed your tutorial exactly, (upi) I am using DHCP with reserved Mac. Host names and ip addresses are correct at startup. all hostnames are in dns zone and i can ping and reverse lookup the bootstrap gets the ign from the webserver, but after a few minutes i see in the vmware gui, that the IP address of the API is showing up in the vm bootstrap. If I start the Master they come up like in your video. and after some minutes the wildcard Ip appear on the master. maybe if my external loadbalancer is not well configured, the bootstrap can detect that the IP address of the API is not in use and continue with internal load balancer. I will give another try by changing the scope Internal to scope External in manifests/cluster-ingress-default-ingresscontroller.yaml spec: endpointPublishingStrategy: loadBalancer: scope: External type: LoadBalancerService Thanks for your answer!
@@ticasse999 You can use my GitHub install-config.yaml as a baseline, it is essentially what I used in the demo. Also, it still sounds like some IPI is kicking off somehow, you can add these properties to try an force the ip resolution platform: vsphere: ... ... apiVIP: api_vip ingressVIP: ingress_vip
Sorry, I had to edit the above note... messaging in TH-cam isn't great... one more note, make sure your workers are "0" and that you have set the masters "masterSchedulable: false" >>> GitHub reference = github.com/ocpdude/vmware-upi-install
@@OCPdude Maybe is because I did a ipi installation before the upi, and some config stayed in cache. I will try from another linux to make all the ignition config. I confirm worker is "0" and masterSchedulable are false. Thanks again!
@@Trekker_KD it's just a linux vm I use which is internal and on the same network as my lab. you can call it a helper node, but I want to be clear that it is 'not' the bootstrap node.
thank you Dude, i was looking for this, really useful, can you please share some more info about DHCP Server here. do we need to do it? Do we need the dHCP server forever or one time only for intitial configuration. Also, are you really adding all MAC Ids in installconfig.yaml?
With the current version of OCP, you do not need to reserve MAC address in DHCP as support for static IP's is available as arguments during your VM builds. When using MAC assignments, they are not in the install-config.yaml. Essentially, your DHCP would have static IP reservations per MAC address/Node and of course they would be in your DNS as well. Here is OCP 4.7 UPI on VMware w/ Static IP's: th-cam.com/video/9__hpUWK5vw/w-d-xo.html
Great jobs! Thanks for your valuable and useful suggestions. Tomorrow I will try to follow your tutorial in our environment. You drawn away clouds from my brain. :-)
Ha! Great, glad you found it useful.
Good work man make it up realy very usefull tutorials for Red Hat OpenShift installation complexity
Thank you for this. Its a great video. Subd.
Thanks for the sub!
Thanks dude❗️🙏🏻
Great explained 👍
It is desirable if you can also detailed your ha proxy setup and also how to prepare your private CA. Hopeful you can refresh this video very soon with these little suggestions (and also the latest OpenShift version!). Thanks!
Here is an example of my ha-proxy configuration github.com/ocpdude/vmware-upi-install/blob/main/haproxy/example-haproxy.cfg For the private CA, there are lots of examples Internet wide, my lab firewall using OpenSSL to generate the certs which I install throughout the lab.
Could you please make a video on Single node openshift deployment in Vsphere ?
Is it possible to share the DNS and HAproxy configuration ?
Hello;
Would it be possible to record similar session on how to create an Openshift single-node cluster on ESXi server? There are some posts in the Internet, but despite deploying every step mentioned, I have not been able to build a working cluster. I have access to an ESXi server in our office but don't have vCenter. Thanks.
What you are asking for is how to setup OCP on ESXi without vCenter? No problem, I’ll get to it.
@@OCPdude Thanks. Yes that's what I need. I created a Centos minimal and used KVM to create a single OCP VM where it is both a master and worker node at the same time. Some say the official documents don't support single-node cluster, but I found several blog posts about creating such a cluster for learning lab (I followed some of them, but didn't manage to make it work at the end, though).
@@adamfreemotion8849 Hi, I haven't recorded the video yet - but if you want the steps to install a single node OCP lab... I've posted a script on GitHub : github.com/ocpdude/crc-vm-build
I built this using CentOS 7.9 Minimal, on ESXi 6.7 U3... I'll try to get the video up sometime this week.
@@OCPdude Thank you again. I'm excitingly waiting for this video :) Just one question; the link you shared uses Code Ready Container. Is it suitable for OKD admin? somewhere I read CRC focuses more on developer side of the openshift than administrator. I'm not developer.
Thank you so much for detailed explanation, I'm new to openshift. I could see couple of ways of openshift installation. My question is companies use IPI way of installing openshift in their infrastructure or they go with UPI only in most cases ?
It was my experience that most enterprises chose UPI for the full customization of the platform, networking (load balancer, etc).
@@OCPdude Thanks for the response. I could see in Redhat openshift documents for all the latest versions (4.x) are with UPI way of installation, this method replaced Ansible playbooks in 4.x or we still use playbooks in the recent versions?
@@marcelonrs6811 No Ansible playbooks are required for any install of OpenShift 4.x
@@OCPdude Okay, Got it. Thank you!
I am trying to configure OKD latest version using VMware sphere 6.5 U1 customised the ISO with static ip and ignition file download on the boot loader itself and OVF file setting diskEnableUUID set to true then I packed OVF and ISO to OVA. imported into sphere. The problem I am facing is
1. On the bootstrap node there is some issue on the Kubelet.service, I have to edit the service and set this env Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice"
2. Then the bootstrap is up, but on worker node it booted up successfully, meanwhile after sometime it got into another issue
runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
I have no idea how to resolve this
Great clarity; thanks much! Can you please give some pointers on how to tweak install-config.yaml to install OCP 4.6.1 on two vSphere clusters under single vCentre (so that high availability can be achieved)?
You will achieve HA in your ocp/kubernetes cluster by installing it as described. The managers provide scheduling failover (ha), and the workers will respond by receiving new jobs from failed nodes. Your applications should be designed in such a way to address fail overs. To get to the point of your question, you may install a second instance of ocp using a different cluster name/dns/folder.... generally speaking, I would recommend sub-domains if you want to keep the same parent domain. ie, .openshift.domain.com & .openshift.domain.com. Make sense?
Hi OCPdude, during the installation my VM instances are not picking up ignition data from configuration params. Pls help.
You base64 encoded your append-bootstrap.yaml, master.yaml and worker.yaml, and then entered them in the VM Options>Advanced as outlined here: th-cam.com/video/6TvyHBdHhes/w-d-xo.html and it's not fetching the bootstrap.yaml from your http server? Is that what you're saying?
OCPdude, thanks for the videos! Is vCenter free to download anymore with a 6.7 vSphere install?
Only the ESXi Hypervisor is free of charge. However, you can install OCP on ESXi without vCenter, or coming soon... (today?) I'm publishing a video on Code Ready Containers - it's a single master/worker OCP deployment for dev/test. You may find this interesting, stay tuned.
Great clarity. thanks so much, can share us how deploy with static IP address.
Great idea. I will rebuild my lab soon and use the static IP method and post those details.
Here you go... th-cam.com/video/anroxnT7pyM/w-d-xo.html
@@OCPdude Great video - i've followed your method using static IPs but no matter what I do my nodes always end up in emergency mode.
@@NathLeeUK I'd recommend making sure you can resolve forward and reverse lookups on the hostname, and if you can, could you email an example of your value or a screen shot? You can find my email address or slack from the about page.
@@OCPdude
FATAL failed to fetch Master Machines: failed to load asset "Install Config": failed to unmarshal install-config.yaml: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field InstallConfig.pullSecret of type string
any idea on this issue
Mac address does not work in yaml for me 😞
Hi
can you share the steps to build up all machines from scratch so that it will be easy to build myself at home lab.
Unlike open source Kubernetes, OpenShift platform relies on CoreOS, therefore you must install it from the install scripts. There are 3 options, 1. a single node (master/worker), 2. IPI where a lot of the build is automated for you and 3. UPI where you have the ability to customize your build/deployment. There is also CodeReady Containers (CRC) which is great for small lab environments. In this link, please see "Datacenter or Local" for these options: console.redhat.com/openshift/create/cloud
Great tutorial. Thank you. You mentioned it is possible to add more compute nodes with 99_openshift-cluster-api_worker-machineset-0.yaml later. Would you be so kind and provide some instructions.
Here you go : th-cam.com/video/3no-WT557ls/w-d-xo.html
@@davorinkocbek4779 About Page
where and how to provision the web server for serving ignition files
I just run a nginx service to serve up the bootstrap.ign file... this could be from a separate vm or hosted from your install machine. You’ll then copy the file over and make sure read permissions are granted.
great detailed video, one query though. did you use HA Proxy for both load balancers?
I have 1 HAProxy load balancer that distributes port 6443 to the masters and 443/80 (and NodePorts) to the workers ingress.
@@OCPdude okay, so we need not have 2 load balancers? we can make rules for both the masters and workers in a single load balancer?
@@karanpatani84 That's correct. Your DNS for *.apps.cluster.domain.com and api.cluster.domain.com should point to your load balancer, and then just create port LB rules to distribute to your master/worker(ingress) nodes. I went ahead and put an example on GitHub for you: github.com/ocpdude/vmware-upi-install/blob/main/haproxy/example-haproxy.cfg
@@OCPdude so we tried setting up the load balancer similar to the one you uploaded on github. we seem to have multiple issues.
1. It seems the ignition files aren't picking up correct SSH keys added to yaml file.
2. The bootstrap VM only comes up with dynamic IP instead of static like we want.
3. The bootstrap VM doesn't come up the first time with base64 encoding string added to it. It would only come up once we remove the string.
4. Once the bootstrap VM comes online the second time with base64 string, we are unable to ssh to it. And keep on getting Permission denied error.
Please help as we are stuck here for quite sometime now. We have created the ignition multiple times now due to cert expiry.
@@karanpatani84 I should first mention that I am not Red Hat support and I'm only trying to help out here. That all understood, this sounds like you have a formatting error in your install-config.yaml file... additionally, you may not have your static ip set correctly in vmware advance options. The load balancer hasn't even come in to play yet since the bootstrap node isn't online. Please email me your install-config.yaml (remove any confidential info - like your domain and ssh_key) and I'll review it. Also, here's the video for properly setting your static IP addresses: th-cam.com/video/anroxnT7pyM/w-d-xo.html
Can you explain how you tell the installer that you have external Load Balancer?
I have an external load balancer, but when the bootstrap start it always assign the api ip adress to himself and provision an internal loadbalancer
Sounds like your doing an IPI install and not this method. Under UPI, you'd have an external load balancer, same as me... just make sure all your forward and reverse DNS is setup correctly. If you're using DHCP (default) then make sure your MAC addresses are set properly, or set your IP addresses manually (th-cam.com/video/anroxnT7pyM/w-d-xo.html)
@@OCPdude I followed your tutorial exactly, (upi)
I am using DHCP with reserved Mac.
Host names and ip addresses are correct at startup.
all hostnames are in dns zone and i can ping and reverse lookup
the bootstrap gets the ign from the webserver, but after a few minutes i see in the vmware gui, that the IP address of the API is showing up in the vm bootstrap.
If I start the Master they come up like in your video. and after some minutes the wildcard Ip appear on the master.
maybe if my external loadbalancer is not well configured, the bootstrap can detect that the IP address of the API is not in use and continue with internal load balancer.
I will give another try by changing the scope Internal to scope External in manifests/cluster-ingress-default-ingresscontroller.yaml
spec:
endpointPublishingStrategy:
loadBalancer:
scope: External
type: LoadBalancerService
Thanks for your answer!
@@ticasse999 You can use my GitHub install-config.yaml as a baseline, it is essentially what I used in the demo. Also, it still sounds like some IPI is kicking off somehow, you can add these properties to try an force the ip resolution
platform:
vsphere:
...
...
apiVIP: api_vip
ingressVIP: ingress_vip
Sorry, I had to edit the above note... messaging in TH-cam isn't great... one more note, make sure your workers are "0" and that you have set the masters "masterSchedulable: false" >>> GitHub reference = github.com/ocpdude/vmware-upi-install
@@OCPdude Maybe is because I did a ipi installation before the upi, and some config stayed in cache.
I will try from another linux to make all the ignition config.
I confirm worker is "0" and masterSchedulable are false.
Thanks again!
whats the node from which u are executing commands
my laptop or bastion host - anything with the 'oc, kubectl and openshift-install' binaries installed.
@@OCPdude bastion ?? The helper node right
@@Trekker_KD it's just a linux vm I use which is internal and on the same network as my lab. you can call it a helper node, but I want to be clear that it is 'not' the bootstrap node.
@@OCPdude yeh got it thanks
thank you Dude, i was looking for this, really useful, can you please share some more info about DHCP Server here. do we need to do it? Do we need the dHCP server forever or one time only for intitial configuration. Also, are you really adding all MAC Ids in installconfig.yaml?
With the current version of OCP, you do not need to reserve MAC address in DHCP as support for static IP's is available as arguments during your VM builds. When using MAC assignments, they are not in the install-config.yaml. Essentially, your DHCP would have static IP reservations per MAC address/Node and of course they would be in your DNS as well.
Here is OCP 4.7 UPI on VMware w/ Static IP's: th-cam.com/video/9__hpUWK5vw/w-d-xo.html