What do you think about Cluster API? Will it become the preferrable way to create, provision, and operate Kubernetes clusters? IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), TH-cam tends to delete comments that contain links. Please do not use them in your comments.
I learnt earlier that crossplane can spin up monitor and manage any infra as long as the target infra has native API for spinning infra, so can we not extend crossplane to spin up self managed k8s cluster on prem like using vSphere API on VMware for example. Not sure if I am asking a relevant question here.
You are right. Crossplane can manage vSphere. Or, to be more precise, it could manage it, but I'm not sure what is the status of the VMware provider. I'm not using it myself so I haven't been following. If that provider is not yet done, it should be relatively easy to create it through the Terrajet project. The best thing would be to raise an issue in the crossplane repo. That would raise the awereness that users are interested and someone will pick it up. Also, if you'd be interested in learning how to create providers, I can give you a hand.
ClusterClass in CAPI solves the issues that you brought up in the video. Its new and not all providers support it yet, but its a single crd to create a cluster and a single place to upgrade the version. For CNI you can also have it installed automatically via the ClusterResourceSet feature in CAPI
Its also interesting that almost all multi cloud commercial offerings for k8s are based in CAPI today: Vmware TKG Rancher Google Anthos Amazon EKS-A OpenShift (partial implementation as they are only using the machine API from CAPI)
It still makes sense to use CAPI indepedently if you want to avoid vendor lock in. While those commercial multi cloud solutions allow user to take advantage of multiple cloud services, it does so by using vendor specific tools. For some of those solutions it also means you have to do multi cloud management from within a parcicular provider's cloud.
The CABPT project uses CAPI to bootstrap a Talos K8S cluster, from there it's trivial to bootstrap Crossplane, and then using Crossplane as a CAPI provider to boostrap additional K8S clusters, I think.
I like the idea of clusterapi by create new nodes and replace them. I upgraded linux nodes in-place for a long time and kubernetes clusters in place recently. The short version: in-place upgrades suck, replacing nodes is much better
thank you. while watching this video it really made me think more deeply about the types of abstraction i really want from all this kubernetes stuff, all this cloud providers and generally devops api. and it ultimately should come down to seeking 2 things. one of them is costs of services. and the other one is performance provided for those costs. and it should all be through a standardized single grpc api. so in my mind the problem is then a lack of standardization across providers. across these platforms. and maybe kubernetes cluster api and crossplane is a step in the right direction. or using openshift etc. however i dont think the transparency and coherency is there to provider QoS guarantees or smart automatic billing contract built into the charging sytem over these same apis. for when the service provided is either better or worse than expected. and for to set upper and lower limits we are talking how many nodes here. rather than how many dollars. or how long the ping time. or other network bandwidth metrics etc. across these same api. and indistinguishable between the provider. it seems like a difficult problem to solve. i suppose there is kubecost however it is in itself another seperate 3rd party service to setup, manage and configure. hmm, but i wonder if it is a place to start at. when dreaming up such kinds of mythical grpc apis. and along that journey, to also be examining which common cloud apis are the most coherent / commonly shared. the scope is broad ranging. i wonder how much is not possible to be covered within the domain of kubernetes itself. and it also makes me wonder the true value of all these different things
I started writing a response but quickly realized that it's too big of a subject. How about talking about it in one of the upcoming AMA sessions or, even better, in the monthly chat reserved for the members? The short version of what I wanted to write: standardization of Cloud will almost certainly not happen any time soon (if ever) and the best we can do is have something "standard" running on top of them (Kubernetes).
@@DevOpsToolkit yes agreed. it is too big a subject. and i totally agree - we will never see any voluntary and 'coherent' to be a well standardized cloud API. (that is simple, logical, and transparent) essentially the 2 things: the performance and the cost. those are the very things which they want to abstract away the most from the customer. 'dont pay attention to that' or 'we make it complicated and hard to understand on purpose'. it reminds me of ISPs broadband. and mobile phone cellular service. except with cloud we have many more service as a result. they kind of multiply by themselves? like some virus or something IDK just this insight alone makes me want to make kubecost a higher priotity (for myself). just on my own list of which 3rd party infrastructure addon services are worth to go deeper into. for my own limited available time. because there clearly are no other ways to solve this problem, right? except with something like kubecost. that focus then lets me a reasonable justification investigate further those related areas. things which are connected to it. for example: how to have a 3rd party monitoring service additions added onto kubecost. possibly as plugins for kubecost. or otherwise at the output side of kubecost. in graphana / prometheus metrics. and connecting the cost to service performance. as some chosen and measured performance metrics. this could be a fun project while learning kubernetes. if planned ahead and broken down into smaller chunk. with stage1 building upon to the next stage2 and so on.
You're right. I saw it in capz.sigs.k8s.io/topics/managedcluster.html after I made the video. The docs are a bit confusing and all over the place. If you go to cluster-api.sigs.k8s.io/reference/providers.html there is no mention of AKS. It's also not mentioned in github.com/kubernetes-sigs/cluster-api-provider-azure. So, one needs to do a bit of digging. Nevertheless, you're right. AKS is there.
Ya the docs arent the most organized. Typically the best is the CAPI book and the Provider specific books if they have them which not all do. I really hope the docs get better but as we all know docs and naming are probably the hardest things in tech today
cluster api not able to reconcile for docker infrastructure provider, I have created a k8s cluster using kind. Initialized it using clusterctl init --infrastructure docker and apply one CAPI manifest for 3 control-node + 3 worker node. It was successfully deployed, all the machines were up (docker containers). After that I am trying to test the benefits of doing this whole thing using CAPI (Reconciliation, full lifecycle management of workload clusters). For that I stopped 1 container(worker node machine), So even though Cluster CAPI detects the change and starts the reconciliation process (this I can say by checking the capd-controller-manager pod logs, it was not able to set up the worker node. I expected it to launch another container and add it to the cluster but it's not able to do that. There is no issue with memory, I have increased the inotify limit. Please help me debug.
Thanks for this video :) Can you make an updated version of this video please? Or can you just mention in the reply what is the current state CAPI? Are the CONS mentioned here been solved. Most importantly cluster update/upgrade?
CAPI is alive and kicking and advanced a lot since i made that video. If you manage only kubernetes clusters, it is s very good choice. If you need more, i would recommend crossplane.
I created kind cluster. I did clutserctl init. Now I run the command `clusterctl generate cluster capikind --kubernetes-version v1.23.3 --control-plane-machine-count=3 --worker-machine-count=3 > capi-kind.yaml`. It shows `Error: failed to identify the default infrastructure provider. Please specify an infrastructure provider`. I'm not using infrastructure. I'm using kind cluster. What infrastructure provider should I specify?
You are running CAPI in a kind cluster and now you are instructing it to create a new cluster. That cluster needs to have nodes and those nodes need to be on some infrastructure. The command you executed says that there should be 3+3 servers.
Not sure what you mean by "register". If you want to run cluster API in that cluster, all you have to do is install it there. But, if you meant that you would like cluster API to take over management of that cluster, I'm afraid that will not work (as far as I know).
You can use something like KinD or any other local k8s to spin up a cluster with ClusterAPI and, afterwards, move the same specs to that cluster. Or, at least, I think that should work. I tend to work almost exclusively with Crossplane these days so I'm more up-to-date with it. In case of Crossplane (and probably ClusterAPI) you might only have issues with a few AWS resources that cannot be named (that generate IDs) and, in case of Crossplane, that is solved by adding those generated IDs as labels. It's probably similar with ClusterAPI.
One year later, I see that cluster API is getting more popular. Many providers support it. And the list is getting longer. Is there a way to use cluster API via Crosspläne?
As far as I know, there is no Cluster API provider for Crossplane. There is probably no technical obstacle but more that no one (yet) go around doing it.
Viktor, great insights as always, speaking of CAPI, any chance that you can do a demo on Anthos or TKG/TCE? wanted to to know if those are great alternatives than implementing by itself.
Can you please show how crossplane can create a cluster from scratch? without a managed k8s cluster. I have not seen it in crossplane, expect that crossplane integrates CAPI to create and operate a cluster,
@@DevOpsToolkit with managed k8s Cluster, I mean a cloud provider managed target cluster like EKS, AKS, GKE etc.. Crossplane might be able to manage such clusters with the respective cloud providers, sure. But the Point with CAPI is that one can build a bare metal edge cluster or any other offering. and in turn Crossplane can use the Cluster API Provided API to create a specific XRD and abstract away many aspects. I see crossplane and CAPI total orthogonal in this respect. And there is also a great hackathon explaining this ;-)
@@DevOpsToolkit Yes, I think that's what he meant, since you were saying on the video that Crossplane creates clusters more easily than Cluster API. I am also interested in seeing that.
Right now, crossplane can only create managed clusters (e.g. EKS, aks, GKE, etc). Or, to be more precise, that's easy to do. If you're looking for self-managed clusters, I would not recommend crossplane or, at least, not without some additional tools. That is especially true if you are running on prem. If you're in cloud, but you want self+managed clusters, you could use crossplane to create infra (e.g. VMw, networking etc.) but you would need something else to setup Kubernetes inside those. For self-managed k8s, I would indeed recommend cluster API or, to be more precise, I would combine it with Rancher. You would still need something else to manage the rest of infra though. I thought that I was referring to cloud when I compared cluster API with crossplane. If I haven't, that's an inintentional omission on my part.
@@DevOpsToolkit exactly that’s what I mean, that’s where CAPI comes into play to make the heavy lifting of actually manage the cluster. From there crossplane can abstract to the higher layer where a CAPI cluster is on the same level as all other managed cluster. Hence to say that crossplane is more easy then CAPI is not really the correct statement, crossplane is great but work on different Level IMHO.
What do you think about Cluster API? Will it become the preferrable way to create, provision, and operate Kubernetes clusters?
IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), TH-cam tends to delete comments that contain links. Please do not use them in your comments.
I learnt earlier that crossplane can spin up monitor and manage any infra as long as the target infra has native API for spinning infra, so can we not extend crossplane to spin up self managed k8s cluster on prem like using vSphere API on VMware for example. Not sure if I am asking a relevant question here.
You are right. Crossplane can manage vSphere. Or, to be more precise, it could manage it, but I'm not sure what is the status of the VMware provider. I'm not using it myself so I haven't been following. If that provider is not yet done, it should be relatively easy to create it through the Terrajet project. The best thing would be to raise an issue in the crossplane repo. That would raise the awereness that users are interested and someone will pick it up. Also, if you'd be interested in learning how to create providers, I can give you a hand.
ClusterClass in CAPI solves the issues that you brought up in the video. Its new and not all providers support it yet, but its a single crd to create a cluster and a single place to upgrade the version. For CNI you can also have it installed automatically via the ClusterResourceSet feature in CAPI
Its also interesting that almost all multi cloud commercial offerings for k8s are based in CAPI today:
Vmware TKG
Rancher
Google Anthos
Amazon EKS-A
OpenShift (partial implementation as they are only using the machine API from CAPI)
That's, in a way, the point I'm trying to make. I think that CAPI will be used as a building block for other solutions rather than directly.
It still makes sense to use CAPI indepedently if you want to avoid vendor lock in. While those commercial multi cloud solutions allow user to take advantage of multiple cloud services, it does so by using vendor specific tools. For some of those solutions it also means you have to do multi cloud management from within a parcicular provider's cloud.
The CABPT project uses CAPI to bootstrap a Talos K8S cluster, from there it's trivial to bootstrap Crossplane, and then using Crossplane as a CAPI provider to boostrap additional K8S clusters, I think.
I like the idea of clusterapi by create new nodes and replace them. I upgraded linux nodes in-place for a long time and kubernetes clusters in place recently. The short version: in-place upgrades suck, replacing nodes is much better
thank you. while watching this video it really made me think more deeply about the types of abstraction i really want from all this kubernetes stuff, all this cloud providers and generally devops api. and it ultimately should come down to seeking 2 things. one of them is costs of services. and the other one is performance provided for those costs. and it should all be through a standardized single grpc api. so in my mind the problem is then a lack of standardization across providers. across these platforms. and maybe kubernetes cluster api and crossplane is a step in the right direction. or using openshift etc.
however i dont think the transparency and coherency is there to provider QoS guarantees or smart automatic billing contract built into the charging sytem over these same apis. for when the service provided is either better or worse than expected. and for to set upper and lower limits we are talking how many nodes here. rather than how many dollars. or how long the ping time. or other network bandwidth metrics etc. across these same api. and indistinguishable between the provider. it seems like a difficult problem to solve. i suppose there is kubecost however it is in itself another seperate 3rd party service to setup, manage and configure. hmm, but i wonder if it is a place to start at. when dreaming up such kinds of mythical grpc apis. and along that journey, to also be examining which common cloud apis are the most coherent / commonly shared. the scope is broad ranging. i wonder how much is not possible to be covered within the domain of kubernetes itself. and it also makes me wonder the true value of all these different things
I started writing a response but quickly realized that it's too big of a subject. How about talking about it in one of the upcoming AMA sessions or, even better, in the monthly chat reserved for the members?
The short version of what I wanted to write: standardization of Cloud will almost certainly not happen any time soon (if ever) and the best we can do is have something "standard" running on top of them (Kubernetes).
@@DevOpsToolkit yes agreed. it is too big a subject. and i totally agree - we will never see any voluntary and 'coherent' to be a well standardized cloud API. (that is simple, logical, and transparent)
essentially the 2 things: the performance and the cost. those are the very things which they want to abstract away the most from the customer. 'dont pay attention to that' or 'we make it complicated and hard to understand on purpose'. it reminds me of ISPs broadband. and mobile phone cellular service. except with cloud we have many more service as a result. they kind of multiply by themselves? like some virus or something IDK
just this insight alone makes me want to make kubecost a higher priotity (for myself). just on my own list of which 3rd party infrastructure addon services are worth to go deeper into. for my own limited available time. because there clearly are no other ways to solve this problem, right? except with something like kubecost.
that focus then lets me a reasonable justification investigate further those related areas. things which are connected to it. for example: how to have a 3rd party monitoring service additions added onto kubecost. possibly as plugins for kubecost. or otherwise at the output side of kubecost. in graphana / prometheus metrics. and connecting the cost to service performance. as some chosen and measured performance metrics.
this could be a fun project while learning kubernetes. if planned ahead and broken down into smaller chunk. with stage1 building upon to the next stage2 and so on.
Just published a video about KubeCost: th-cam.com/video/GpChs_4exxQ/w-d-xo.html
You can also provision AKS via CAPI in terms of managed clusters
You're right. I saw it in capz.sigs.k8s.io/topics/managedcluster.html after I made the video. The docs are a bit confusing and all over the place. If you go to cluster-api.sigs.k8s.io/reference/providers.html there is no mention of AKS. It's also not mentioned in github.com/kubernetes-sigs/cluster-api-provider-azure. So, one needs to do a bit of digging. Nevertheless, you're right. AKS is there.
Ya the docs arent the most organized. Typically the best is the CAPI book and the Provider specific books if they have them which not all do. I really hope the docs get better but as we all know docs and naming are probably the hardest things in tech today
cluster api not able to reconcile for docker infrastructure provider, I have created a k8s cluster using kind.
Initialized it using clusterctl init --infrastructure docker and apply one CAPI manifest for 3 control-node + 3 worker node. It was successfully deployed, all the machines were up (docker containers).
After that I am trying to test the benefits of doing this whole thing using CAPI (Reconciliation, full lifecycle management of workload clusters).
For that I stopped 1 container(worker node machine), So even though Cluster CAPI detects the change and starts the reconciliation process (this I can say by checking the capd-controller-manager pod logs, it was not able to set up the worker node. I expected it to launch another container and add it to the cluster but it's not able to do that.
There is no issue with memory, I have increased the inotify limit.
Please help me debug.
Thanks for this video :) Can you make an updated version of this video please? Or can you just mention in the reply what is the current state CAPI? Are the CONS mentioned here been solved. Most importantly cluster update/upgrade?
CAPI is alive and kicking and advanced a lot since i made that video. If you manage only kubernetes clusters, it is s very good choice. If you need more, i would recommend crossplane.
@@DevOpsToolkit 👍
@DevOpsToolkit, If CAPI can bootstrap Crossplane, and then Crossplane can bootstrap Talos, wouldn't this become a golden path for IDP/PaaS?
Awesome! I was really looking forward to seeing you talk about cluster API!
I created kind cluster. I did clutserctl init. Now I run the command `clusterctl generate cluster capikind --kubernetes-version v1.23.3 --control-plane-machine-count=3 --worker-machine-count=3 > capi-kind.yaml`. It shows `Error: failed to identify the default infrastructure provider. Please specify an infrastructure provider`. I'm not using infrastructure. I'm using kind cluster. What infrastructure provider should I specify?
You are running CAPI in a kind cluster and now you are instructing it to create a new cluster. That cluster needs to have nodes and those nodes need to be on some infrastructure. The command you executed says that there should be 3+3 servers.
@@DevOpsToolkit ok
Hey, I have an already running cluster, I would like to register it with the cluster APi, how do I do that ?
Not sure what you mean by "register". If you want to run cluster API in that cluster, all you have to do is install it there. But, if you meant that you would like cluster API to take over management of that cluster, I'm afraid that will not work (as far as I know).
@@DevOpsToolkit yes I wanted cluster API to take over it’s management
You can use something like KinD or any other local k8s to spin up a cluster with ClusterAPI and, afterwards, move the same specs to that cluster. Or, at least, I think that should work. I tend to work almost exclusively with Crossplane these days so I'm more up-to-date with it. In case of Crossplane (and probably ClusterAPI) you might only have issues with a few AWS resources that cannot be named (that generate IDs) and, in case of Crossplane, that is solved by adding those generated IDs as labels. It's probably similar with ClusterAPI.
Can you please compare this with another project? Like, Gardener.
I haven't used gardener in a while now so i need to refresh my memory first and then i can make a video about it.
what if we dont pass `--infrastructure digitalocean` while generating yaml file? whats the difference?
I believe that, in that case, it'll generate generic manifests that you'd need to modify heavily.
One year later, I see that cluster API is getting more popular. Many providers support it. And the list is getting longer.
Is there a way to use cluster API via Crosspläne?
As far as I know, there is no Cluster API provider for Crossplane. There is probably no technical obstacle but more that no one (yet) go around doing it.
Viktor, great insights as always, speaking of CAPI, any chance that you can do a demo on Anthos or TKG/TCE? wanted to to know if those are great alternatives than implementing by itself.
Adding them to my TODO list... :)
Tce is amazing!!!!
Best distro based on CAPI by far!!!!!
Can you please show how crossplane can create a cluster from scratch? without a managed k8s cluster. I have not seen it in crossplane, expect that crossplane integrates CAPI to create and operate a cluster,
I'm not sure what you mean by "without a managed k8s cluster"? Do you mean a cluster that is not GKE, EKS, AKS, or something similar?
@@DevOpsToolkit with managed k8s Cluster, I mean a cloud provider managed target cluster like EKS, AKS, GKE etc.. Crossplane might be able to manage such clusters with the respective cloud providers, sure. But the Point with CAPI is that one can build a bare metal edge cluster or any other offering. and in turn Crossplane can use the Cluster API Provided API to create a specific XRD and abstract away many aspects. I see crossplane and CAPI total orthogonal in this respect. And there is also a great hackathon explaining this ;-)
@@DevOpsToolkit Yes, I think that's what he meant, since you were saying on the video that Crossplane creates clusters more easily than Cluster API. I am also interested in seeing that.
Right now, crossplane can only create managed clusters (e.g. EKS, aks, GKE, etc). Or, to be more precise, that's easy to do. If you're looking for self-managed clusters, I would not recommend crossplane or, at least, not without some additional tools. That is especially true if you are running on prem. If you're in cloud, but you want self+managed clusters, you could use crossplane to create infra (e.g. VMw, networking etc.) but you would need something else to setup Kubernetes inside those.
For self-managed k8s, I would indeed recommend cluster API or, to be more precise, I would combine it with Rancher. You would still need something else to manage the rest of infra though.
I thought that I was referring to cloud when I compared cluster API with crossplane. If I haven't, that's an inintentional omission on my part.
@@DevOpsToolkit exactly that’s what I mean, that’s where CAPI comes into play to make the heavy lifting of actually manage the cluster. From there crossplane can abstract to the higher layer where a CAPI cluster is on the same level as all other managed cluster. Hence to say that crossplane is more easy then CAPI is not really the correct statement, crossplane is great but work on different Level IMHO.
Testing a comment without link
It works without links most of the time.
Thanks for the awesome work!!! 👌
Just found your channel and I'm gonna binge watch 😀
I managed to hack his cluster