Thank you for the nice video! In Percona we have 4 database operators, 2 are tested with a custom framework written in Bash, and 2 are tested with Kuttl. Since Kuttl is almost dead we are currently considering either moving to Chainsaw or writing something in python or golang.
@TomislavPlavcic I would suggest giving Chsinsaw a try. It might fit your needs or it might not. If it doesn't, the question is whether it's easier to add that something to the project or start from scratch. More often than not, it is the former.
Thanks for another great video! I don't particularly like the approach of writing more and more yaml but it's definitively good news having a new player in the scene for kubernetes testing
I just started to study Kubernetes, and I just stuck with my on site cluster. I need to make a network among few pcs across different places. On all pcs instances are running on multipass instances. How to connect those instances and make 1 cluster form multiple nodes? I can see that control plane receives node IP’s that are locally assigned by multipass instances? Thank you
It seems Chainsaw could be very useful for rapidly writing operator tests... Declare things that you want to create (or patch, delete etc...) in the cluster, assert the operator creates the CRDs that it manages... Unfortunately for what I'm working on right now, it does not seem to fulfill my needs... I'm working on a Controller that watches over Services with some specifics Annotations and makes requests to an external API... So the Controller does not make any changes to the cluster... I was looking for something that would ease the pain of having to manually create mocks everywhere for me to test the Reconcile loop but chainsaw does not seem to solve that
Chainsaw does not solve all that. You still need to test that your code (doesn't matter whether it is running in a controller or elsewhere) makes correct API calls. You probably do not need even to run it in kubernetes for that.
Suppose you have a Composition with dependencies on extra resources, for example a Composition may read the status.atProvider field of a Managed Resource which doesn't belong to the Compositon itself. How do you test this with Chainsaw?
@@DevOpsToolkit Just to make sure I understand. So let's say I have composition A and B. B wants to read the status.atProvider of a resource created in A. If I want to write a test, I would need to have an existing Claim of A in my cluster with real resources created in order to fill in the status.atProvider field. Finally I could create a Claim of B and then the test could work, right?
Crossplane resources are always created, at least in my case. However, most of the time I do not setup ProviderConfigs with credentials to AWS/GCP/Azure. The result is the issue you're mentioning. Data that comes from those is never put into MRs since MRs never managed to talk to AWS/GCP/Azure. If you do want to test them, you do need to create ProviderConfigs and, in those cases, MRs will be creating and destroying "real" AWS/GCP/Azure resources. That takes time and costs money so I do that rarely. I would have "special" tests dedicated only to that and run them only when needed (when I do something that would change that behavior). I'm not sure whether I managed to explain myself or whether I understood your question. Ping me if I didn't and I'll send you a link if you'd like to talk about it over a call and, maybe, do screen share.
I guess that the answer will differ depending on ones use case. In my case, primary objective is to confirm that resource expansion works. If i create s Deployment, i want to confirm that it created a replicaset which created pods which requested storage, etc.
@@DevOpsToolkit here's what I have trouble understanding: so the deployment resource expansion (replicaset, pods etc.) is k8s functionality right? aim should be testing your own cluster configuration, correct? so assertion in tests should be covering cluster specifics? e.g. a negative case would be: we have a deployment which is configured with data that is not viable to our cluster configuration (e.g. user has no rights to create resources in a particular namespace), hence it should not expand resources.
@mmihailicenko normally, i would test that the policías ir rbac that' prevent user from doing what should not be done is indeed there. You can also take it the other way around and test that the resource user tried to create was not created. P.S. Plesse let me know if i misunderstood your question.
@@DevOpsToolkit almost, (sorry for a bit misleading example) more interested in k8s test coverage concept. When you have to cover cluster with tests from scratch how do you determine what to automate?
@mmihailicenko i rarely think of a cluster as a whole but of each individual project. The examples you saw in the video are like that'. They are from a Crossplane project so I'm ensuring that all the resources that composition males are done correctly.
Do you know any good tools for test data management? Usually its about data in databases but not only. Generating, subsetting, versioning, anonymizing production data. So far ive tested Redgate's tools and i have mixed opinion about them. Anonymizer and DB Clone are good but generator and subsetter are not. There is also Delphix but i dont know it. Are there any opensource solutions?
We don't test our manifests, but we want. But this is not enough for us. We don't want to create real database. What should we test? Only the manifest applied? I think we need to precondition step where we install our crds etc. Testing manifests is really difficult and I cannot imagine a good way here
You can apply anything, including CRDs. Whether you'll test against a real database or not depends. It's the same dillema people have with any other type of testing. Sometimes you want to mock or simulate dependencies and at other times you want the "real deal". Ideally, you'll have both and use mock or simulations while developing so that it is very fast and run a full suite through CI pipelines. I tend to use Chainsaw for testing my Crossplane compositions. While writing them, I avoid creating configurations so that Crossplane cannot create "real" resources like databases. That allows me to practive TDD since tests take only a few seconds (unless there is an assert failure). Once I'm done I push my changes to Git, that initiates GitHub Actions build during which I am testing using "real" resources in AWS, GCP, and Azure. That takes quite some time, but that's okay since those tests rarely fail so I don't need to wait for them. Most of the issues are encountered locally while writing code.
chainsaw seems nice, but to be honest the example presented is so complex and tightly coupled to CrossPlane which becomes not understandable :( it would have been better a simple but complete deployment (configmap, secret, deployment, maybe pvc, service, pdb)... CrossPlane is nice, but it cannot be used as example for everything! Especially if someone doesn't know its syntax like me... Here the focus should be how chainsaw works, not the tons of CrossPlane yaml manifests in another format...
I used Crossplane compositions as an example simply because that's one of the projects i use chainsaw. Still, you're right. I should have created a demo repo.
![ILLSLICK - KILLSHOT REMIX [FULL VERSION]](http://i.ytimg.com/vi/RIK8RrqIAzE/mqdefault.jpg)
Do you test your Kubernetes manifests/resources? If you do, what do you use?
Thank you for the nice video! In Percona we have 4 database operators, 2 are tested with a custom framework written in Bash, and 2 are tested with Kuttl. Since Kuttl is almost dead we are currently considering either moving to Chainsaw or writing something in python or golang.
@TomislavPlavcic I would suggest giving Chsinsaw a try. It might fit your needs or it might not. If it doesn't, the question is whether it's easier to add that something to the project or start from scratch. More often than not, it is the former.
Thanks for another great video! I don't particularly like the approach of writing more and more yaml but it's definitively good news having a new player in the scene for kubernetes testing
An awesome video like usual ❤
Anther Awesome one !!!
I just started to study Kubernetes, and I just stuck with my on site cluster. I need to make a network among few pcs across different places. On all pcs instances are running on multipass instances. How to connect those instances and make 1 cluster form multiple nodes?
I can see that control plane receives node IP’s that are locally assigned by multipass instances? Thank you
Unfortunately, I work mostly with managed-Kubernetes (e.g., GKE, EKS, AKS, etc.) so I might not be the right person to help with that one.
It seems Chainsaw could be very useful for rapidly writing operator tests... Declare things that you want to create (or patch, delete etc...) in the cluster, assert the operator creates the CRDs that it manages...
Unfortunately for what I'm working on right now, it does not seem to fulfill my needs... I'm working on a Controller that watches over Services with some specifics Annotations and makes requests to an external API... So the Controller does not make any changes to the cluster... I was looking for something that would ease the pain of having to manually create mocks everywhere for me to test the Reconcile loop but chainsaw does not seem to solve that
Chainsaw does not solve all that. You still need to test that your code (doesn't matter whether it is running in a controller or elsewhere) makes correct API calls. You probably do not need even to run it in kubernetes for that.
Suppose you have a Composition with dependencies on extra resources, for example a Composition may read the status.atProvider field of a Managed Resource which doesn't belong to the Compositon itself. How do you test this with Chainsaw?
In those cases you would have to set providerconfig so that destination resources are created. I run those rarely since they require much more time.
@@DevOpsToolkit Just to make sure I understand. So let's say I have composition A and B. B wants to read the status.atProvider of a resource created in A. If I want to write a test, I would need to have an existing Claim of A in my cluster with real resources created in order to fill in the status.atProvider field. Finally I could create a Claim of B and then the test could work, right?
Crossplane resources are always created, at least in my case. However, most of the time I do not setup ProviderConfigs with credentials to AWS/GCP/Azure. The result is the issue you're mentioning. Data that comes from those is never put into MRs since MRs never managed to talk to AWS/GCP/Azure. If you do want to test them, you do need to create ProviderConfigs and, in those cases, MRs will be creating and destroying "real" AWS/GCP/Azure resources. That takes time and costs money so I do that rarely. I would have "special" tests dedicated only to that and run them only when needed (when I do something that would change that behavior).
I'm not sure whether I managed to explain myself or whether I understood your question. Ping me if I didn't and I'll send you a link if you'd like to talk about it over a call and, maybe, do screen share.
@@DevOpsToolkit thank you, I'd definitively appreciate a talk around this if you have a time slot to dedicate to me tomorrow
@@IvanRizzante Please ping me on LinkedIn (www.linkedin.com/in/viktorfarcic/) or BlueSky (vfarcic.bsky.social/) and I'll send you my calendar link.
How do you determine testing scope? How to test your k8s cluster specifics and not k8s functionality?
I guess that the answer will differ depending on ones use case. In my case, primary objective is to confirm that resource expansion works. If i create s Deployment, i want to confirm that it created a replicaset which created pods which requested storage, etc.
@@DevOpsToolkit here's what I have trouble understanding: so the deployment resource expansion (replicaset, pods etc.) is k8s functionality right? aim should be testing your own cluster configuration, correct? so assertion in tests should be covering cluster specifics? e.g. a negative case would be: we have a deployment which is configured with data that is not viable to our cluster configuration (e.g. user has no rights to create resources in a particular namespace), hence it should not expand resources.
@mmihailicenko normally, i would test that the policías ir rbac that' prevent user from doing what should not be done is indeed there. You can also take it the other way around and test that the resource user tried to create was not created.
P.S. Plesse let me know if i misunderstood your question.
@@DevOpsToolkit almost, (sorry for a bit misleading example) more interested in k8s test coverage concept. When you have to cover cluster with tests from scratch how do you determine what to automate?
@mmihailicenko i rarely think of a cluster as a whole but of each individual project. The examples you saw in the video are like that'. They are from a Crossplane project so I'm ensuring that all the resources that composition males are done correctly.
Do you know any good tools for test data management? Usually its about data in databases but not only. Generating, subsetting, versioning, anonymizing production data. So far ive tested Redgate's tools and i have mixed opinion about them. Anonymizer and DB Clone are good but generator and subsetter are not. There is also Delphix but i dont know it. Are there any opensource solutions?
Unfortunately, that's one of the areas I never got into deep so I can't recommend anything :(
We don't test our manifests, but we want. But this is not enough for us. We don't want to create real database. What should we test? Only the manifest applied? I think we need to precondition step where we install our crds etc. Testing manifests is really difficult and I cannot imagine a good way here
You can apply anything, including CRDs.
Whether you'll test against a real database or not depends. It's the same dillema people have with any other type of testing. Sometimes you want to mock or simulate dependencies and at other times you want the "real deal". Ideally, you'll have both and use mock or simulations while developing so that it is very fast and run a full suite through CI pipelines.
I tend to use Chainsaw for testing my Crossplane compositions. While writing them, I avoid creating configurations so that Crossplane cannot create "real" resources like databases. That allows me to practive TDD since tests take only a few seconds (unless there is an assert failure). Once I'm done I push my changes to Git, that initiates GitHub Actions build during which I am testing using "real" resources in AWS, GCP, and Azure. That takes quite some time, but that's okay since those tests rarely fail so I don't need to wait for them. Most of the issues are encountered locally while writing code.
chainsaw seems nice, but to be honest the example presented is so complex and tightly coupled to CrossPlane which becomes not understandable :( it would have been better a simple but complete deployment (configmap, secret, deployment, maybe pvc, service, pdb)...
CrossPlane is nice, but it cannot be used as example for everything! Especially if someone doesn't know its syntax like me... Here the focus should be how chainsaw works, not the tons of CrossPlane yaml manifests in another format...
I used Crossplane compositions as an example simply because that's one of the projects i use chainsaw. Still, you're right. I should have created a demo repo.