Yes, we are heavily use it to define our app stack, but we use helm based installation for system components, terraform for infra. Considering you can define terraform using jsom my goal is to consolidate all of it using single configuration model: app stack, system stack and even infrastructure definitions using JSONNET. Surprised you guys only getting to it now 😜.
@@joebowbeer our app stack is rendered in pipeline on the deployment branches using gitactions automagically, these never get merged to master. and argocd just deploys from particular branch/folder
I've been using cuelang and jsonnet for general configurations and they work much better than kustomize, i love to simply evaluating the values, but also ranges where these are valid
Really glad you made a video on this topic. One thing you didn't mention explicitly, is the potential big customizability advantage of jsonnet libraries over 3rd party Helm charts. No more forking or needing to apply kustonize patches on top of Helm as soon as the chart doesn't fullfil your needs. I wish more companies with complex projects offered jsonnet libraries instead of inflexible Helm charts.
I agree. However, as you said, not many companies offer their third-party apps as Jsonnet packages/libraries. So, that means that we're stuck with Helm, or we need to write manifests (in Jsonnet) for third-party apps ourselves, which is too error-prone and represents too much maintenance work.
Thanks for share it! I have read about another Open Source project in way the called Acorn. This project has started from the co-founders of Rancher before suse's adqusition
I already have Acorn in my TODO list... :) I would not compare it with Jsonnet but, rather Docker Compose (for Kubernetes) combined with, let's say, Tilt.
Regarding this mentioned advantage "at scale", I think that Helm handles that well too. A company cam develop a Library Chart that all teams should use. Thanks to that, it is possible to provide fast updates to all charts at once and enforce some good practices (label for team) as well.
It is certainly possible to do something similar with Helm, but that's more of a workaround than a solution baked into the architecture. P.S. I am not convinced about Jsonnet myself, but, for now, just listing the advantages.
Well clearly by default jsonnet (or tanka specifically?) only seems to output json or yaml. and that is what they seem to say and promote on their website. however in fact the jsonnet language also includes quite a few standard library functions that can output the jsonnect source data into certain other formats. including (but not limited to?)... ini, toml, python formatted data, and some types or subset of xml. Those seem to be the supported custom types. Of course because (as it says) the versatility of being able to convert json to other formats. However what other 3rd party libraries could be out there idk but you could expect to want one for nginx config files, which is similar to C language. But i think the real value here is that you can make custom outputters and do that all within jsonnet language itself by writting native jsonnet lbraries. And share those libraries to any other users of jsonnet. Without having to bolt on or cobble together some non standard hacks or shell scripts as an extra runtime step to launch after generating the json output from jsonnet. So that i think adds a real value especially for having a single configuration language for not just the k8s configs. But also for all of the 'whatever applications' and everything else inbetween. That can be all driven from single source shared definitions. I am not sure how that fits with tanka feature set as a cli tool. If tanka supports all of that and the other output formats. However there might be other jsonnet tooling with does that. Finally another alternative to jsonnet is supposed to be CUE maybe? But it seems more abstracted and farther off than jsonnet. More abstracted. So that is less clear to me how it compares / where it fits into the wider picture.
The Dashboards As Resources in Kubernetes (DARK) project is an interesting approach: it adds a GrafanaDashboard custom resource and provides a tool to convert json dashboards into custom resource manifests. Does Crossplane have a CRD for GrafanaDashboard?
I haven't seen such a CRD in Crossplane. That does not necessarily mean that there isn't one. It's OSS and there are quite a few Crossplane providers floating around but not being pushed into the crossplane org.
After using jsonnet with drone ci, I can see the power, but no one else on my team wanted to learn it to contribute. It's far easier to sift through yaml than working through how jsonnet wires everything up. I will say it was nice to add 4 lines to a jsonnet file and generate 90 lines of yaml.
That's a common theme I saw with other companies/teams. One or two people might jump into Jsonnet but it is rarely accepted by the rest of the company.
Very nice video, and very good points for pros/cons as someone who did some work in jsonnet land. Just wondering: when you mention to create CRDs, do you have crossplane in mind or are they other ways?
I have crossplane in mind mostly because that's my choice. There are others . I will publish at least two videos on that subject (and unrelated to crossplane) in the coming weeks.
I really liked the approach of using jsonnet and tanka. This is very clear to me how to apply this to generate my own manifests or in-house applications but if I have the helm chart of a nginx-controller where the chart is maintained by a third party, how I could apply jsonnet to override the values file file without touching the helm templates?
Hey viktor, I've been recently using cdktf for this purpose and falling in love with it so much that I replace our whole terraform stack with it. Have you looked into it yet?
Great video as always. What would you use to put Kubernetes behind VPN ? I was thinking about it recently and was not sure, what would be the correct solution... I can side load vpn sidecard to deployments... but that does not sound correct... what if I have traefik ingress for incoming traffic, and I put that behind vpn, how I would go and address is without internal DNS.. ? Or should I just put VPN on each node NW, in the OS ?
@@DevOpsToolkit Teleport is not bad, but last time I asked about price for enterprise edition (we needed azure ad integration) the price was waaay to high. I love the session recording... but still vpn is "free" and already setup through company.
Never use Kanka but from first view it looks like a mess ;o) I have used a little bit of Kustomize, which seems to be quite hacky, but mostly using Helm which seems to be perfect because everything is versioned (chart and subcharts), packaged (all subcharts are part of the parent chart - you don't care if somebody deletes the Helm registry for your depndencies as you have all depending charts packaged with your chart), chart can be signed and it uses Golang templating! You can also write a Helm chart that's actually a library chart - super handy to keep your chart code DRY.
I don't think jsonnet+tanka is a replacement of helm or kustomize. All those tools have different advantages. Each of them is nice solving a specific problem, but none of them is the silver bullet solving everything and making our life way easier. A meta-language on top of JSON makes everything complex and prone to human mistakes jsonnet+tanka are nice if you have a bunch of applications with almost same yaml structure, but if not all exceptions are going to make things harder to define, harder to read, harder to maintain
prone to human mistakes - JSONNET its not JSON, it depends how you write it, you can actually use test functions for validations of proper data structure, you can enforce scemas. Developing helm templates on the other hand is more dangeratous (thou simpler and more readable), one wrong indentation and its gone.
I disagree, helm templates are much more error-prone. As for kustonize, it depends, if your project is decently complex, it becomes much harder to reason about. Also, including a bunch of exceptions would just be someone using jsonnet poorly. Just like with any other language, you shouldn't try to create an extremely broad codebase used by tons of projects which will devolve in a bunch of exceptions. Same thing would apply to someone trying to create a extremely re-usable base helm chart. Either way, this is just bad programming, not bad tooling.
Tbh, I hate jsonnet. The only reason why I even have to learn it was because the folks who maintain Prometheus operator use jsonnet. Didn't we all start embracing config over imperative programming for simplicity and improved sanity? Who had the bright idea to bring programming back into configuration management? I really couldn't understand what problems they were trying to solve by using jsonnet.
Too much fuzzy just to extend a data first JSON to do some things a real language with much more flexibility and readability do, and another syntax to learn. I wouldn't explore that. There is a alternative called pulumi, which is like terraform but with language support like typescript, python, C#, Java and Yaml. I prefer to use pulumi which is not only restricted to kubernetes and offers full language support. But in many cases kustomize and helm are just fine.
What do you think of Jsonnet and Tanka? Could it accomplish it's goal of replacing Helm and/or Kustomize?
Yes, we are heavily use it to define our app stack, but we use helm based installation for system components, terraform for infra. Considering you can define terraform using jsom my goal is to consolidate all of it using single configuration model: app stack, system stack and even infrastructure definitions using JSONNET. Surprised you guys only getting to it now 😜.
How does ArgoCD's first-class support for jsonnet affect decision? Better to render client-side or let ArgoCD render in the cluster?
I tend to render it on client-side or, to be more precise, in pipelines.
@@joebowbeer our app stack is rendered in pipeline on the deployment branches using gitactions automagically, these never get merged to master. and argocd just deploys from particular branch/folder
I get a kapitan feeling but feels less innovating than kpt... Am I lost ? LOL
I'm old enough to remember when the Jsonnet folks also wanted to push Ksonnet as future and Helm as the past. That didn't turned out as planned..
I've been using cuelang and jsonnet for general configurations and they work much better than kustomize, i love to simply evaluating the values, but also ranges where these are valid
Really glad you made a video on this topic. One thing you didn't mention explicitly, is the potential big customizability advantage of jsonnet libraries over 3rd party Helm charts. No more forking or needing to apply kustonize patches on top of Helm as soon as the chart doesn't fullfil your needs. I wish more companies with complex projects offered jsonnet libraries instead of inflexible Helm charts.
I agree. However, as you said, not many companies offer their third-party apps as Jsonnet packages/libraries. So, that means that we're stuck with Helm, or we need to write manifests (in Jsonnet) for third-party apps ourselves, which is too error-prone and represents too much maintenance work.
Interesting video Viktor! Thx for creating! IMHO I would choose cuelang over jsonnet in 2022!
As always, very informative! Thank you Victor! ❤
Thanks for share it! I have read about another Open Source project in way the called Acorn. This project has started from the co-founders of Rancher before suse's adqusition
I already have Acorn in my TODO list... :)
I would not compare it with Jsonnet but, rather Docker Compose (for Kubernetes) combined with, let's say, Tilt.
@@DevOpsToolkit Yes, please!
It's finally done: th-cam.com/video/uj6Gan10Wig/w-d-xo.html
Regarding this mentioned advantage "at scale", I think that Helm handles that well too. A company cam develop a Library Chart that all teams should use. Thanks to that, it is possible to provide fast updates to all charts at once and enforce some good practices (label for team) as well.
It is certainly possible to do something similar with Helm, but that's more of a workaround than a solution baked into the architecture.
P.S. I am not convinced about Jsonnet myself, but, for now, just listing the advantages.
Well clearly by default jsonnet (or tanka specifically?) only seems to output json or yaml. and that is what they seem to say and promote on their website. however in fact the jsonnet language also includes quite a few standard library functions that can output the jsonnect source data into certain other formats. including (but not limited to?)... ini, toml, python formatted data, and some types or subset of xml. Those seem to be the supported custom types. Of course because (as it says) the versatility of being able to convert json to other formats. However what other 3rd party libraries could be out there idk but you could expect to want one for nginx config files, which is similar to C language. But i think the real value here is that you can make custom outputters and do that all within jsonnet language itself by writting native jsonnet lbraries. And share those libraries to any other users of jsonnet. Without having to bolt on or cobble together some non standard hacks or shell scripts as an extra runtime step to launch after generating the json output from jsonnet. So that i think adds a real value especially for having a single configuration language for not just the k8s configs. But also for all of the 'whatever applications' and everything else inbetween. That can be all driven from single source shared definitions. I am not sure how that fits with tanka feature set as a cli tool. If tanka supports all of that and the other output formats. However there might be other jsonnet tooling with does that.
Finally another alternative to jsonnet is supposed to be CUE maybe? But it seems more abstracted and farther off than jsonnet. More abstracted. So that is less clear to me how it compares / where it fits into the wider picture.
Well said about jsonnet. It's power is in libraries.
CUE is interesting. Planning to do a video about it soon.
The Dashboards As Resources in Kubernetes (DARK) project is an interesting approach: it adds a GrafanaDashboard custom resource and provides a tool to convert json dashboards into custom resource manifests.
Does Crossplane have a CRD for GrafanaDashboard?
I haven't seen such a CRD in Crossplane. That does not necessarily mean that there isn't one. It's OSS and there are quite a few Crossplane providers floating around but not being pushed into the crossplane org.
I did not know about DARK until now. It looks very interesting. Adding it to my TODO list... :)
You can also check out, if you haven't already, the grafana-operator, it adds GrafanaDashboarda, GrafanaDatasource, CRDs etc etc, very cool project
I did. It's awesome but, unfortunately, it remains one of the very few third party apps provided through jsonnet.
After using jsonnet with drone ci, I can see the power, but no one else on my team wanted to learn it to contribute. It's far easier to sift through yaml than working through how jsonnet wires everything up.
I will say it was nice to add 4 lines to a jsonnet file and generate 90 lines of yaml.
That's a common theme I saw with other companies/teams. One or two people might jump into Jsonnet but it is rarely accepted by the rest of the company.
Very nice video, and very good points for pros/cons as someone who did some work in jsonnet land.
Just wondering: when you mention to create CRDs, do you have crossplane in mind or are they other ways?
I have crossplane in mind mostly because that's my choice. There are others . I will publish at least two videos on that subject (and unrelated to crossplane) in the coming weeks.
Great, looking forward to those
I really liked the approach of using jsonnet and tanka.
This is very clear to me how to apply this to generate my own manifests or in-house applications but if I have the helm chart of a nginx-controller where the chart is maintained by a third party, how I could apply jsonnet to override the values file file without touching the helm templates?
That's the main issue with jsonnet or any other helm competitor. Third party charts are, for now, irreplacible.
but, do i need to pain my nails to use jsonnet?
It's a bonus, especially if you have a 12 year old daughter who likes doing that.
Hey viktor, I've been recently using cdktf for this purpose and falling in love with it so much that I replace our whole terraform stack with it. Have you looked into it yet?
I haven't tried it (yet). Right now, I'm more focused on crossplane, CAPI, and similar tools that manage infra through k8s resources.
Great video as always. What would you use to put Kubernetes behind VPN ? I was thinking about it recently and was not sure, what would be the correct solution... I can side load vpn sidecard to deployments... but that does not sound correct... what if I have traefik ingress for incoming traffic, and I put that behind vpn, how I would go and address is without internal DNS.. ? Or should I just put VPN on each node NW, in the OS ?
You can try ditching VPN altogether and use something like Teleport (th-cam.com/video/zVEbml1IAOQ/w-d-xo.html) to access your apps.
@@DevOpsToolkit Teleport is not bad, but last time I asked about price for enterprise edition (we needed azure ad integration) the price was waaay to high. I love the session recording... but still vpn is "free" and already setup through company.
@@vladoportos It's been years since I worked with a system behind a VPN so I'm not sure I can help with that one :(
@@DevOpsToolkit no worries, I'll figure something out.. I also do not put kubernetes behind vpn normally 😀
How this fits to GitOps? Tools like ArgoCD deploy Helm or Kustomize.
Argo CD supports jsonnet. Also, there is the option to output jsonnet to yaml to store it in git.
Never use Kanka but from first view it looks like a mess ;o) I have used a little bit of Kustomize, which seems to be quite hacky, but mostly using Helm which seems to be perfect because everything is versioned (chart and subcharts), packaged (all subcharts are part of the parent chart - you don't care if somebody deletes the Helm registry for your depndencies as you have all depending charts packaged with your chart), chart can be signed and it uses Golang templating! You can also write a Helm chart that's actually a library chart - super handy to keep your chart code DRY.
This looks too complicated. I don't think most users need this. Putting logic in config files is ugly plus this is another DSL to learn. No, just NO!
is it just me or you lowered your tempo for this vid Viktor? :)
Not intentionally.... It might be one of the videos I recorded around 3am so it could be simply tiredness.
I don't think jsonnet+tanka is a replacement of helm or kustomize. All those tools have different advantages. Each of them is nice solving a specific problem, but none of them is the silver bullet solving everything and making our life way easier.
A meta-language on top of JSON makes everything complex and prone to human mistakes
jsonnet+tanka are nice if you have a bunch of applications with almost same yaml structure, but if not all exceptions are going to make things harder to define, harder to read, harder to maintain
prone to human mistakes - JSONNET its not JSON, it depends how you write it, you can actually use test functions for validations of proper data structure, you can enforce scemas. Developing helm templates on the other hand is more dangeratous (thou simpler and more readable), one wrong indentation and its gone.
I disagree, helm templates are much more error-prone. As for kustonize, it depends, if your project is decently complex, it becomes much harder to reason about. Also, including a bunch of exceptions would just be someone using jsonnet poorly. Just like with any other language, you shouldn't try to create an extremely broad codebase used by tons of projects which will devolve in a bunch of exceptions. Same thing would apply to someone trying to create a extremely re-usable base helm chart. Either way, this is just bad programming, not bad tooling.
# T.I.L
Tbh, I hate jsonnet. The only reason why I even have to learn it was because the folks who maintain Prometheus operator use jsonnet. Didn't we all start embracing config over imperative programming for simplicity and improved sanity? Who had the bright idea to bring programming back into configuration management? I really couldn't understand what problems they were trying to solve by using jsonnet.
I think cuelang is much better.
Cue is getting more and more traction and the number of tools that support it or built on top of it is increasing.
Too much fuzzy just to extend a data first JSON to do some things a real language with much more flexibility and readability do, and another syntax to learn. I wouldn't explore that. There is a alternative called pulumi, which is like terraform but with language support like typescript, python, C#, Java and Yaml. I prefer to use pulumi which is not only restricted to kubernetes and offers full language support. But in many cases kustomize and helm are just fine.