Do you have an IDP? If you do, how did you build it? IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), TH-cam tends to delete comments that contain links. Please do not use them in your comments.
One thing I absolutely LOVE about this video is that it is actually prescriptive. I have watched so many platform engineering videos lately and I haven't seen a single one, except to a small extent Dave Farley's on CD, but other than that they're all idealogical philosophy that emphasize fairly obvious tenets instead of actual implementations.
I've been developing an IDP for our engineers and I didn't have a word to call it. Having you articulate it in this way is extremely valuable. Thank you!
I completely agree with every word. In fact, I've built my own IDP without knowing this acronym. It is called Kubero and is open-source and completely free to use. It has a UI, API, and CLI. It is entirely kubernetes native. It also follows the workflows from Heroku. So it is very easy for DevOps to start a new App. It has a Built-in CI/CD and deploy's to Kubernetes with a 'git push' and starts new instances on pull requests.
Thank you, thank you, thank you! This was like a Christmas gift. I’ve been preaching this to the team for months and this puts it all into a great presentation. Well done sir!
Hello Viktor, hi channel. I'm happy to have found your channel. I really don't want to sound like a silly fanboy, but what you laid out in this video makes so much sense. I just started with building my own development platform or system. I did not exactly know, what I need and how to call this, but IDP is pretty much it. It is all about abstracting the complexity away from the customers. In our case these customers are our devs, as you mentioned. My primary obstacle at this point is the fact, that I need to learn how to leverage the Kubernetes API as a cloud native tool for an on premise oriented development and deployment toolkit. A very interesting aspect is the extensibility of the Kubernetes API. I really need to figure out, how to use it and how to build CRD's for tasks like bare metal installations or setting up virtual servers on e.g a single VPS server or on a local machine. No idea, if this is possible, but worth a try imho. Building an IDP with very limited local resources could well turn out to be a mad mans plan, but at least worth trying imho. You mentioned not to build it from scratch but to use and adapt existing tools. Just started to look around and I guess in later videos from you I will find more info. I'm glad you are around. I will sure have to consult this channel quite a lot in the coming months. Thanks, Viktor.
I have no words to describe how I needed this video. IDP is something I want to build but I have no idea, until now, from where I should begin. Thanks for share you knowledge.
Code Rant: Resource Provisioning Complexity Clock 12 o'clock: Clicking buttons in the AWS Management Console 3 o'clock: Calling APIs 6 o'clock: DSL 9 o'clock: GitOps 12 o'clock: Clicking buttons in an internal re-write of the AWS Management Console except now CloudTrail pushes its changes to Git to trigger CI/CD on infrastructure.
I often change my perspective. I think that's healthy and much better than sticking with the same views forever and ever. Very often, I do not stand behind things I said in the past. Tech changes and so do my views.
This so much this. been doing this for years and seem to be the only person in a rather large team doing this. Specifically I wrote an IDP around gitlab-ci to automatically create pipelines :D
This is a massive video, would you say that this is covered in your book(s)? I tried standing up Backstage a while back but realized it quickly took me too far away from what I set out to do. Using the Kubernetes API for everything, simple statement, a lot to really unpack. Great video! Thank you!
None of the things I'm publishing in YT since September last year are covered in my books. It's all-new material and the main reason I switched to TH-cam is to be able to publish faster and more frequently. As for Kube API... That's why Crossplane, KubeVela, Cluster API, and a few other tools are great. Even though they might be using different approaches, they all have in common the objective of extending Kube API with additional CRDs and controllers. Backstage is having a steep initial learning curve and setup. I'll be publishing more about it soon. Finally, we (the industry) are not yet there. There are still pieces missing from the solution I'm proposing. The good news though is that we're getting there very fast. That will be my main focus this year.
@Banana in Condom I wrote 8-9 books so far, but most of them were removed from sales since the tech described in them is now deprecated. Two of those are good though and you can find the info in devopstoolkitseries.com/.
This is so cool. We are actually wanting to build an EDP. External Developer Platform. i.e. a very specialized and opinionated platform, but allowing "normal" devs to build applications that are built to be shared and used by multiple tenants (true SaaS) and ready for scale from day one. The only concerns we have currently are how to make things "secure". Your video shows the interfaces that we need to concentrate on. We have some ideas, and this video also supports our initial directions. We want devs pushing to repos, but we don't want them messing with custom resources. They need to be "standardized" for any and all developers. One of our many, many challenges.
You can never create a standard for all developers. That would be like having a goal to reach infinity. Instead, you can either create an opinionated platform that serves some use cases (e.g., Heroku) or create tools that enable ops to create opinionated platforms used by devs. I believe that we need both types of solutions. P.S. Is what you're building public? If it is, it would be great if I could take a look.
@@DevOpsToolkit Yes. It will be very opinionated. And no, nothing public yet. It will be a while before the platform can be made public. :) We'll be using it internally first (i.e. dog fooding) and once we have some initial apps to be able to deploy for others to use (i.e. for winning hearts and minds of developers and pocket books of customers), we'll be making it public.
Please consider covering advanced argocd topics like helm secrets and config plugin management, these topics are rarely covered in TH-cam. Also you’re the best 🙌🏽
@@DevOpsToolkit also please consider covering advantages and disadvantages of using flux and argo. I know there is a movement for gitops but there are quite a few pain point that I feel are never addressed by the community such as multi environment + multi region deployment modeling in this format. Most teams and developers I've seen talking about this have resorted to hackish ways to resolve this problem and almost (if not all) tutorials around argo never get around to covering this
A fantastic presentation Viktor! I will now ask all of my engineers and anyone I coach to watch this video. The information presented here was extremely well done.
Completely agree with your sentiments about K8s being so much more than a container orchestrator and becoming a universal API (and cloud OS). Also agree and look forward to UI tools like Rancher changing from altering state directly to pushing to git instead. I guess (and hope) that's why Rancher is building Fleet!
That's my hope for Rancher as well. If they do start using Git as config/manifest storage and Fleet to synchronize it, they will have a killer solution. The only thing that makes me a bit pessimistic is that Fleet is still far from being a good solution. Let's hope they get there.
Great video @viktor , so insightful! we have tried acheivinig the same thing using Crossplane, argocd and airflow. But at the moment (6 months back), we didnt have few cloud components supported in Crossplane for AWS (EC2 instances and Elasticsearch to be more specific), so we have just put the efforts on hold, once crossplane supports more components we will definetly resume the efforts from we left off.
A few months ago, Crossplane released Jet project that allows people to generate Crossplane providers using code generation based on Terraform modules. As a result, all AWS, Azure, and GCP resources are now supported by separate (Jet) providers. Quite a few other vendors are now supported as well and, even if something you need is still missing, it's very easy to create it. You might want to try it out again.
I 100% agree that IDPs are going to explode. I had a chat with Kaspar (humanitec) back in 2020 about their product and this whole term 'internal developer platform' idea. I was exploring moving dev envs to ephemeral containers on k8s and I needed a simple way for devs to launch applications. I had come across Backstage at the time at well. Unfortunately, I was kicked off that project and it was given to some ops folks to figure out. And that's why today devs install kubectl to manage their own vclusters to do any development.
If we look at the 'DevOps' infinity loop - a lot of the products are on the Deploy/Operate/Monitor corner. I'm currently focusing on the build/test/release bit.
I like your energy and mindset. "Define and implement a tool for developers to self-service themselves" I would love to hear your opinion on what is awaiting us in the next 10 years
A terraform module is the exact same thing as a CR and a CRD is exactly the same thing as the golang library backing the terraform module, except unlike the hashicorp public module registry, helm is significantly smaller and less maintained and I have to write the literal CRD code.
Thank you! Awesome video! Very detailed coverage on process and getting from desired state to actual state. Does it have a part 2 which discuss in details on observing the actual state?
Hey Viktor, you've mentioned "numerous studies that show IDPs increase productivity often drastically". Would you happen to have some to share? If so, that would be awesome because it would help to support this idea in future discussions.
Damn! very informative thanks a lot! we are about to deploy one of our apllication on EKS in order to slowly move another 2 or 3 of our apps. Would you recommend to build the IDP from the very beginning or is it overkill to have an IDP for 3 apps? I am still uncertain if IDP is made for intensive production need or if it's also recommended for small needs. I guess it takes a lot of time to put an IDP together. thanks a lot again for the gold mine!
@@DevOpsToolkit i am sorry my question was not clear. what I wanted to simply ask is: Is it relevant to spend time building and IDP if we only have 2 or 3 apps (10% of all our app) on k8s?
Oh yeah. I might not have been clear in that video that I am advocating for using Kubernetes as a control plane no matter where the apps, infra, and services are located. It's about having an API that can manage resources no matter where they are, not about having an API only for the apps running in the same Kubernetes cluster. Among other things, Kubernetes is about managing resources and I'm on a mission to explain that Kubernetes is NOT (only) about runnning applications packaged as container images. So yes. If you should build and IDP, you should do it on top of Kubernetes independently of where those aps are. Kubernetes is our best bet for a control plane right now.
@@DevOpsToolkit Let's say you were pretty clear on that idea. I guess I am lacking the operational logic and experience to make sens of that concept...hard to understand how k8s can be the reference API entrypoint even for things not running in containers. I will try to look for exemple and other use case of k8s but I am afraid it's going to be hard.. let see
@@anthonymc6140 Kubernetes out-of-the-box is mostly focused on containers. However, Noone is supposed to use Kubernetes as-is. Neither the project design nor the community has the goal for people to use it as-is but, rather, as a base on top of which other vendors, projects, or end-users can build whatever makes sense for them to have (e.g., a platform). Kubernetes is all about extensibility through CRDs. For example, when you install Argo CD, you get CRDs like `Application` and `Project`. Almost any other Kubernetes-native project brings its own CRDs and corresponding controllers that do something. Now, what that something is depends on what you install. For example, if you install KubeVirt (watch th-cam.com/video/oO8VEmpojz0/w-d-xo.html) you are getting CRDs and controllers that allow you to manage virtual machines that do not have anything to do with containers. Another example would be Crossplane that, depending on which providers you install, allows you to manage AWS, Azure, Google Cloud, or (almost) any other type of resources that also have nothing to do with containers (watch th-cam.com/video/n8KjVmuHm7A/w-d-xo.html, th-cam.com/video/yrj4lmScKHQ/w-d-xo.html, and th-cam.com/video/AtbS1u2j7po/w-d-xo.html). There are many other examples and it all boils down to two things. What Kubernetes does depends on which CRDs and controllers are running in that cluster so there are no real limits of its scope. Here comes the important part. Kubernetes is not easy. You need to be experienced with it to start extending (initially through third-party tools and later yourself). So, you might not want to jump into building an IDP as the first thing you do in Kubernetes. But, over time, that is your best bet. P.S. When I say that Kubernetes is hard, I must also stress that most of the things Kubernetes does are easier in it than anywhere else. It's just that some advanced stuff might be too much for some no matter whether that "stuff" is done in Kubernetes or elsewhere. So, it's actually easy considering what it does. P.P.S. "Kubernetes is not only about containers" is one of the main reasons why I got involved with Crossplane. P.P.P.S. I already started working on a video that will provide hands-on instructions how to build an IDP. However, it'll probably take ~2 more months until it goes live. It'll be one of those videos that last for 30-45 minutes but require months of preparation.
Large corporations end up with IDPs for everything, they aren't hard to use individually but you just need to know many of them in order to achieve something, you end up with the original problem, it takes a decade to master all the IDPs you need... you end up with jobs like mine where you are part of a product team (dev) as an SME on all the IDPs relevant for the product. Especially important in teams with high turnover rates.
Thanks for this video! Did you take a look at tanzu application platform for the buy part? It tries to be not too opinionated by allowing to swap pieces. A review of TAP would be really appreciated!
Thank you very much from South Korea. And I have one question. How much of freedom should be given to developers? Should they access to all the tools(eg, Jenkins, Harbor..) behind IDP or they just need access to IDP and SCM? If I give access to tools under the IDP, their freedom is maximum but it seems to me little hard to manage. What is your suggestion?
Yes. In my case, when I asked to them, most of developers want IDP only but some of them(less than 5% of whole devlopers) who know well about devops want to full access to the tools behind and that is making decision difficult. That is why I want to know how other companies do.
@@user-ch5hu2kk4c The main purpose of IDPs is to abstract and simplify the tooling and the processes so that people who are not experts in specific areas can still operate. That means that in the ideal situation the IDP layer on top of that tooling abstracts everything just as, for example, AWS abstracts hypervisors from it's users. So, the goal is often not to have access to the internal tooling below IDP. However, almost noone gets to that point 100%. It's very hard to build everything so companies often go step by step. So, for example, if you built option to deploy apps, you do not need access to the tools like pipelines or GitOps. But, if you did not yet build your own solution for observing logs, you have to give people access to Grafana/Loki or whatever is used for that. Also, you need to have the "break the glass" option. The IDP might fail or not provide a solution for specific scenarios and, in that case, having the access to the tools below it is a must. At the end of the day, it's what your users need. If 5% want the direct access to the tools instead of going through the IDP, it means that they feel more comfortable using those tools than what you built. That means that you have 5% of unhappy customers that are not getting what they need.
Hi thanks for the video 🙂 We are planning to build an IDP for creating cicd processes automatically. Our company is currently migrating to Jenkins and we are debating whether to use this IDP just for creating the initial jenkins files or also to enforce the changes through the IDP. In that way the developers will not be able to change things directly to the Jenkins files. What do you think is the right thing to do?
I do not think that enforcement should be done in pipelines but on the server-side. To begin with, pipelines might not be the only processes that make changes to the system. Also, such an approach severely limits the "freedom" of those using or defining the pipelines. So, instead of, for example, forcing execution of security scanning in a pipeline, move it to the registry where artifacts are pushed, clusters where apps are deployed, etc.
Working on a laptop workflow right now. Out of interest: What are we using these days for CICD? I see a lot of CD videos.(argo,flux,pulumi,crossplane,terraform)
I will vote for Terrform for GKE and Nodepool manage + bring up ArogoCD then ArgoCD to config whole cluster by HelmChart +Vault for secret Argoworkflow + ArogoEvent for CI by git webhook
In that combination, I would suggest using terraform modules. That should enable others to manage infra by focusing on what matters to them and hiding the rest of th complexity (even though GCP is simpler than the rest). You still need something to define what apps are and you can add AOM or crossplane for that.
Hi, about the definition of CRDs in the IDP, can we say that ArgoCD's Application (or ApplicationSet) CRD can be considered as one of concrete example? If not, can you elaborate what kind of CRD that you have in mind for the IDP?
Argo CD Application is a CRD and is a good example of a usage of custom resources. Using it alone, will enable devs to define where there apps are, but it will not help them to define what their apps are. They would still need to learn everything there is to know about Deployments, StatefulSets, Services, Ingresses, VirtualServices, etc. What I'm suggesting is to create CRD that help people define their apps by enabling people to be in control of the things they care about (e.g., host, number of replicas, etc.) and hiding the things they do not care about. Eventually, we cannot expect everything to be able to assemble dozens of k8s resources to get something that resembles an application. Now, to get there, we can either use off-the-shelf solutions like, for example, KNative or create custom CRDs. If we choose the latter, that can be done with, for example, AOM/KubeVela, Crossplane, and a few other tools.
I think you could give more attention to the RBAC part. I totally agree with you that Git is the way to go for business days. But when we think in a crisis scenario, sometimes we get our hands dirty and take action making patches in production.
Hi Viktor! I'm working on a (golang) cli tools which should be an opensource platform for devops teams to build an IDP on top of it. Shortly explained, you can define a "repository" which the k8s.yaml will be placed, An "application" which is the logic unit for a service (still thinking about it). After that you could create any defined resource (pipeline, k8s yamls, etc..) to this defined application in the given repository. the tool only manage an "environment" repository which then expects ArgoCD/Flux to enforce on the cluster. Devops teams can define their own resources (templates), and developers can easily use them, all through the cli. What do you think about the idea ? My intentions is the give a general and easy base for an IDP A lot on my mind right now.
My problem is, that everything seems to be about cloud. How do you establish an IDP for on-prem infrastructure like Shares, DNS (Windows Server DNS, BIND), Backup etc.? AFAIK you would have to write your own crossplane providers for each infrastructure piece, or am I missing something?
It's not only for Cloud. Nevertheless, most of the "new" solutions are focused on Cloud (at least in the beginning) but, at least conceptually, you should be able to do the same on-prem (private cloud). Now, I don't use on-prem myself and, if I do, it would still depend on which providers you're using. Depending on that, there might or might not already be Crossplane providers that cover your needs. If there aren't, and if they do exist in Terraform, you can use Terrajet to generate new Crossplane providers relatively fast (and contribute them :) ). Finally, Crossplane is not the only tool that manages non-Kubernetes resources from Kubernetes. You could, for example, use Cluster API. It won't give you something similar to Crossplane compositions so you might want to combine the two though. So, on-prem is indeed much harder than public Cloud (in general, not related to Crossplane) due to many limitations, one of them being that some of the tools start with the support only for public cloud, while others are exclusive to it.
I'm trying to build an IDP for a typical understaffed, underserved college that only has a few bioscience researchers and IT dept. says NOPE. I need a free IDP that creates IDPs.
To be clear, Crossplane is not the only project that uses Kubernetes controllers to do something other than manage containers. Ingress, for example, creates external load balancers. Persistent volumes are managing external storage. Cluster API is managing all sorts of infra related to Kubernetes clusters. ACK (AWS) and CloudConnect (Google) are managing their resources. And so on and so forth. I'm recommending Crossplane here mostly because it went firther than any other tool of that kind. Nevertheless, those that I mentioned and many others are a proof that Kubernetes API and the control loop are much more than a way to manage containers.
Sadly .....Even i did a good job (i think) company only think devops/infra/sre waste money and not actual contribute on business layer and they always just focusing on in between PM and Developer meet some Marketing request .......
yeah, the moment when i started advocating for something like idp, i became the enemy of the project owner. Because its not only about installing flux, prometheus and other tools. Its also important to interconnect them, automate things, etc. But some people don't see the value as this is not coming from business. So sadly in their eyes this is waste of time and money.
@@DevOpsToolkit yeah so i just spend most of my time and use company resouce to try adopt things learn from your channel 😂I realize learns things its always belongs to me not company .😎
@@andrejab74 i stand with you . but once put everthings line up in place , the feeling just like completed 5000 puzzle and frame it to company , eveyone need to look up to you ,company cant growth without you . (and then you can start another 5000....)
Is GitLab an example of IDP? From your video, it makes me feel that an IDP is the core platform that control everything the developers and the operation team are using. Am I right? Thanks!
It might be considered IDP, in a way, but not fully. To be more precise, it all depends on whether developers are familiar and want to use all the tools available in GitLab. Let's say, for the sake of argument, that is true. Still, how do you define and manage infra? GitLab does not give you the tools for that. How do you define and manage apps? GitLab does not do that either. Essentially, GitLab covers the need for Git and pipelines. There's much more to "everything needed for development and operations" than that.
@@DevOpsToolkit for infra part, GitLab supports integration to Terraform, does it count as control to infra? For manage app part, can you explain more on that part? Or giving some examples that the tool can support app management?
@@tonyccy123 Whether Terraform counts or not depends on experience developers have with both the destination platform (e.g., AWS) and Terraform itself. Typically, we do not expect all the devs to know in depth the destination platform. If that's the case, you need to enable them to manage infra without going into all the details. If you're using Terraform, that could mean creation of modules specific to your company and the way you manage infra. Apps are similar to infra. If, for example, you're running apps in Kubernetes, everyone would need to understand Kubernetes in detail (e.g., deployments, statefulsets, services, ingresses, virtual services, etc.). If they don't, as is often the case, you need to figure out how to enable devs to manage their apps (write manifests, operate, etc.) without knowing Kubernetes (or whatever the platform is) in detail. The point is that IDP enables everyone to do everything from having an idea to running something in production. It's self-service that does not require people to request something from someone (e.g., open a JIRA ticket if you need a cluster). Given that it's impossible for everyone to know everything, the only option is to create internal services that enable people to do whatever they need to do and tailor-made to serve the needs of specific organization while still adapting to varying degrees of expertise. You can think of IDP being the same to devs what AWS is to OPS.
What you are saying is that every DevOps Engineer should specialize on one stage of what DevOps means and then build teams of specialized people that combine that knowledge and build a strong IDP either for existing projects or a template kind of IDP that could be molded to any sort of project and would make DevOps obsolete. Have I understood anything?
I'm not sure whether that would be DevOps teams (or if such teams should exist). What I am sure is that the teams not working on end-user apps should be creating services that enable others to be self-sufficient instead of being reactive by waiting for someone to make a request (e.g., deploy this app, create a server, etc.). When such services are combined, they form IDP. Now, those working on an IDP should possess various skills and experience. It's about Cloud, Kubernetes, serverless, infra, security, etc. (the exact list depends on what a company needs). In other words, what IDP is to devs is the same as, for example, AWS is to ops. You do not request something from AWS (create a VM for me). Instead, AWS built services that ops can consume. The same should be created for app devs. They need to have the freedom to do what they need to do with the assumption that we cannot expect everyone to be experienced in everything.
A trully great video. Thank you! I have 2 questions: 1. Can OpenShift be considered as an IDP? Or it is just a "Kubernetes Enterprise" platform? 2. Is there a list that you can propose for ready IDPs that a company can purchase and do some (or all) of these activities that you mentioned in the video? I have a feeling that a universal "starter" IDP platform is missing. Just like before Jenkins, when there was a need for some "platform" to "stich" things together. And now we have Kubernetes but with an open source and extensible "starter" IDP with plugins, small companies would be able to build on that and customize them according to their needs.
1. I don't think that OpenShift alone can be considered an IDP. It's too complex to enable everyone to use it freely. It could be a base for an IDP, but not an IDP alone. 2. I don't think that there is a good solution off the shelf right now. I do believe that we're getting there, but there are none that I know of that are available off the shelf. I'm not even sure we'll ever get there. Instead, I believe that we'll be getting tools that will enable us to create platforms tailor-made for company needs. Right now, the best bet is to use tools like Crossplane or OAM/KubeVela and combine them with Backstage. P.S. I am heavily involved in Crossplane, so I might not be objective (even though I'm trying to be).
@@DevOpsToolkit yes, it fits well as the documentation and umbrella portal for the IDP you build from all these integrated tools. We use it do document the assembled and composed solution.
It can, just as beginner can do anything else. That bring said, having someone experienced working with a beginner is something that should be considered (just as with anything else).
It's possible, but not advisable. Being a platform engineer means understanding the needs of others and knowing how to fullfil those needs. That comes with experience. I suggest teaming up with someone a bit more experienced.
@@DevOpsToolkit Hey Victor, you are my biggest inspiration. Would you please be my mentor while I learn operations ? I am planning to post my learning journey on youtube from the very beginning as an example for the community. I have come a long way to this point and I would be really grateful to you if you please mentor me on my further journey.😢
Unfortunately, I have a real issue with time. With my job and all the other activities (like this channel), I do not have any time left available. I can do a call with you if that helps, but not a recurring mentoring. If that sounds helpful, please pick any time that works for you from calendly.com/vfarcic/meet?month I'm terribly sorry for that response, but I tend to have too many things on my plate.
@@DevOpsToolkit Totally understandable man, I am going through the same thing while I am just a student. Thank You so much for the meeting. Love You. You should remove the link before people start jumping into your calendar 😆.
Overall, I love most of the ideas in here. However, the disadvantage with this approach is that it discourages new ideas by locking into an approved platform. At least, that's how it's usually done. The IDP here needs to be extensible with very clear information on how to do that and an inclusive ownership team that allows it. Gatekeeping is very easy in this world.
The problem is that many companies have departments that do not understand that they have customers (developers) and that they need to serve their needs or risk losing the business (customers/developers). That risk (losing customers) is often mitigated by forcing people to use whatever those departments offer. That is the real problem. Not knowing who the customers are and not serving their needs... That results in the issues you're mentioning. Give developers the freedom to choose and treat them as customers. If they choose Heroku instead of what you're offering, you'll lose (internal) business. If you do that, you'll be forced to serve their needs and keep them happy just as any business is focused on keeping the customers happy so that they do not go to the competition. Gatekeeping is a horrible idea. Helping others do the right thing and making them more productive is what needs to be the goal.
@@DevOpsToolkit I agree, but unless you mention that your IDP *must* also include controls to prevent gatekeeping and encourage inclusion, it will be used by large organizations to do so. It's the nature of the beast and I see it every day in my consulting. If I missed that, apologies, but I didn't hear it called out, hence my comment. This is why I enourage that platforms like this be inclusive using things like InnerSourcing to facilitate that.
Unfortunate name, IDP. IdP is already taken for "Identity Provider". We definitely need a name for it, though. IDOP (Internal Developer & Operator Platform)? or IDDP (Internal Development and Delivery Platform)?
It indeed it. The only reason I'm calling it IDP is to make sure that it's clear that the customers are developers. Often, platforms are either not easy to use, do not fulfill the needs of developers, or heavily restrict the freedom of devs. You can think of IDP as being a subset of what platforms are.
No. The management of IDP is not done by the same audience as IDP itself. Now, if the question is whether IDP should be managed by Kubernetes (no matter who is involved in that management), the answer is yes.
Look I agree with what you're saying in principal, but I fail to see how your CRD argument fundamentally differs from taking a bunch of already-written terraform modules from the internet and laying yaml over them? That's just like CRDs + CRs + operators, but all the work is already done. Then you can push the yaml to git and CI converts the yaml to json and the json to hcl and then runs terraform. Now I don't really like this approach, but I'm merely providing a counterpoint here that although the kubernetes api as backend idp idea is more modern, it's not new. To me what would be new is something that actually exposed really awesome IAM, RBAC and a UI. And k8s api does have great RBAC. But it does nothing with IAM nor a UI. Anyway k8s api is not the only cloud agnostic api. Your videos are the best devops vids on the net.
I would agree that Terraform wrapped into CRDs + CRs + operators would produce the same result if you would make it for each resource, and not for the project as a whole. I need to be able to use Kube API to see the parent resource (e.g., Terraform project) but also individual resources (e.g., ec2 instance, ELB, VPC, etc.). That's the part I'm not getting. Without it, I cannot query resources, and I cannot hook them into logging, monitoring, etc. For that to work, Terraform project running in k8s would need to "expand" into individual resources. Also, I believe that the lifecycle of each of those resources should be separate (and not tied to a project). And so on and so forth... What I'm really trying to say is that Terraform would have to have a CRD for each resource in a module for that to work as well as a mechanism to tie those resources together. You're (probably) right that k8s API is not the only cloud agnostic API. Nevertheless, everyone (more or less) agreed that Kube API is the thing. No other API has the qualities of Kube API and such a wide support at the same time.
@DevOps Toolkit Hm, yeah I think we missed each other here. You're coming at it from the "Kubernetes is everything" standpoint and I'm saying *skip* all of that and you could simply use terraform. If you're not using kubernetes, terraform will literally do anything for you that you need. It is cloud agnostic. It will build infrastructure, observability, cicd, security, source control, anything. So in that sense it does the exact same thing that k8s does. But the pain in the ass with k8s is that you have to manage your own platform, so it's like a step backwards for most teams that could just use lambda, app runner, cloud run, whatever. So my point is merely that using kube api really isn't that new of an idea -- the apis that run terraform modules make all the same api calls that kube api does to crds. So for example to your point about logging I mean no one uses kube api for production logging. Even if you run a production centralized logging cluster in Kubernetes no one is going to use the literal kube api to query logs...They're going to use whatever the logging product api is. And to your point about CRs and CRDs I mean that is literally what terraform modules are. They are little Go libraries exposed as custom reusable terraform resources, just like crds are little go libraries exposed as custom resources. But like I said I'm not a tf fanboy or anything, I just think k8s is a last resort and that serverless or more simple scheduled container platforms like app runner or ecs should definitely come first because k8s has such a steep learning curve and is a nightmare to onboard developers who don't give a shit at all about maintaining their own stuff.
DevOps is about FAR MORE than this. This is only a third of the focus of DevOps. Coming from the dev side myself I recognise this myopic view of DevOps as dev centric but it is very badly wrong.
@@thescourgeofathousan That is indeed a very probable explanation. DevOps tends to be everything and nothing depending on who you talk to, so depending on who you talk to or what you read, the explanation is going to be very different. Even Patrick (father of DevOps, a common guest in our podcast) is defining DevOps so broadly that it can be interpreted in an infinite number of ways. If there is a definition more common than others, it revolves about being a change in culture but, more often than not, that is not interpreted like that in practice. In any case, I agree. I do not yet know for certain for DevOps is and I'm very vocal and public in saying that in my talks. Also, I am shifting my views on it frequently. You are indeed right when you say that I "don't know DevOps".
@@DevOpsToolkit there is a very clear definition of DevOps in the DevOps Handbook which is a distillation of the concept from the input of all the major players in its inception. I don’t care who came up with the name, there is no one “father of DevOps”. There are multiple fathers AND mothers and in any case it was a continuation of decades of learnings from both the software and manufacturing industries so the scope of of it is even definable by going back along those routes. In my experience when a concept in development becomes talked about as “everything and nothing” it is because the true scope of the origins have been lost amongst the cloud of bs propagated by tool vendors, consultants and even individuals or camps of factions without a clear sense of their own place within the whole. I experienced the same damned thing over the course of the late 2000’s/early 2010’s with REST. People preaching all kinds of crap that clearly wasn’t REST and when challenged saying “oh well REST means different things to different people”. Yeah, if you corrupt the signal and change the story over time to fit what you can do rather than uplifting your skills if you’re a dev/consultant or trying to sell tools that aren’t needed or just try to twist the message to fir existing tools if you’re a tool vendor then YES the concept with start to mean different things to different people. But the definition and the history were right there for anyone to learn in Roy Fielding’s dissertation and in the blogs and interviews etc of Tim Berners Lee and others if they wanted. But no one did want. Even the most altruistic devs I worked with at the time wanted to ingest the information and transform it to fit into their preconceived ideas they had built up throughout the course of their experiences. Sorry for the rant, I’m just so exhausted and tired of people being unable to stop the context in their heads from perverting the information every time we have a major advance in the way we do things. It always falls apart. And even those who try to save it are inevitably constrained by the historical context in their heads and end up just bolstering the one splinter that fits best with that context. Going into new clients and having to constantly combat the pervasive cloud of rubbish and buzzwords and get them to buy into the true message is really wearing me down.
I should also point out that I am not trying to necessarily push back on anything more general about any originating intent or ongoing spirit etc that Mr Dubois has expressed on your podcast. Maintaining a strong idea of those kinds of originating intents is vital to the evolution of a concept/movement as they serve to push the boundaries outward where they are needed and at the same time pull things back when they begin to stray from the point. But the maintenance of a coherent definition and set of practices as that concept/movement evolves is VITAL to its future. That future is lost without respect for BOTH origins AND clearly defined current state. Without both those things charlatans (tool vendors, cookie-cutter consultancies, well-meaning but ignorant proponents, etc) will corrupt the message until the movement dissolves into a cloud of “well, you know it means different things to different people”.
Do you have an IDP? If you do, how did you build it?
IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), TH-cam tends to delete comments that contain links. Please do not use them in your comments.
One thing I absolutely LOVE about this video is that it is actually prescriptive. I have watched so many platform engineering videos lately and I haven't seen a single one, except to a small extent Dave Farley's on CD, but other than that they're all idealogical philosophy that emphasize fairly obvious tenets instead of actual implementations.
I've been developing an IDP for our engineers and I didn't have a word to call it. Having you articulate it in this way is extremely valuable. Thank you!
I completely agree with every word. In fact, I've built my own IDP without knowing this acronym. It is called Kubero and is open-source and completely free to use.
It has a UI, API, and CLI. It is entirely kubernetes native. It also follows the workflows from Heroku. So it is very easy for DevOps to start a new App. It has a Built-in CI/CD and deploy's to Kubernetes with a 'git push' and starts new instances on pull requests.
Thank you, thank you, thank you! This was like a Christmas gift. I’ve been preaching this to the team for months and this puts it all into a great presentation. Well done sir!
Hello Viktor, hi channel. I'm happy to have found your channel. I really don't want to sound like a silly fanboy, but what you laid out in this video makes so much sense. I just started with building my own development platform or system. I did not exactly know, what I need and how to call this, but IDP is pretty much it. It is all about abstracting the complexity away from the customers. In our case these customers are our devs, as you mentioned. My primary obstacle at this point is the fact, that I need to learn how to leverage the Kubernetes API as a cloud native tool for an on premise oriented development and deployment toolkit. A very interesting aspect is the extensibility of the Kubernetes API. I really need to figure out, how to use it and how to build CRD's for tasks like bare metal installations or setting up virtual servers on e.g a single VPS server or on a local machine. No idea, if this is possible, but worth a try imho. Building an IDP with very limited local resources could well turn out to be a mad mans plan, but at least worth trying imho. You mentioned not to build it from scratch but to use and adapt existing tools. Just started to look around and I guess in later videos from you I will find more info. I'm glad you are around. I will sure have to consult this channel quite a lot in the coming months. Thanks, Viktor.
Thanks you for the words of praise.
You're right. Follow-up videos go into details and hands-on so keep watching :)
I have no words to describe how I needed this video. IDP is something I want to build but I have no idea, until now, from where I should begin. Thanks for share you knowledge.
Code Rant: Resource Provisioning Complexity Clock
12 o'clock: Clicking buttons in the AWS Management Console
3 o'clock: Calling APIs
6 o'clock: DSL
9 o'clock: GitOps
12 o'clock: Clicking buttons in an internal re-write of the AWS Management Console except now CloudTrail pushes its changes to Git to trigger CI/CD on infrastructure.
Awesome video Viktor. I watched this after "DevOps is Dead", the two videos dovetail into each other perfectly. Thank you!
They do. I should have made them in the opposite order but the part of me that fails at planning prevailed.
Thank you for this videos! We have been building that type of platform internally and didn't have a name for it until now! Amazing content!
@Viktor ...u changed the whole perspective man !!! Kudos to ur way of knowledge transfer !!!!
I often change my perspective. I think that's healthy and much better than sticking with the same views forever and ever. Very often, I do not stand behind things I said in the past. Tech changes and so do my views.
Hvala Viktore , odlican video , cist i jasan! Odlicna produkcija! Pozdrav!
Great stuff, just in the right moment as I started to build an IDP!
This is pure gold. Very informative, dense and helpful. I'll have to watch it a couple more times to absorb everything.
This so much this. been doing this for years and seem to be the only person in a rather large team doing this. Specifically I wrote an IDP around gitlab-ci to automatically create pipelines :D
you described my life, and predicted my future.
You just gave me a reason to learn more about CRDs
Your channel has been super helpful 🔥🔥
This is a massive video, would you say that this is covered in your book(s)? I tried standing up Backstage a while back but realized it quickly took me too far away from what I set out to do. Using the Kubernetes API for everything, simple statement, a lot to really unpack. Great video! Thank you!
None of the things I'm publishing in YT since September last year are covered in my books. It's all-new material and the main reason I switched to TH-cam is to be able to publish faster and more frequently.
As for Kube API... That's why Crossplane, KubeVela, Cluster API, and a few other tools are great. Even though they might be using different approaches, they all have in common the objective of extending Kube API with additional CRDs and controllers.
Backstage is having a steep initial learning curve and setup. I'll be publishing more about it soon.
Finally, we (the industry) are not yet there. There are still pieces missing from the solution I'm proposing. The good news though is that we're getting there very fast. That will be my main focus this year.
@Banana in Condom I wrote 8-9 books so far, but most of them were removed from sales since the tech described in them is now deprecated. Two of those are good though and you can find the info in devopstoolkitseries.com/.
Thank you for giving this overview! Briliant! Career changing even. I will be looking for companies who think like that
Totally agree with this based on own experiences
Thanks as always for keeping us up to date. Great stuff.
Wow just loved this video. Totally makes sense why to use an IDP. Thank you so much
Such a great way to look at current landscape of devops. I feel enlightened by this content
This is so cool. We are actually wanting to build an EDP. External Developer Platform. i.e. a very specialized and opinionated platform, but allowing "normal" devs to build applications that are built to be shared and used by multiple tenants (true SaaS) and ready for scale from day one. The only concerns we have currently are how to make things "secure". Your video shows the interfaces that we need to concentrate on. We have some ideas, and this video also supports our initial directions. We want devs pushing to repos, but we don't want them messing with custom resources. They need to be "standardized" for any and all developers. One of our many, many challenges.
You can never create a standard for all developers. That would be like having a goal to reach infinity. Instead, you can either create an opinionated platform that serves some use cases (e.g., Heroku) or create tools that enable ops to create opinionated platforms used by devs. I believe that we need both types of solutions.
P.S. Is what you're building public? If it is, it would be great if I could take a look.
@@DevOpsToolkit Yes. It will be very opinionated. And no, nothing public yet. It will be a while before the platform can be made public. :) We'll be using it internally first (i.e. dog fooding) and once we have some initial apps to be able to deploy for others to use (i.e. for winning hearts and minds of developers and pocket books of customers), we'll be making it public.
Your content is pure gold, trully thank you !
Please consider covering advanced argocd topics like helm secrets and config plugin management, these topics are rarely covered in TH-cam. Also you’re the best 🙌🏽
Adding it to my TODO list... :)
@@DevOpsToolkit also please consider covering advantages and disadvantages of using flux and argo. I know there is a movement for gitops but there are quite a few pain point that I feel are never addressed by the community such as multi environment + multi region deployment modeling in this format. Most teams and developers I've seen talking about this have resorted to hackish ways to resolve this problem and almost (if not all) tutorials around argo never get around to covering this
@@blabos123 Great suggestion. Adding it to my TODO list... :)
A fantastic presentation Viktor! I will now ask all of my engineers and anyone I coach to watch this video. The information presented here was extremely well done.
Completely agree with your sentiments about K8s being so much more than a container orchestrator and becoming a universal API (and cloud OS).
Also agree and look forward to UI tools like Rancher changing from altering state directly to pushing to git instead. I guess (and hope) that's why Rancher is building Fleet!
That's my hope for Rancher as well. If they do start using Git as config/manifest storage and Fleet to synchronize it, they will have a killer solution. The only thing that makes me a bit pessimistic is that Fleet is still far from being a good solution. Let's hope they get there.
We're currently building an IDP with Backstage IO. But you make some good points about Kubernetes API and CRDs. Hmmm. Now I don't know what to do!
OpenShift is a good example that provides an easy enough to use/understand IDP
Wow!! This is the best. Thank you!
Thank you always for your videos. Are you planning to have an IDP series where you show how to actually build this IDP?
I will, but in the second part of 2023. The first part is reserved for something else (something very big).
@@DevOpsToolkit thank you. I am looking forward to it😊
Great video @viktor , so insightful!
we have tried acheivinig the same thing using Crossplane, argocd and airflow. But at the moment (6 months back), we didnt have few cloud components supported in Crossplane for AWS (EC2 instances and Elasticsearch to be more specific), so we have just put the efforts on hold, once crossplane supports more components we will definetly resume the efforts from we left off.
A few months ago, Crossplane released Jet project that allows people to generate Crossplane providers using code generation based on Terraform modules. As a result, all AWS, Azure, and GCP resources are now supported by separate (Jet) providers. Quite a few other vendors are now supported as well and, even if something you need is still missing, it's very easy to create it. You might want to try it out again.
@@DevOpsToolkit Sure thanks, will definetly try this out. keep uploading great videos like this.
I 100% agree that IDPs are going to explode. I had a chat with Kaspar (humanitec) back in 2020 about their product and this whole term 'internal developer platform' idea. I was exploring moving dev envs to ephemeral containers on k8s and I needed a simple way for devs to launch applications. I had come across Backstage at the time at well. Unfortunately, I was kicked off that project and it was given to some ops folks to figure out. And that's why today devs install kubectl to manage their own vclusters to do any development.
And so on and so forth!
If we look at the 'DevOps' infinity loop - a lot of the products are on the Deploy/Operate/Monitor corner.
I'm currently focusing on the build/test/release bit.
Thank you so much for this video, very easy to follow, straight to the point and helped me accommodate all the concepts correctly in my head.
Thanks!
Thanks a ton, Artur.
This is incredible, thank you
I like your energy and mindset.
"Define and implement a tool for developers to self-service themselves"
I would love to hear your opinion on what is awaiting us in the next 10 years
The software industry is moving so fast that predicting even the next year or two is very challenging. 10 years from now is just too much.
Totally agree, another great vid!
Thanks Viktor, a really good video 👍
A terraform module is the exact same thing as a CR and a CRD is exactly the same thing as the golang library backing the terraform module, except unlike the hashicorp public module registry, helm is significantly smaller and less maintained and I have to write the literal CRD code.
Thank you! Awesome video!
Very detailed coverage on process and getting from desired state to actual state. Does it have a part 2 which discuss in details on observing the actual state?
That's in my TODO list.... :)
Nice video as usually!
Hey Viktor, you've mentioned "numerous studies that show IDPs increase productivity often drastically". Would you happen to have some to share? If so, that would be awesome because it would help to support this idea in future discussions.
The one I like the most is the DORA report. You can find it in www.devops-research.com/research.html.
@@DevOpsToolkit Thanks for sharing! 🙏
I just stared looking at VMware Tanzu Application Platform (TAP). It looks interesting.
Damn! very informative thanks a lot! we are about to deploy one of our apllication on EKS in order to slowly move another 2 or 3 of our apps. Would you recommend to build the IDP from the very beginning or is it overkill to have an IDP for 3 apps? I am still uncertain if IDP is made for intensive production need or if it's also recommended for small needs. I guess it takes a lot of time to put an IDP together. thanks a lot again for the gold mine!
Building IDP is not different from building any other type of an app or a system. The more users you have, the bigger the ROI.
@@DevOpsToolkit i am sorry my question was not clear. what I wanted to simply ask is: Is it relevant to spend time building and IDP if we only have 2 or 3 apps (10% of all our app) on k8s?
Oh yeah. I might not have been clear in that video that I am advocating for using Kubernetes as a control plane no matter where the apps, infra, and services are located. It's about having an API that can manage resources no matter where they are, not about having an API only for the apps running in the same Kubernetes cluster. Among other things, Kubernetes is about managing resources and I'm on a mission to explain that Kubernetes is NOT (only) about runnning applications packaged as container images.
So yes. If you should build and IDP, you should do it on top of Kubernetes independently of where those aps are. Kubernetes is our best bet for a control plane right now.
@@DevOpsToolkit Let's say you were pretty clear on that idea. I guess I am lacking the operational logic and experience to make sens of that concept...hard to understand how k8s can be the reference API entrypoint even for things not running in containers. I will try to look for exemple and other use case of k8s but I am afraid it's going to be hard.. let see
@@anthonymc6140 Kubernetes out-of-the-box is mostly focused on containers. However, Noone is supposed to use Kubernetes as-is. Neither the project design nor the community has the goal for people to use it as-is but, rather, as a base on top of which other vendors, projects, or end-users can build whatever makes sense for them to have (e.g., a platform).
Kubernetes is all about extensibility through CRDs. For example, when you install Argo CD, you get CRDs like `Application` and `Project`. Almost any other Kubernetes-native project brings its own CRDs and corresponding controllers that do something. Now, what that something is depends on what you install. For example, if you install KubeVirt (watch th-cam.com/video/oO8VEmpojz0/w-d-xo.html) you are getting CRDs and controllers that allow you to manage virtual machines that do not have anything to do with containers. Another example would be Crossplane that, depending on which providers you install, allows you to manage AWS, Azure, Google Cloud, or (almost) any other type of resources that also have nothing to do with containers (watch th-cam.com/video/n8KjVmuHm7A/w-d-xo.html, th-cam.com/video/yrj4lmScKHQ/w-d-xo.html, and th-cam.com/video/AtbS1u2j7po/w-d-xo.html). There are many other examples and it all boils down to two things. What Kubernetes does depends on which CRDs and controllers are running in that cluster so there are no real limits of its scope.
Here comes the important part. Kubernetes is not easy. You need to be experienced with it to start extending (initially through third-party tools and later yourself). So, you might not want to jump into building an IDP as the first thing you do in Kubernetes. But, over time, that is your best bet.
P.S. When I say that Kubernetes is hard, I must also stress that most of the things Kubernetes does are easier in it than anywhere else. It's just that some advanced stuff might be too much for some no matter whether that "stuff" is done in Kubernetes or elsewhere. So, it's actually easy considering what it does.
P.P.S. "Kubernetes is not only about containers" is one of the main reasons why I got involved with Crossplane.
P.P.P.S. I already started working on a video that will provide hands-on instructions how to build an IDP. However, it'll probably take ~2 more months until it goes live. It'll be one of those videos that last for 30-45 minutes but require months of preparation.
Thanks! Really useful information.
Large corporations end up with IDPs for everything, they aren't hard to use individually but you just need to know many of them in order to achieve something, you end up with the original problem, it takes a decade to master all the IDPs you need... you end up with jobs like mine where you are part of a product team (dev) as an SME on all the IDPs relevant for the product. Especially important in teams with high turnover rates.
Argo Workflows is absolutely fantastic! I just wish their community had better support though...
Thanks for this video! Did you take a look at tanzu application platform for the buy part? It tries to be not too opinionated by allowing to swap pieces. A review of TAP would be really appreciated!
Adding it to my TODO list... :)
I would go with Backstage. Fully flexible
Thank you very much from South Korea. And I have one question. How much of freedom should be given to developers? Should they access to all the tools(eg, Jenkins, Harbor..) behind IDP or they just need access to IDP and SCM?
If I give access to tools under the IDP, their freedom is maximum but it seems to me little hard to manage. What is your suggestion?
That depends on them. They are your customers. Ask them what they want and what they need.
Yes. In my case, when I asked to them, most of developers want IDP only but some of them(less than 5% of whole devlopers) who know well about devops want to full access to the tools behind and that is making decision difficult. That is why I want to know how other companies do.
@@user-ch5hu2kk4c The main purpose of IDPs is to abstract and simplify the tooling and the processes so that people who are not experts in specific areas can still operate. That means that in the ideal situation the IDP layer on top of that tooling abstracts everything just as, for example, AWS abstracts hypervisors from it's users.
So, the goal is often not to have access to the internal tooling below IDP. However, almost noone gets to that point 100%. It's very hard to build everything so companies often go step by step. So, for example, if you built option to deploy apps, you do not need access to the tools like pipelines or GitOps. But, if you did not yet build your own solution for observing logs, you have to give people access to Grafana/Loki or whatever is used for that.
Also, you need to have the "break the glass" option. The IDP might fail or not provide a solution for specific scenarios and, in that case, having the access to the tools below it is a must.
At the end of the day, it's what your users need. If 5% want the direct access to the tools instead of going through the IDP, it means that they feel more comfortable using those tools than what you built. That means that you have 5% of unhappy customers that are not getting what they need.
Deeply appreciate for clarification. Now I see.
Hi thanks for the video 🙂
We are planning to build an IDP for creating cicd processes automatically.
Our company is currently migrating to Jenkins and we are debating whether to use this IDP just for creating the initial jenkins files or also to enforce the changes through the IDP. In that way the developers will not be able to change things directly to the Jenkins files. What do you think is the right thing to do?
I do not think that enforcement should be done in pipelines but on the server-side. To begin with, pipelines might not be the only processes that make changes to the system. Also, such an approach severely limits the "freedom" of those using or defining the pipelines. So, instead of, for example, forcing execution of security scanning in a pipeline, move it to the registry where artifacts are pushed, clusters where apps are deployed, etc.
Working on a laptop workflow right now.
Out of interest: What are we using these days for CICD? I see a lot of CD videos.(argo,flux,pulumi,crossplane,terraform)
I will vote for
Terrform for GKE and Nodepool manage + bring up ArogoCD
then ArgoCD to config whole cluster by HelmChart +Vault for secret
Argoworkflow + ArogoEvent for CI by git webhook
In that combination, I would suggest using terraform modules. That should enable others to manage infra by focusing on what matters to them and hiding the rest of th complexity (even though GCP is simpler than the rest). You still need something to define what apps are and you can add AOM or crossplane for that.
Hi, about the definition of CRDs in the IDP, can we say that ArgoCD's Application (or ApplicationSet) CRD can be considered as one of concrete example? If not, can you elaborate what kind of CRD that you have in mind for the IDP?
Argo CD Application is a CRD and is a good example of a usage of custom resources. Using it alone, will enable devs to define where there apps are, but it will not help them to define what their apps are. They would still need to learn everything there is to know about Deployments, StatefulSets, Services, Ingresses, VirtualServices, etc. What I'm suggesting is to create CRD that help people define their apps by enabling people to be in control of the things they care about (e.g., host, number of replicas, etc.) and hiding the things they do not care about. Eventually, we cannot expect everything to be able to assemble dozens of k8s resources to get something that resembles an application. Now, to get there, we can either use off-the-shelf solutions like, for example, KNative or create custom CRDs. If we choose the latter, that can be done with, for example, AOM/KubeVela, Crossplane, and a few other tools.
Any plans to do a video on backstage in the future?
I already have it on my TODO list :)
@@DevOpsToolkit +1 :)
I think you could give more attention to the RBAC part. I totally agree with you that Git is the way to go for business days. But when we think in a crisis scenario, sometimes we get our hands dirty and take action making patches in production.
That's true. Sometimes, in times of crisis, we do "random" things, but those are (or should be) the exception rather than the rule.
@@DevOpsToolkit yep. A good topic to talk about could be how to deal with breaking glass scenarios.
@@brunodias8466 Adding it to my TODO list... :)
@@DevOpsToolkit Great! I'm sure that you will give significant insights! \o/
Hi Viktor!
I'm working on a (golang) cli tools which should be an opensource platform for devops teams to build an IDP on top of it.
Shortly explained, you can define a "repository" which the k8s.yaml will be placed, An "application" which is the logic unit for a service (still thinking about it).
After that you could create any defined resource (pipeline, k8s yamls, etc..) to this defined application in the given repository.
the tool only manage an "environment" repository which then expects ArgoCD/Flux to enforce on the cluster.
Devops teams can define their own resources (templates), and developers can easily use them, all through the cli.
What do you think about the idea ? My intentions is the give a general and easy base for an IDP
A lot on my mind right now.
That sounds great!
My problem is, that everything seems to be about cloud. How do you establish an IDP for on-prem infrastructure like Shares, DNS (Windows Server DNS, BIND), Backup etc.? AFAIK you would have to write your own crossplane providers for each infrastructure piece, or am I missing something?
It's not only for Cloud. Nevertheless, most of the "new" solutions are focused on Cloud (at least in the beginning) but, at least conceptually, you should be able to do the same on-prem (private cloud).
Now, I don't use on-prem myself and, if I do, it would still depend on which providers you're using. Depending on that, there might or might not already be Crossplane providers that cover your needs. If there aren't, and if they do exist in Terraform, you can use Terrajet to generate new Crossplane providers relatively fast (and contribute them :) ).
Finally, Crossplane is not the only tool that manages non-Kubernetes resources from Kubernetes. You could, for example, use Cluster API. It won't give you something similar to Crossplane compositions so you might want to combine the two though.
So, on-prem is indeed much harder than public Cloud (in general, not related to Crossplane) due to many limitations, one of them being that some of the tools start with the support only for public cloud, while others are exclusive to it.
I'm trying to build an IDP for a typical understaffed, underserved college that only has a few bioscience researchers and IT dept. says NOPE. I need a free IDP that creates IDPs.
Thank you, but how crd and kubernetes can do deployments for something else than containers ? i don't figure it out.
Try Crossplane :) You'll find quite a few videos about it in this channel.
To be clear, Crossplane is not the only project that uses Kubernetes controllers to do something other than manage containers. Ingress, for example, creates external load balancers. Persistent volumes are managing external storage. Cluster API is managing all sorts of infra related to Kubernetes clusters. ACK (AWS) and CloudConnect (Google) are managing their resources. And so on and so forth. I'm recommending Crossplane here mostly because it went firther than any other tool of that kind. Nevertheless, those that I mentioned and many others are a proof that Kubernetes API and the control loop are much more than a way to manage containers.
Sadly .....Even i did a good job (i think) company only think devops/infra/sre waste money and not actual contribute on business layer and they always just focusing on in between PM and Developer meet some Marketing request .......
A company not trying to make people more productive is bound to fail eventually.
yeah, the moment when i started advocating for something like idp, i became the enemy of the project owner. Because its not only about installing flux, prometheus and other tools. Its also important to interconnect them, automate things, etc. But some people don't see the value as this is not coming from business. So sadly in their eyes this is waste of time and money.
@@andrejab74 That's the moment when you might want to start thinking what's best for you, not the company.
@@DevOpsToolkit yeah so i just spend most of my time and use company resouce to try adopt things learn from your channel 😂I realize learns things its always belongs to me not company .😎
@@andrejab74 i stand with you . but once put everthings line up in place , the feeling just like completed 5000 puzzle and frame it to company , eveyone need to look up to you ,company cant growth without you . (and then you can start another 5000....)
Is GitLab an example of IDP? From your video, it makes me feel that an IDP is the core platform that control everything the developers and the operation team are using. Am I right? Thanks!
It might be considered IDP, in a way, but not fully. To be more precise, it all depends on whether developers are familiar and want to use all the tools available in GitLab. Let's say, for the sake of argument, that is true. Still, how do you define and manage infra? GitLab does not give you the tools for that. How do you define and manage apps? GitLab does not do that either. Essentially, GitLab covers the need for Git and pipelines. There's much more to "everything needed for development and operations" than that.
@@DevOpsToolkit
for infra part, GitLab supports integration to Terraform, does it count as control to infra?
For manage app part, can you explain more on that part? Or giving some examples that the tool can support app management?
@@tonyccy123 Whether Terraform counts or not depends on experience developers have with both the destination platform (e.g., AWS) and Terraform itself. Typically, we do not expect all the devs to know in depth the destination platform. If that's the case, you need to enable them to manage infra without going into all the details. If you're using Terraform, that could mean creation of modules specific to your company and the way you manage infra.
Apps are similar to infra. If, for example, you're running apps in Kubernetes, everyone would need to understand Kubernetes in detail (e.g., deployments, statefulsets, services, ingresses, virtual services, etc.). If they don't, as is often the case, you need to figure out how to enable devs to manage their apps (write manifests, operate, etc.) without knowing Kubernetes (or whatever the platform is) in detail.
The point is that IDP enables everyone to do everything from having an idea to running something in production. It's self-service that does not require people to request something from someone (e.g., open a JIRA ticket if you need a cluster). Given that it's impossible for everyone to know everything, the only option is to create internal services that enable people to do whatever they need to do and tailor-made to serve the needs of specific organization while still adapting to varying degrees of expertise. You can think of IDP being the same to devs what AWS is to OPS.
@@DevOpsToolkit detailed and clear explanation! Got it now, thanks.
What you are saying is that every DevOps Engineer should specialize on one stage of what DevOps means and then build teams of specialized people that combine that knowledge and build a strong IDP either for existing projects or a template kind of IDP that could be molded to any sort of project and would make DevOps obsolete. Have I understood anything?
I'm not sure whether that would be DevOps teams (or if such teams should exist). What I am sure is that the teams not working on end-user apps should be creating services that enable others to be self-sufficient instead of being reactive by waiting for someone to make a request (e.g., deploy this app, create a server, etc.). When such services are combined, they form IDP. Now, those working on an IDP should possess various skills and experience. It's about Cloud, Kubernetes, serverless, infra, security, etc. (the exact list depends on what a company needs).
In other words, what IDP is to devs is the same as, for example, AWS is to ops. You do not request something from AWS (create a VM for me). Instead, AWS built services that ops can consume. The same should be created for app devs. They need to have the freedom to do what they need to do with the assumption that we cannot expect everyone to be experienced in everything.
A trully great video. Thank you! I have 2 questions:
1. Can OpenShift be considered as an IDP? Or it is just a "Kubernetes Enterprise" platform?
2. Is there a list that you can propose for ready IDPs that a company can purchase and do some (or all) of these activities that you mentioned in the video?
I have a feeling that a universal "starter" IDP platform is missing. Just like before Jenkins, when there was a need for some "platform" to "stich" things together. And now we have Kubernetes but with an open source and extensible "starter" IDP with plugins, small companies would be able to build on that and customize them according to their needs.
1. I don't think that OpenShift alone can be considered an IDP. It's too complex to enable everyone to use it freely. It could be a base for an IDP, but not an IDP alone.
2. I don't think that there is a good solution off the shelf right now. I do believe that we're getting there, but there are none that I know of that are available off the shelf. I'm not even sure we'll ever get there. Instead, I believe that we'll be getting tools that will enable us to create platforms tailor-made for company needs.
Right now, the best bet is to use tools like Crossplane or OAM/KubeVela and combine them with Backstage.
P.S. I am heavily involved in Crossplane, so I might not be objective (even though I'm trying to be).
Love it!
And add a developer portal on top! With e.g. backstage
Yes. Backstage is the closest we have today as a tool to provide a user interface of that kind.
@@DevOpsToolkit yes, it fits well as the documentation and umbrella portal for the IDP you build from all these integrated tools. We use it do document the assembled and composed solution.
Can a beginner be a platform engineer ?
It can, just as beginner can do anything else. That bring said, having someone experienced working with a beginner is something that should be considered (just as with anything else).
It's possible, but not advisable. Being a platform engineer means understanding the needs of others and knowing how to fullfil those needs. That comes with experience. I suggest teaming up with someone a bit more experienced.
@@DevOpsToolkit Hey Victor, you are my biggest inspiration. Would you please be my mentor while I learn operations ? I am planning to post my learning journey on youtube from the very beginning as an example for the community. I have come a long way to this point and I would be really grateful to you if you please mentor me on my further journey.😢
Unfortunately, I have a real issue with time. With my job and all the other activities (like this channel), I do not have any time left available. I can do a call with you if that helps, but not a recurring mentoring. If that sounds helpful, please pick any time that works for you from calendly.com/vfarcic/meet?month
I'm terribly sorry for that response, but I tend to have too many things on my plate.
@@DevOpsToolkit Totally understandable man, I am going through the same thing while I am just a student. Thank You so much for the meeting. Love You.
You should remove the link before people start jumping into your calendar 😆.
Overall, I love most of the ideas in here. However, the disadvantage with this approach is that it discourages new ideas by locking into an approved platform. At least, that's how it's usually done. The IDP here needs to be extensible with very clear information on how to do that and an inclusive ownership team that allows it. Gatekeeping is very easy in this world.
The problem is that many companies have departments that do not understand that they have customers (developers) and that they need to serve their needs or risk losing the business (customers/developers). That risk (losing customers) is often mitigated by forcing people to use whatever those departments offer. That is the real problem. Not knowing who the customers are and not serving their needs... That results in the issues you're mentioning. Give developers the freedom to choose and treat them as customers. If they choose Heroku instead of what you're offering, you'll lose (internal) business. If you do that, you'll be forced to serve their needs and keep them happy just as any business is focused on keeping the customers happy so that they do not go to the competition. Gatekeeping is a horrible idea. Helping others do the right thing and making them more productive is what needs to be the goal.
@@DevOpsToolkit I agree, but unless you mention that your IDP *must* also include controls to prevent gatekeeping and encourage inclusion, it will be used by large organizations to do so. It's the nature of the beast and I see it every day in my consulting. If I missed that, apologies, but I didn't hear it called out, hence my comment. This is why I enourage that platforms like this be inclusive using things like InnerSourcing to facilitate that.
@@DerekMurawsky You're right. I should have called that out in the video.
Brilliant
Unfortunate name, IDP. IdP is already taken for "Identity Provider". We definitely need a name for it, though. IDOP (Internal Developer & Operator Platform)? or IDDP (Internal Development and Delivery Platform)?
Isn't this just called platform engineering?
It indeed it. The only reason I'm calling it IDP is to make sure that it's clear that the customers are developers. Often, platforms are either not easy to use, do not fulfill the needs of developers, or heavily restrict the freedom of devs. You can think of IDP as being a subset of what platforms are.
Aren't you going to need an IDP to manage and deploy your IDP?
No. The management of IDP is not done by the same audience as IDP itself. Now, if the question is whether IDP should be managed by Kubernetes (no matter who is involved in that management), the answer is yes.
Look I agree with what you're saying in principal, but I fail to see how your CRD argument fundamentally differs from taking a bunch of already-written terraform modules from the internet and laying yaml over them? That's just like CRDs + CRs + operators, but all the work is already done. Then you can push the yaml to git and CI converts the yaml to json and the json to hcl and then runs terraform. Now I don't really like this approach, but I'm merely providing a counterpoint here that although the kubernetes api as backend idp idea is more modern, it's not new.
To me what would be new is something that actually exposed really awesome IAM, RBAC and a UI. And k8s api does have great RBAC. But it does nothing with IAM nor a UI. Anyway k8s api is not the only cloud agnostic api.
Your videos are the best devops vids on the net.
I would agree that Terraform wrapped into CRDs + CRs + operators would produce the same result if you would make it for each resource, and not for the project as a whole. I need to be able to use Kube API to see the parent resource (e.g., Terraform project) but also individual resources (e.g., ec2 instance, ELB, VPC, etc.). That's the part I'm not getting. Without it, I cannot query resources, and I cannot hook them into logging, monitoring, etc. For that to work, Terraform project running in k8s would need to "expand" into individual resources. Also, I believe that the lifecycle of each of those resources should be separate (and not tied to a project). And so on and so forth...
What I'm really trying to say is that Terraform would have to have a CRD for each resource in a module for that to work as well as a mechanism to tie those resources together.
You're (probably) right that k8s API is not the only cloud agnostic API. Nevertheless, everyone (more or less) agreed that Kube API is the thing. No other API has the qualities of Kube API and such a wide support at the same time.
@DevOps Toolkit Hm, yeah I think we missed each other here. You're coming at it from the "Kubernetes is everything" standpoint and I'm saying *skip* all of that and you could simply use terraform. If you're not using kubernetes, terraform will literally do anything for you that you need. It is cloud agnostic. It will build infrastructure, observability, cicd, security, source control, anything. So in that sense it does the exact same thing that k8s does. But the pain in the ass with k8s is that you have to manage your own platform, so it's like a step backwards for most teams that could just use lambda, app runner, cloud run, whatever.
So my point is merely that using kube api really isn't that new of an idea -- the apis that run terraform modules make all the same api calls that kube api does to crds. So for example to your point about logging I mean no one uses kube api for production logging. Even if you run a production centralized logging cluster in Kubernetes no one is going to use the literal kube api to query logs...They're going to use whatever the logging product api is.
And to your point about CRs and CRDs I mean that is literally what terraform modules are. They are little Go libraries exposed as custom reusable terraform resources, just like crds are little go libraries exposed as custom resources.
But like I said I'm not a tf fanboy or anything, I just think k8s is a last resort and that serverless or more simple scheduled container platforms like app runner or ecs should definitely come first because k8s has such a steep learning curve and is a nightmare to onboard developers who don't give a shit at all about maintaining their own stuff.
first commit!
DevOps is about FAR MORE than this.
This is only a third of the focus of DevOps.
Coming from the dev side myself I recognise this myopic view of DevOps as dev centric but it is very badly wrong.
There's definitely much more that needs to be done. I'm just not sure we should call all that DevOps.
@@DevOpsToolkit then you don’t know DevOps.
Go back to the DevOps handbook.
@@thescourgeofathousan That is indeed a very probable explanation. DevOps tends to be everything and nothing depending on who you talk to, so depending on who you talk to or what you read, the explanation is going to be very different. Even Patrick (father of DevOps, a common guest in our podcast) is defining DevOps so broadly that it can be interpreted in an infinite number of ways. If there is a definition more common than others, it revolves about being a change in culture but, more often than not, that is not interpreted like that in practice.
In any case, I agree. I do not yet know for certain for DevOps is and I'm very vocal and public in saying that in my talks. Also, I am shifting my views on it frequently. You are indeed right when you say that I "don't know DevOps".
@@DevOpsToolkit there is a very clear definition of DevOps in the DevOps Handbook which is a distillation of the concept from the input of all the major players in its inception.
I don’t care who came up with the name, there is no one “father of DevOps”.
There are multiple fathers AND mothers and in any case it was a continuation of decades of learnings from both the software and manufacturing industries so the scope of of it is even definable by going back along those routes.
In my experience when a concept in development becomes talked about as “everything and nothing” it is because the true scope of the origins have been lost amongst the cloud of bs propagated by tool vendors, consultants and even individuals or camps of factions without a clear sense of their own place within the whole.
I experienced the same damned thing over the course of the late 2000’s/early 2010’s with REST.
People preaching all kinds of crap that clearly wasn’t REST and when challenged saying “oh well REST means different things to different people”.
Yeah, if you corrupt the signal and change the story over time to fit what you can do rather than uplifting your skills if you’re a dev/consultant or trying to sell tools that aren’t needed or just try to twist the message to fir existing tools if you’re a tool vendor then YES the concept with start to mean different things to different people.
But the definition and the history were right there for anyone to learn in Roy Fielding’s dissertation and in the blogs and interviews etc of Tim Berners Lee and others if they wanted. But no one did want. Even the most altruistic devs I worked with at the time wanted to ingest the information and transform it to fit into their preconceived ideas they had built up throughout the course of their experiences.
Sorry for the rant, I’m just so exhausted and tired of people being unable to stop the context in their heads from perverting the information every time we have a major advance in the way we do things. It always falls apart. And even those who try to save it are inevitably constrained by the historical context in their heads and end up just bolstering the one splinter that fits best with that context.
Going into new clients and having to constantly combat the pervasive cloud of rubbish and buzzwords and get them to buy into the true message is really wearing me down.
I should also point out that I am not trying to necessarily push back on anything more general about any originating intent or ongoing spirit etc that Mr Dubois has expressed on your podcast. Maintaining a strong idea of those kinds of originating intents is vital to the evolution of a concept/movement as they serve to push the boundaries outward where they are needed and at the same time pull things back when they begin to stray from the point.
But the maintenance of a coherent definition and set of practices as that concept/movement evolves is VITAL to its future.
That future is lost without respect for BOTH origins AND clearly defined current state.
Without both those things charlatans (tool vendors, cookie-cutter consultancies, well-meaning but ignorant proponents, etc) will corrupt the message until the movement dissolves into a cloud of “well, you know it means different things to different people”.
It was a great session learned alot regarding IDP i will surely start looking more into it.
Thanks @viktor
Thanks!