How to Propagate Secrets Everywhere with External Secrets Operator (ESO) and Crossplane
ฝัง
- เผยแพร่เมื่อ 1 มิ.ย. 2024
- We dive into the powerful synergy between External Secrets Operator (ESO) and Crossplane to efficiently manage and propagate secrets across your Kubernetes clusters, databases, and secrets managers. Learn how to securely and seamlessly integrate with cloud providers' secret management systems using ESO, and see how to leverage Crossplane's infrastructure as code capabilities to ensure your secrets are consistently deployed wherever they're needed.
#ExternalSecretsOperator #Crossplane #KubernetesSecrets #ESO
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/vfarcic/216c5...
🔗 External Secrets Operator (ESO): external-secrets.io
🎬 Manage Kubernetes Secrets With External Secrets Operator (ESO): • Manage Kubernetes Secr...
🎬 Crossplane - GitOps-based Infrastructure as Code through Kubernetes API: • Crossplane - GitOps-ba...
🎬 How To Shift Left Infrastructure Management Using Crossplane Compositions: • How To Shift Left Infr...
🎬 Crossplane Composition Functions: Unleashing the Full Potential: • Crossplane Composition...
🎬 OpenFunction: The Best Way to Run Serverless Functions on Kubernetes?: • OpenFunction: The Best...
🔗 Kubernetes Compositions: github.com/vfarcic/crossplane...
🔗 SQL Compositions: github.com/vfarcic/crossplane...
🎬 Kubernetes Deployment Order and Dependencies Demystified: • Kubernetes Deployment ...
🎬 Argo CD Synchronization is BROKEN! It Should Switch to Eventual Consistency!: • Argo CD Synchronizatio...
▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendar.app.google/Q9eaDUHN8... to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Introduction
01:09 The Problem
05:51 Secrets Between Kubernetes Clusters
14:48 Pushing and Pulling PostgreSQL Database Secrets - วิทยาศาสตร์และเทคโนโลยี
Thanks for another great video! I'd like to explore databases as a service with your videos.
About secrets management with ESO, I use it all the time and I love it.
I still find copying secret directly between kubernetes clusters a useful option, especially when you have a control plane cluster managing other clusters and secrets are consumed in kubernetes.
This way you don't need to install (and secure) multiple ESO instances.
I think it would be also useful to see an example of secret rotation in action (meaning not waiting for the secret to update in kubernetes but rotating it transparently while applications are using it).
Hey Viktor, Thanks for the information! your videos help me get the bigger picture of how things are managed in a mature k8s environment at scale. I'm a beginner starting out with Kubernetes and related tools in general, although I understand these approaches oftentimes there's confusion on how and whether are these applicable in my practice environment. (on prem cluster). Would be great if you could recommend a somewhat advanced project in order to get better at Kubernetes and the related tools. Thank you for your time reading this long paragraph.
thanks, love this advanced use cases!! I use reflector, to clone secret between namespaces within the same cluster and or eso with kubernetes provider to share secrets between clusters
We use it in my job the ESO, but the big problem in crossplane is reading secrets in composition without using a status for retaining this information.
Read the title singing Lando's song... "Secrets up and down side to side like a roller coaster..." 😂
Plot twist: If you send your secret to a "secret manager" or an ESO, the secret is no longer "secret". Just because something works, it does not mean it works securely...
Not directly related but doesn't have Cross plane a depends on tree for delaying execution loops until requirements are met?
Been using vault webhook mutate and seems better for apps since injects secrets into running scope but it's harder to integrate
There is no need (most of the time) for a dependency loop since each resource has its own control loop. Those that cannot be created right away are created when the data they need (often available in other resources) becomes available.
Hi Victor, firstly thank you for sharing your wisdom :) I have one small question - looking on the final diagram. (besides other) you have created "external secret" in order to pull "root password" from aws secrets manager. How that "root password" was created? How did you put the password there? Was it generated? Or does the aws rds have even the capability to create secrets manager secret after the creation of rds? thank you
I generated that password with s simple aws command. It can be random if needed.
I don't generate secrets like that one through manifests since that might end up being accidentally pushed to git.
Have you seen by any chance ESO constantly updating K8s secrets, even though the value hasn't changed? Currently when looking at Events, some "externalsecret" objects are sitting at 27k+ Count..
I haven't had that problem. What's in events when you describe that external secret?
@@DevOpsToolkit The reason is updated without the secret being updated.. You can see the count yourself
│ Normal Updated 2m19s (x2487 over 41h) external-secrets Updated Secret │
Is ESO now the preferred method of integrating secrets with an external store (thinking Azure KeyVault in my case) considering it supports both push and pull now? Also, does this mean Crossplane's ESS (External Secret Stores) alpha feature will be dropped, or do they serve slightly different purposes?
I prefer ESO (not sure about others).
Crossplane ESS is limited to Vault (at least now) while using ESO directly allows quite a few other providers.
Not sure what is on the roadmap with Crossplane ESS...
@@DevOpsToolkit Is not ESO significantly less secure compared to aws ascp and similar?
@michals.2604 are you referring to ascp combined with SSCSID? If that's the case, the answer is yes. SSCSID is more secured since it does not use kubernetes secrets but volumes so data is not stored in etcd. That being said, whether storing something in etcd is secure or not depends on many factors like whether etcd is encrypted (it is in eks), whether it is accessible directly (it isn't in eks), etc. A more important issue is that you won't be able to use it with most third party apps since they are typically designed to expect secrets and not volume mounts.
Isn't password rotation an issue here? There would clearly be an outage when the password is changed depending on how often the secretsmanager contents are synced.
The proper way to roll a password over without downtime would be to allow the old and new password during a certain time window but it seems to leak the entire "magic" of this setup when done
If you do need both the new and the old password (which i don't think you should need), you should use separate entries in your secrets manager. In any case, unless you plan to create secrets manually, the only reasonable path forward is with some operator.
@@DevOpsToolkit yeah I just mean it would be nice if the external secret would also abstract password rotation away. It's very difficult because it depends on the specifics of the actual systems using and providing the password
I've seen many apps go down because some team goes full YOLO and changes the password in the password manager without actually checking where it gets synced in the cluster and then everything is just crashlooping, so clearly a drawback to the "magic" of just having a secret show up in k8s. It wouldn't be a problem if the abstraction of ExternalSecret took into account what kind of workflow is needed to "change" the value (both pull and push aspect of it). It's just a building block so something more is needed if you don't want downtime, otherwise "eventually" it will be alright
some companies still have dumb policies like rotating every password at least once a year etc
Password rotation must follow the same rules as, let's say, changes to APIs or deployments of new releases. Changes need to be backwards compatible, at least until all start using it. Simply changing something does not work, no matter whether that's a change to secrets, APIs, releases, or anything else.
I'm confused about this external secrets operator in general; seems like we alreay need a secret to connect to the "external source of secrets". Doesn't that basically defeat the purpose? I mean the secret will be there if a bad actor gets access to the cluster, and they can use it to get the actual secrets, no?
maybe you might look into spiffe / spire. It solves the bottom turtle problem...
It's more for convenience and less for security, but the bad actor would need to get access to the control plane so it is for sure more secure than giving the app your aws root accesskey :P
Good practice would be to use IRSA to connect to the external source of secrets instead of credentials stored in a secret
Sorry, what's IRSA?
@CarmineEspositoIT I wanted to make it more general so that it can apply to almost anything. Still, you're completely correct. If using AWS, it should be IRSA.