Terraform with Multiple State Files
ฝัง
- เผยแพร่เมื่อ 25 ส.ค. 2024
- Terraform with Multiple State Files | Sign up to our free Discord community: / discord
SUBSCRIBE: bit.ly/2WBPF1W
JOIN THE COMMUNITY: / discord
About The Cloud Coach:
Michael Crilly, aka "The Cloud Coach", is a leading expert in DevOps and automation. The Coach has automated everything from Cisco hardware to AWS resources using various tools like Python, Ruby, Go, Terraform, Packer, CloudFormation, CI/CD, JenkinsCI, GitOps, and much more. The Coach also loves to teach everyone and anyone about the joy of automating complex Cloud resources to deliver a fast, better infrastructure that's cheaper to operate and more stable to manage.
Connect with The Cloud Coach:
My Discord community: / discord
The VIP program: www.thecloud.c...
The Newsletter: www.thecloud.c...
Checkout my FREE courses!
What are IAC & CAC?: www.thecloud.c...
Terraform Crash Course: www.thecloud.c...
Ansible Crash Course: www.thecloud.c...
Packer Crash Course: www.thecloud.c...
Checkout my PREMIUM courses!
Terraform Crash Course: Premium: www.thecloud.c...
Terraform: Understanding State: www.thecloud.c...
CI/CD Introduction: www.thecloud.c...
Great video! Definitely a big fan of Terragrunt when it comes to the multi statefile setup, definitely cleaner!
I've never seen the appeal of Terragrunt, but I came into Terraform at 0.12.
Thank you Michael, I have been searching for a long time the way to pass values from one project to another ! (data + terraform_remote_state)
Loved it !! You clear my doubt as i am implementing terraform for an organization. Thanks for help.
Great stuff man, I love the separation of concern. Thanks
It’s the only way to go!!
Brilliant one!! thanks for the video Michael ...my neurons are making new connections ... how does a module fit-in in this scenario ..whew!! thanks for sharing.
You’re welcome!
Well-explained and concise, as always. Thanks!
Great video! It helped me to prepare for the Terraform certification.
Awesome! Let us know how you get on!
As always clear. Thanks for sharing bro :)
Bless you my friend
Great! really helpful and easy to understand. Keep up the awesome work man!
That's a great piece of info you just presented. Great job!
Whilst this does have some use cases, I've typically found this to be the much more work for the devops eng to deal with. Terraform is very good at discovering how a change affects multiple resources in the state file. Separating out to separate states doesnt let tf discover/resolve those. terraform_remote_state can help a little. I think splitting out resources to separate state files can be limited to resources common across several environments/projects is ok - e.g. centralised log solution. One other possible case is per-developer environments. One state file per dev allows for developers to change their own env without affecting other devs, allowing new development to be prototyped, and a simple "terraform plan" tells them how to get back to the standard setup. Thanx for the content
Some great points there mate. Overall I think as a tool Terraform has to be deployed in a manner that suits the environment and expectations of the business. So if you’re happy with long execution times and all in one state files then that’s fine too.
Don’t forget with large enterprises and governments there’s often a need to separate out access to systems based on an engineer’s role in the organisation. Don’t want developers manipulating IAM policies to suit their needs for example.
Excellent explanation. Thank you.
Can you please do a video on how Terraform project should be created for multiple environments.
Will do! Coming soon!
I used work spaces for multiple stages, but I am looking at how I can deploy multiple stages for each customer
@@MichaelCrilly hi, is this video released? Would love to see it for a project I have to setup.
On our team we were looking into ways we could speed up our deploys. This might be useful. I’d need to see the speed benefits gained vs development time to refactor to split up some resources into separate states. We also do a similar thing you mentioned and put all of our policies and roles into a different state that can only be ran manually by a select few people (instead of allowing ci cd to modify them). Good info!
It will speed things up for sure. I believe granular states can introduce complexity but also a lot of flexibility, security and speed when managing them.
Awesome video with explanation! Thanks a lot!
You’re very welcome.
I love the knowledge that you share!
Thank you for the great content! That was really clarifying
very good explanation. i get involved into your courses
Wow!!
Thank you Mike, awesome
Glad you liked it!
Thanks a lot
The problem with this approach is you can’t terraform plan everything together before doing any terraform applys. Say you added a new resource to ‘networking’, outputted it and referenced it in ‘application’ - A terraform plan on ‘networking’ would complete but a terraform plan on ‘application’ would fail due to the outputs not being in the networking remote state file yet. This makes CI/CD harder as let’s say these changes were all part of the same commit, and you wanted to plan them all as part of a PR. It would fail due to the dependencies.
New to this, but I believe that's one of the problems tools like Atmos from @cloudposse try to solve.
Thank you very much. That helps
You're welcome!
Very nice!
Excellent video Michael! Thanks for sharing it.
One question: what if the remote state isn't up-to-date with the actual system? Using provider supplied data seems more reliable from this point of view. What are the tradeoffs of both solutions? Does a scenario using remote state for "static"/config data coming from TF, and provider for data about the live system, makes sense, and would it be reliable and maintainable?
On to next one of yours...
Cheers from France :)
Great video Michael. :)
Does this separation of state files helps in locking state file with another admins? Maybe I missed it in the video. Thank you and you got a good flow in videos and great explenation. :)
Thanks mate. Yes, when you use multiple state files you can more easily lock them all individually, allowing multiple admins to work across different states at the same time.
brilliant. More real time training would be a good one to focus on by taking aproject and start working on it, including showing the docummentation and demonstrate how one can understand documentation would be the hot cake.
I’m working on it, actually :-)
best video ever !
I agree ;-)
@@MichaelCrilly 😀👍
If you wanted seperate env's, say, Test, Pre-Prod, Prod etc. would you use the same code with all the same resources in each of the env's and then just plan and apply those changes to the env of choice hence not breaking prod when trying to add a new resource?
That’s one way of doing it. You can use tfvars files to help with this.
@@MichaelCrilly Cool, many thanks :)
This looks great, but I wonder how to put this in a yaml file where you end up having a bunch of resources dependencies, e.g. an API needs to be in a VNET, so if we change the api we need to make sure VNET exists and so on
YAML file?
You will have a tightly coupled dependencies between an API and a VNET whether you use multiple state files or not :-)
have you tried using data "my network resource" calls from application instead of data "terraform_remote_state" ? my concern with using terraform_remote_state is that you're depending on a state file that could be using another terraform version.
Good point. However I’d simply pin the Terraform version in the code. Problem solved.
How would you scale this to work with multiple stages dev, QAT, UAT, and live as well as duplicating all the stages for each for multiple use cases.
A case by case basis, really. Most of the time dev doesn’t perfectly match prod, for example. So can dev be one state? Possibly!
I am struggling with a issue regarding remote state management for quite a while now. So, actually while using this technique we are only able to use 5 output attributes from the remote state file. I have a requirement of using 6 attributes from 1 state file for using in further deployment. Can anybody suggest how do I achieve that? It'll be really very helpful and I'll be able to sleep.
Join The DevOps Lounge and ask there: discord.gg/devopslounge
@@MichaelCrilly I am not able to join discord that's why, if you can give a clue here only. It will be helpful.
It’s really difficult to answer here. Sorry. It’s too restricted an environment.
@@MichaelCrilly yeah, I understand. No problem, thanks anyways
how does this work when using prod and dev workspaces?
Good question. You'll have to jump on the Discord to discuss.
Hello, I like the way how you describe it. How i can get this file for tranning perpose? or if you have it in git and i can fork it ?
No git repo, sorry.
Hi, have you thought of doing a video and demonstrating how to build 5 Linux VM's with a shared subnet to everyone?...seems like there's no such tutorial on TH-cam, and yes you can check it out your self :)
That’s just putting the instances in the same subnet. What challenges can I help you overcome?