People who aren't DevOps engineers might not fully understand the tremendous value you share in your videos. In this video, you intuitively and comprehensively teach Terragrunt, an amazing tool used by some of the best DevOps teams in the industry, and not only that, you use it as an instrument to easily teach advanced AWS topics. If that wasn't enough, you cover the entire scope of infrastructure environments; development, staging, and production. What more can I say other than it's absolutely perfect?
I've been on the software development (CI) side for 30+ years and only recently had the opportunity to contribute to the CD side of a new startup. Thanks for sharing these great tutorials! Helped me a lot while getting started on my DevOps journey.
i dont think you understand how much your explanation and combining so many different elements of devops into one video has helped me so much. Like, exactly what i was looking for. thank you so much!
Awesome video for using Terraform Mudule based with Terragrunt and store state in aws S3. I followed all the way and now I am maintaining this infrastructure for my production. Waiting for next video related with this video sir.
Terragrunt rocks! We use it for managing complex multi-client infra. Bunch of includes, dependencies, etc :) terraform without terragrunt usable only for very simple projects.
Very Helpful for beginners like me. Appreciate the effort of posting these. I just have question, when I'm adding the Labels and Taints inside the node_groups, terragrunt doesn't seem to detect it. What should I been missing? I just wanted to add multiple taints and labels for each nodegroup. I haven't added sec group as well to the node group, that will might be same instance.
Thanks! Well you would need to add taints to the module itself here - github.com/antonputra/tutorials/blob/main/lessons/160/infrastructure-modules/eks/3-nodes.tf#L26 Example - github.com/antonputra/tutorials/blob/2d1421a4c6b91e8ee22017f8252aee170586b711/lessons/152/terraform/7-nodes.tf#L98-L102 But instead of hardcoding use github.com/antonputra/tutorials/blob/main/lessons/160/infrastructure-modules/eks/3-nodes.tf#L2 variable
@@AntonPutra Thank you, I was able to add in on the module. However, I'm still trying to figure out how to make an object(taint/labels) inside the node groups to not require an input/s. I have set labels and taints inside the module by using "each.value" but when I skip it on a node group, it fails saying "Unsuitable value for var.node_groups set using the TF_VAR_node_groups │environment variable: all map elements must have the same type." Still looking on what should I have to add to make this possible. Here what I add on module tho, "labels = each.value.labels" taint { key = each.value.taint.key value = each.value.taint.value effect = each.value.taint.effect }
I created my channel just to comment on this video. This is video is very good. You are helping a lot of people like me. You are going a great job. Please keep it up. Thanks.
Thanks! Yes, 100% use for_each to iterate over objects in Terraform. count is mostly used as a flag to indicate whether to create an object or not. Do you have an example for me using count?
55:29 It would be amazing to update the project to do all the terraform IAM setup (showcased at the end using the AWS console) using terragrunt as well.
Great Tutorial, make sure to use the same instance size as in the video and NOT t.2 micro. My autoscaler wouldn't launch and I could not figure out why, now I know that it's the instance size. My guess would be that in order for autoscaler to work - instance size should not be less than t3a.xlarge. Anyway, thanks Anton, I learned a lot
Do you have any recommendations for locking down access to an EKS cluster? For example, would using my private IP to restrict and manage admin access to the cluster be a good approach? Thanks for a fantastic tutorial by the way!
Hi Anton Putra, Thank you for the excellent tutorial on setting up an EKS cluster! It was incredibly helpful and easy to follow. I wanted to ask for your guidance on adding Ubuntu 20.04 nodes to my EKS cluster. Could you provide some instructions or point me towards a resource that explains how to do this? Thanks in advance for your help and for the great content you create!
Sure, you need to build your image using this official packer script - github.com/awslabs/amazon-eks-ami They are called self-managed nodes here is a reference - docs.aws.amazon.com/eks/latest/userguide/worker.html
There's a significant limitation in how to isolate different environments when using Terraform workspaces. I have a dedicated section somewhere in this video where I explain exactly why you would choose folders rather than workspaces - th-cam.com/video/GgQE85Aq2z4/w-d-xo.html
well of course, it does the same function, scale your k8s cluster. it just instead of adjusting desired size of auto scaling group it creates standalone ec2 instances. i have few videos covering karpenter
"Great demo! , first I noticed you configured the Terraform backend (S3 bucket) in each terragrunt.hcl file within the Dev environment subfolders (eks, kubernetes-addons, vpc). Wouldn't it be more efficient to define the backend configuration once at the root level and use Terragrunt's dependencies block to inherit it across modules? Just curious about the approach. second thing: you exclude the eks addons when you applied it , because it depends on eks creation first , do i need to apply it seperately , last thing, why you didn't push the infrastructure-live-v4 " where the Terragrunt files are located" and prefered to keep it local ?....... Thanks!"
Hey Anton, amazing video. The build-up to the final .hcl file put it all together. Can you explain what the differences were between the live-v1/v2/v3/v4 folders and when doing this for a completely different environments, do we need to go through all those steps or can we go straight to v4
Thanks! v1 is just raw terraform code v2 uses terraform modules v2 & v4 both terragrunt and different modules You can go straight to v4 if you use open source modules or build your own and go for v4
great videos! is there tutorial how to add more kubernetes-addons like aws load balancer controllers, efs csi drivers etc. using this deployment method?
Thanks, sure you just need to add all of your addons in this module - github.com/antonputra/tutorials/tree/main/lessons/160/infrastructure-modules/kubernetes-addons and create a specific variable such as enable_alb_controller.
you can use read_terragrunt_config function example: locals { common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl")) } reference - terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config
Hi Anton great tutorial as usual, thanks a lot, but i wanted to ask, how do we handle eks upgrade with terraform , is it enough just by changing the cluster_version and eks will handle the rest, or do we need to do something to scale the cluster first, and do the drain and cordon etc. fact: Putra in my language means prince.
Thank you! Based on my multi-year experience managing Kubernetes clusters in AWS and GCP, sometimes it can be as easy as updating the version and applying Terraform. In that case, Terraform will upgrade the control plane and then node pools one by one (drain, terminate, etc). In some cases, for example, when upgrading to 1.21 or 1.22, there were breaking changes in Kubernetes which forced us to create a brand-new clusters and migrate apps there due to the new and deprecated APIs.
@AntonPutra in production environment, when you need to upgrade the eks version… will you do that via terraform ..? or thats where terraform ends its job. because i guess upgrading a live eks cluster with deployments is risky . Thanks
Not really, EKS would first drain nodes and respect the constraints that you place. You need to make sure you have pod disruption budgets for all your apps to avoid any downtime. kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#configure-poddisruptionbudget
In the long run it’s much easier to maintain your own resources. Less abstractions less dependencies. I’m talking about 1+ year. Less technical debt you acquire is better for you and your team.
I didn't get the question. In Terragrunt, we usually create all infrastructure using Terraform itself and use dependency references between different parts of Terragrunt. Data blocks are usually the exception, not the common practice.
It creates one for you. You can get id of the group using "cluster_security_group_id" output variable and add additional rules using the following resource "aws_security_group_rule".
@@rohmersicat3429 I need to know your use case. For example if you your application (pod) needs access to RDS, you add EKS security groups as source in your configuration. Optionally you can setup Security groups for pods
I keep getting cyclic dependency error when I run "terragrunt run-all plan/apply " on dev folder. I downloaded "Anton Putra's" code and and tried running on it and it still gives same error. Can anyone help me. Also I am using local backend and modules.
Sure, here for exampe - github.com/antonputra/tutorials/tree/main/lessons/167/terraform Also, I have terraform module version - github.com/antonputra/tutorials/tree/main/lessons/156
Great video, I am managing a few clients but but with only 2 env, so don't see that much gain of Terragrunt, when initial setup and refactoring would be very complex. But information is very valueable. The ideo of using terraform modules like these addons is super, gonna start this from Monday :) Another thing that is still stopping me from using Terrafrunt is using Terraform cloud as backend, since i couldn't find good explaind example. Maybe that could be idea for next video? :)
@@kayoutube690 Due to the recent fork of Terraform to OpenTF, I suggest waiting to see how all those "providers" will be adopted by OpenTF. Terragrunt will no longer be able to use Terraform and will switch to OpenTF because of licensing issues.
Sorry to say, but you are going very fast and it becomes very hard to understand and have to watch the video repeatedly which is time wasting.But overall your videos are very helpful.
Thank you very much for this AMAZING video! If you're creating your cluster in a region other than 'us-east-1', it's necessary to modify the kubernetes-addons/1-cluster-autoscaler.yaml file to add the 'awsRegion' value to the Helm chart, as it defaults to searching for the node group in "us-east-1". Otherwise, you'll get the error "Node ip-xxx.xxxxx.compute.internal should not be processed by the cluster autoscaler (no node group config)"
If you change the region to something other than `us-east-1`, the autoscaler needs `awsRegion` to be set in `infastructure-modules/kubernetes-addons/1-cluster-autoscaler.tf`
I figured it out, put the autoscaler.yaml into the `dev/kubernetes-addons/` directory and add the values like normal with `helm_release` `values = [ file("${path.module}/autoscaler.yaml") ]`
🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com
People who aren't DevOps engineers might not fully understand the tremendous value you share in your videos. In this video, you intuitively and comprehensively teach Terragrunt, an amazing tool used by some of the best DevOps teams in the industry, and not only that, you use it as an instrument to easily teach advanced AWS topics. If that wasn't enough, you cover the entire scope of infrastructure environments; development, staging, and production. What more can I say other than it's absolutely perfect?
Wow thanks! I appreciate it!
I've been on the software development (CI) side for 30+ years and only recently had the opportunity to contribute to the CD side of a new startup. Thanks for sharing these great tutorials! Helped me a lot while getting started on my DevOps journey.
Thank you so much!!
i dont think you understand how much your explanation and combining so many different elements of devops into one video has helped me so much. Like, exactly what i was looking for. thank you so much!
Thank you! Glad to hear that!
This is a fantastic tutorial. Extremely helpful to me.
Thank you so much for the support!
Anton you are such a "go-to" guy for this stuff. Keep up the amazing work you are doing for the community. We appreciate you so much. 👊
Thanks Robert!
@@AntonPutra 10/10 videos
Thanks
Awesome video for using Terraform Mudule based with Terragrunt and store state in aws S3. I followed all the way and now I am maintaining this infrastructure for my production. Waiting for next video related with this video sir.
Thanks you!
Thanks!
Thank you, Stephan! I appreciate your support!
Valeu!
thank you!!
Thanks
Terragrunt rocks!
We use it for managing complex multi-client infra. Bunch of includes, dependencies, etc :) terraform without terragrunt usable only for very simple projects.
agree
Really useful having all this information for a beginner. I was tearing my hair out going from blog post to blog post.
thanks Joshua
Anton, you are such a great DevOps Engineer ...keep it up, my friend!
Thank you!
Thanks a lot for a tutorial that covers Terragrunt from the beginning.
Спасибо!
pojaluysta =)
Thank you 🙏 for sharing this i was searching about this yesterday and you uploaded this
Anton you are great.
My pleasure!
Fantastic video! I truly appreciated the concise and fast-paced explanation. I've liked and subscribed. Thank you!
Thank Sergey!
Very Helpful for beginners like me. Appreciate the effort of posting these.
I just have question, when I'm adding the Labels and Taints inside the node_groups, terragrunt doesn't seem to detect it. What should I been missing? I just wanted to add multiple taints and labels for each nodegroup. I haven't added sec group as well to the node group, that will might be same instance.
Thanks! Well you would need to add taints to the module itself here - github.com/antonputra/tutorials/blob/main/lessons/160/infrastructure-modules/eks/3-nodes.tf#L26
Example - github.com/antonputra/tutorials/blob/2d1421a4c6b91e8ee22017f8252aee170586b711/lessons/152/terraform/7-nodes.tf#L98-L102
But instead of hardcoding use github.com/antonputra/tutorials/blob/main/lessons/160/infrastructure-modules/eks/3-nodes.tf#L2 variable
@@AntonPutra Thank you, I was able to add in on the module. However, I'm still trying to figure out how to make an object(taint/labels) inside the node groups to not require an input/s. I have set labels and taints inside the module by using "each.value" but when I skip it on a node group, it fails saying "Unsuitable value for var.node_groups set using the TF_VAR_node_groups │environment variable: all map elements must have the same type." Still looking on what should I have to add to make this possible.
Here what I add on module tho,
"labels = each.value.labels"
taint {
key = each.value.taint.key
value = each.value.taint.value
effect = each.value.taint.effect
}
This channel is very util. Hello from Brazil.
❤️
Thank you so much Anton! Amazing knowledge!
Thank you, Tiago!
As always, great content by Anton on TG. Super helpful. I'll be applying many of these concepts. Thank you!
thank you!!
Amazing explanation, especially the last part. I could relate it with the code we have in our live environment. Got my doubts cleared. Thanks.
thank you!
I created my channel just to comment on this video. This is video is very good. You are helping a lot of people like me. You are going a great job. Please keep it up. Thanks.
thank you! ❤️
Fantastic video. Thank you so much!
Thank you Chris!
Excellent
Thanks
@@AntonPutra can you make some videos on Kubernetes CRD and some automation on that.
@@sujithsurendran7686 Operator is coming soon, but what do you mean by automation?
Top quality, thanks for sharing 🎉
Thank you Andres!
Nice video and perfect speed. Spasibo
Spasibo vam =)
спасибо, кучу времени сэкономил)
pojaluysta :)
Thank you, Anton! Amazing tutorial.
thank you!
Thanks Anton, great content. Just a question: isn't better to use for_eash instead of count in subents?
Thanks! Yes, 100% use for_each to iterate over objects in Terraform. count is mostly used as a flag to indicate whether to create an object or not. Do you have an example for me using count?
55:29 It would be amazing to update the project to do all the terraform IAM setup (showcased at the end using the AWS console) using terragrunt as well.
Thanks for the feedback
Great Tutorial, make sure to use the same instance size as in the video and NOT t.2 micro. My autoscaler wouldn't launch and I could not figure out why, now I know that it's the instance size. My guess would be that in order for autoscaler to work - instance size should not be less than t3a.xlarge. Anyway, thanks Anton, I learned a lot
Cool, thanks for the tip
You are the best. The tutorial is really good. Thanks for it. Please keep it up.
thank you!
Do you have any recommendations for locking down access to an EKS cluster? For example, would using my private IP to restrict and manage admin access to the cluster be a good approach? Thanks for a fantastic tutorial by the way!
Great tutorial, love it! What do you think of Terraspace?
Thanks! I haven't used Terraspace yet.
Hi Anton Putra,
Thank you for the excellent tutorial on setting up an EKS cluster! It was incredibly helpful and easy to follow.
I wanted to ask for your guidance on adding Ubuntu 20.04 nodes to my EKS cluster. Could you provide some instructions or point me towards a resource that explains how to do this?
Thanks in advance for your help and for the great content you create!
Sure, you need to build your image using this official packer script - github.com/awslabs/amazon-eks-ami
They are called self-managed nodes here is a reference - docs.aws.amazon.com/eks/latest/userguide/worker.html
hi Antonio, could you please elaborate why you preferred to use folders named by the env name (dev/staging) rather than the terraform workspaces?
There's a significant limitation in how to isolate different environments when using Terraform workspaces. I have a dedicated section somewhere in this video where I explain exactly why you would choose folders rather than workspaces - th-cam.com/video/GgQE85Aq2z4/w-d-xo.html
is there possibility to replace cluster autoscaler with karpenter?
well of course, it does the same function, scale your k8s cluster. it just instead of adjusting desired size of auto scaling group it creates standalone ec2 instances. i have few videos covering karpenter
@@AntonPutra this channel should be on top. The content you are providing is marvelous. thanks a lot
"Great demo! , first I noticed you configured the Terraform backend (S3 bucket) in each terragrunt.hcl file within the Dev environment subfolders (eks, kubernetes-addons, vpc). Wouldn't it be more efficient to define the backend configuration once at the root level and use Terragrunt's dependencies block to inherit it across modules? Just curious about the approach. second thing: you exclude the eks addons when you applied it , because it depends on eks creation first , do i need to apply it seperately , last thing, why you didn't push the infrastructure-live-v4 " where the Terragrunt files are located" and prefered to keep it local ?....... Thanks!"
Hey Anton, amazing video. The build-up to the final .hcl file put it all together. Can you explain what the differences were between the live-v1/v2/v3/v4 folders and when doing this for a completely different environments, do we need to go through all those steps or can we go straight to v4
Thanks!
v1 is just raw terraform code
v2 uses terraform modules
v2 & v4 both terragrunt and different modules
You can go straight to v4 if you use open source modules or build your own and go for v4
great videos! is there tutorial how to add more kubernetes-addons like aws load balancer controllers, efs csi drivers etc. using this deployment method?
Thanks, sure you just need to add all of your addons in this module - github.com/antonputra/tutorials/tree/main/lessons/160/infrastructure-modules/kubernetes-addons and create a specific variable such as enable_alb_controller.
Amazing Video🌟
got a question, how could Github infra accessed private Github module without a token ?
go to settings and add "deploy key"
Thanks for the video. very insightful.❤
quick query: How can we share values between staging and production using this architecture?
you can use read_terragrunt_config function
example:
locals {
common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl"))
}
reference - terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config
Hi, in 58:40 when i run terragrunt run-all plan command, it's not working. showing can't read resource with the name of eks
Check if you used the same name for the cluster
Hi Anton great tutorial as usual, thanks a lot, but i wanted to ask, how do we handle eks upgrade with terraform
, is it enough just by changing the cluster_version and eks will handle the rest, or do we need to do something
to scale the cluster first, and do the drain and cordon etc.
fact: Putra in my language means prince.
Thank you! Based on my multi-year experience managing Kubernetes clusters in AWS and GCP, sometimes it can be as easy as updating the version and applying Terraform. In that case, Terraform will upgrade the control plane and then node pools one by one (drain, terminate, etc). In some cases, for example, when upgrading to 1.21 or 1.22, there were breaking changes in Kubernetes which forced us to create a brand-new clusters and migrate apps there due to the new and deprecated APIs.
Where's the link for the part 2?
@AntonPutra
in production environment,
when you need to upgrade the eks version…
will you do that via terraform ..?
or thats where terraform ends its job.
because i guess upgrading a live eks cluster with deployments is risky .
Thanks
Not really, EKS would first drain nodes and respect the constraints that you place. You need to make sure you have pod disruption budgets for all your apps to avoid any downtime.
kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#configure-poddisruptionbudget
What a nice video sr!
very clean tf code. good work
we can use terragrunt to wrap up for a bigger infrastructure on a same cloud provider
It can be a library for all resources
agree it's very helpful when you have lots of environments and components
i'm curious, is there a reason why the tf files were prefixed with numbers ? like the below
0-provider, 1-vpc and so on
It's just for the tutorial to sort them in the IDE and show the logical order of how you would create infrastructure components.
11:42 that dev/vpc/dev/vpc/terraform.tfstate file doesn't make completely sense
Hi, do you recommend Terragunt for large scale projects? sometime it becomes hard to maintain.
I do, especially for big companies with lots of environments, but keep it simple and beginner friendly.
thumb up and subscription strictly required!!!
Thanks!
hi thanks a lot for the video!!
can we migrate existing terraform code running on AWS to Terragrunt is that easy to do?
Yes, you can, but it's not easy since you need to import state for all components you already have.
thank you bro!
Why you wrote your own module instead of using official VPC module ? is there a reason or just for teaching purposes
In the long run it’s much easier to maintain your own resources. Less abstractions less dependencies. I’m talking about 1+ year. Less technical debt you acquire is better for you and your team.
How do you reference data blocks that we use to fetch from AWS like authentication credentials in terragrunt?
I didn't get the question. In Terragrunt, we usually create all infrastructure using Terraform itself and use dependency references between different parts of Terragrunt. Data blocks are usually the exception, not the common practice.
Hello master can we do manage own security group instead of using the default one ? and if that so how can i add the security group to use with K8s ?
It creates one for you. You can get id of the group using "cluster_security_group_id" output variable and add additional rules using the following resource "aws_security_group_rule".
@@AntonPutra yes what i mean if i want to allow port 8080 how can i add it on vpc ?
@@rohmersicat3429 I need to know your use case. For example if you your application (pod) needs access to RDS, you add EKS security groups as source in your configuration. Optionally you can setup Security groups for pods
AMAZING!!!
Thanks, Lukas!
I keep getting cyclic dependency error when I run "terragrunt run-all plan/apply " on dev folder. I downloaded "Anton Putra's" code and and tried running on it and it still gives same error. Can anyone help me. Also I am using local backend and modules.
Could you provide this same example but without using terragrunt? Just terraform?
Sure, here for exampe - github.com/antonputra/tutorials/tree/main/lessons/167/terraform
Also, I have terraform module version - github.com/antonputra/tutorials/tree/main/lessons/156
Amazing!!!
Thanks Ian! =)
Do you have any git link to have the code please
Hello, why is my pod not triggering for scale up of the worker nodes, `pod didn't trigger scale-up`.
Best way to find out is to check autoscaler controller logs. Kubectl logs …
Best way to find out is to check autoscaler controller logs. Kubectl logs …
Great video, I am managing a few clients but but with only 2 env, so don't see that much gain of Terragrunt, when initial setup and refactoring would be very complex. But information is very valueable. The ideo of using terraform modules like these addons is super, gonna start this from Monday :)
Another thing that is still stopping me from using Terrafrunt is using Terraform cloud as backend, since i couldn't find good explaind example. Maybe that could be idea for next video? :)
Thanks, well you can always create your own or try to use www.runatlantis.io
how to know when should use Terragrunt instead Terraform?
when you have more then 2 envs
I think git as a source is difficult to use specially if the organizations are using MFA auth for their git. But over all your video is awesome!
I disagree. GitOps is a standard; even in Google SRE books, they specifically mention following the same git approach.
You're right! But I'm referring only to jenkins. I never use GitOps but i would love to explore this tool.
@@kayoutube690 Due to the recent fork of Terraform to OpenTF, I suggest waiting to see how all those "providers" will be adopted by OpenTF. Terragrunt will no longer be able to use Terraform and will switch to OpenTF because of licensing issues.
Why you have no ".terraform" dir when init ?
i believe it's just hidden
Can you share the code
Sure, you'll find link the video description
@@AntonPutra please provide email to keep in touch. Some projects I have
Sorry to say, but you are going very fast and it becomes very hard to understand and have to watch the video repeatedly which is time wasting.But overall your videos are very helpful.
Thanks for the feedback, trying to make them short and concise
Thank you very much for this AMAZING video!
If you're creating your cluster in a region other than 'us-east-1', it's necessary to modify the kubernetes-addons/1-cluster-autoscaler.yaml file to add the 'awsRegion' value to the Helm chart, as it defaults to searching for the node group in "us-east-1". Otherwise, you'll get the error "Node ip-xxx.xxxxx.compute.internal should not be processed by the cluster autoscaler (no node group config)"
If you change the region to something other than `us-east-1`, the autoscaler needs `awsRegion` to be set in `infastructure-modules/kubernetes-addons/1-cluster-autoscaler.tf`
You can set it using this variable - github.com/antonputra/tutorials/blob/main/lessons/160/autoscaller.yaml#L66
@@AntonPutra How is that yaml file being consumed? Perhaps I needed to re-init terragrunt?
I figured it out, put the autoscaler.yaml into the `dev/kubernetes-addons/` directory and add the values like normal with `helm_release`
`values = [
file("${path.module}/autoscaler.yaml")
]`