For Production, I would chose CloudFormation + ECS/Elastic Beanstalk + RDS. With ECS/Elastic Beanstalk, you can directly deploy your container, without having to worry about the docker host, set/change environment variables during deploy, easy to scale up/down
Yes I agree - for production or even some non-production environments, I would pick CloudFormation + ECS + RDS. The purpose of this video was to demonstrate CloudFormation with some use-case for which I chose Docker installation. I was expecting a comment and feedback like this :) I am thinking to make a few videos around "CloudFormation + ECS + AutoScale" which I think might be a series of short videos or a course. Thanks!
Amazing work Chandra, generally it is tough to find intermediate material on any technology, appreciate your work please keep on rolling new videos, on devops.
I like the way you edit your videos, which makes it crisp, clear and concise. If I may suggest, 3:47-4:00 you should mask your AWS secrets as anyone can use it to hack your AWS account (hope you deleted the IAM user used in this video :) )
Thanks for the feedback and suggestions @Bennette Stephen John. I thought of masking the AWS secrets but I chose to clean up all the resources and api keys etc after finishing up the video. In some cases I think masking will still be necessary. Thanks!
Excelent work Chandra! This was the introduction I needed. Many thanks! I'm interested in the enhancements you mentioned about the scripting section (UserData). What would be the options within CloudFormation to take this part to the next level? Would you recommend any other solutions that might be easy to integrate/learn with CloudFormation? Thanks again for all your work!
Thanks for the feedback Vikas. I am not exactly sure I understood your question. If you're trying to automate the process of creating docker image on every code commit and that you want that image be pushed over to the registry, that would be to create a simple or part of CI (Continuous Integration) pipeline. If that's what you're looking to build using CloudFormation and entirely on AWS, here is a high level of what goes into CloudFormation stack definition, 1. Create code repository resource (AWS::CodeCommit::Repository); You would use this as your source repo. 2. Create container image registry resource (AWS::ECR::Repository); This is where you would want the docker images pushed. 3. Create code build project resource (AWS::CodeBuild::Project), the source for this will be CodeCommit repo. Environment of CodeBuild project is where you would define the instructions to build docker image and push to registry. You will probably need to define buildspec.yml in the root of your code repo with necessary instructions to build & push. You may configure code build project to trigger on every code commit. I haven't used CodeBuild but if I were to create a CI on AWS, this is how I would go about. Now you can use Github code repository instead of CodeCommit in which case the Source definition of CodeBuild project will require some additional inputs. I hope this helps.
This is not a recommend approach for a real production stack but if you are just trying to automate the process of what is being done in this tutorial, one quick option would be to use CloudFormation Output variable. You can define CloudFormation output, value of which will be "!GetAtt DatabaseInstance.Endpoint.Address". Once the stack is created, you could write a small shell script to describe stack (using AWS CLI - aws cloudformation describe-stack ..), grep the output value and assign it to an env variable. In the compose file app.yml, read the database endpoint from that environment variable. I don't think you can avoid waiting but have the script wait until the RDS instance is ready before starting docker app. As I mentioned in this short tutorial, you could use CloudFormation itself to automate your stack (infra & app). Eg: To define and automate tracking of completion state of resources and then deploy the application. ECS is one good option where you don't deal with docker compose but define all the container services in CloudFormation template.
Hi. Your videos are very helpful. I am referring those for my daily tasks. Could you please make a video on CloudFormation for EMR with "Bootstrap Scripts(Postgres/MySql DB installation) & Step Execution(Spark Job)" Thanks
Thank you Prashant. I appreciate your feedback and request. When time permits I will look into making a tutorial on EMR but I can't really promise now. If I can't make the video, I will try to post some code examples on github.com/devteds .
I had one question in docker.How the developers work culture would be using containerization concept like check in and checkouts and how the developer environment is to be created on demand to test their own code etc
In general containerized development environment helps the developers in several ways, 1. Consistent runtime, libraries, database versions - that means developers don't have to worry about installing, upgrading software versions and focus on application development. No tooling blockers and focus on getting the things done. 2. Consistent OS - this is a big plus. Developers can use any OS of their choice in the local environment. The basic OS & libraries that is needed for application is packaged in the container. This helps when you have a mix of developers that have preference for their primary OS. With the application dockerized, setting up environments on demand or the first time one a developer machine is all quick an easy. You probably need to write some scripts to seed data to the environment you spin up. If the application is not run as containers in production you can still containerize in development and CI environments to use the exact same versions of OS, libraries and softwares. That still helps. Sometimes it is hard to introduce containerization or docker in a dev team. For some it is some new tool to learn which takes back seat. But once introduced the team will notice improvement in productivity. I hope that answers your question.
Using your config, I got an error while the task was creating the EC2 instance. "The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request."
Likely there is no default VPC. If there isn't one already, you might want to create a default vpc and try again. FYI - To create default on aws management console - under services, go to VPCs and from 'Actions' dropdown you should be able to create a new default vpc. Make sure to delete the failed stack before you try running 'aws cloudformation create ... " command. Hope this helps.
Thanks for asking my suggestion but I don't think I can be of much help providing guidance for a devops career path. But I will send you some information and links that might help.
hi your video very nice and helpful, could you please help me to install Java 8 and maven and tomcat on the ec2 instance using cloud formation. thanks in advance please create a video or share any documents.
Thank you Kashish. I am glad you found the videos useful. I don't do much Java but I have that in my list of videos to make. I can't be sure of timeline as when I can publish one on Java. In the meantime if I come across any tutorials or blogs on Java 8 + Maven + Cloudformation, I will send them to you.
Thimma Reddy, Did you mean EBS block device mapping or create some directories to mount to the container? If it's about creating directories to mount to containers, you can add those commands in the UserData. If you're looking to add EBS kinda stuff, you might want to refer the CloudFormation EC2 reference documentation & examples. Following links might help - docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#cfn-ec2-instance-blockdevicemappings docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ec2.html
If you are using mac, you may use the "Terminal" app. The one I use on mac is called "iTerm". If you use windows 10, I think there are some good options such as "Bash on Ubuntu on Windows 10" and OpenSSH client. In the older windows, I don't know if there many options besides Putty or Cygwin.
I keep receiving the below error at cloudformation and the stack gets rolled_back everytime: The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request. Client Request Token:Console-CreateStack-fe29ceb9-82d9-4f4a-9f06-2a3402fd8999 The following resource(s) failed to create: [AppNode]. . Rollback requested by user.
Sorry for the delayed reply. I hope you were able to resolve the issue by now. If not, please refer my response to ddubs123's (one of the viewers) question.
James, I agree that using environment variables for database passwords without any encryption is not secure. Scope of this video was very minimal and is a get-started type for cloud-formation & container application. Besides the database password (or in general secret configs - using some encryption or secret management solutions) there are many other topics that the application team need to consider before the application and infrastructure is production-ready.
@@ChandraShettigar Storing passwords in your code might not be the best way to get started. I know password management is difficult, but it's also fundamental to the practice.
@@JamesTharpe I agree with you on the dev practice. In this case I had to narrow the scope to focus more on CloudFormation basics. May be I should have left a comment about securing database password. Thanks for the feedback.
I really enjoyed that. Given how many painful tutorials are out there. You rock. Thanks.
Thank you Christian 🙂
This is magnificient. I absorbed the know-how in under 25 min sitting. Thank you for your work, chandra!
Rihana, Thank you. I am glad you found it helpful.
This is shockingly well made.
Thank you.
Just 20 minutes of this session and I now find it way more easier, thanks :)
Thanks Dheeraj. I'm glad it was helpful.
I like the pace of your tutorial. This would easily be 2 hours long if some of those other guys do it.
Thank you Ravi.
For Production, I would chose CloudFormation + ECS/Elastic Beanstalk + RDS. With ECS/Elastic Beanstalk, you can directly deploy your container, without having to worry about the docker host, set/change environment variables during deploy, easy to scale up/down
Yes I agree - for production or even some non-production environments, I would pick CloudFormation + ECS + RDS. The purpose of this video was to demonstrate CloudFormation with some use-case for which I chose Docker installation. I was expecting a comment and feedback like this :) I am thinking to make a few videos around "CloudFormation + ECS + AutoScale" which I think might be a series of short videos or a course. Thanks!
Amazing work Chandra, generally it is tough to find intermediate material on any technology, appreciate your work please keep on rolling new videos, on devops.
Thank you Adarsh 🙂
Very good presentation Chandra - very simple and elegant
Thank you rajdeeprandhawa1
I like the way you demonstrate this video.Good job
Thanks Shivaji. I hope this was useful too.
Thanks for quick overview on CloudFormation....this helped me lot !!
Thanks for the feedback Velu.
I like the way you edit your videos, which makes it crisp, clear and concise. If I may suggest, 3:47-4:00 you should mask your AWS secrets as anyone can use it to hack your AWS account (hope you deleted the IAM user used in this video :) )
Thanks for the feedback and suggestions @Bennette Stephen John. I thought of masking the AWS secrets but I chose to clean up all the resources and api keys etc after finishing up the video. In some cases I think masking will still be necessary. Thanks!
great presentation style, keep up the good work
Thanks for the kind words, Sagar.
Excellent, Please create more videos on Cloudformation using cli.
Thank you Sarfaraz. Please check Episode #9 (th-cam.com/video/Gr2yTSsVSqg/w-d-xo.html) which is also a CloudFormation CLI but for ECS stack.
Right speed for a typical human brain to consume without buffering
Thank you Pluasok. I am still trying to adjust the pace so it is not too fast for many. I am glad you found the pace right.
excellent video man, thanks for sharing your knowledge
Thank you Pankaj
Thanks Chandra, this tutorial is awesome!
Miklos, you're so welcome!
Excelent work Chandra! This was the introduction I needed. Many thanks! I'm interested in the enhancements you mentioned about the scripting section (UserData). What would be the options within CloudFormation to take this part to the next level? Would you recommend any other solutions that might be easy to integrate/learn with CloudFormation? Thanks again for all your work!
Exquisite! Thanks for sharing.
Pandian, Thank you.
Excellent way of explanation !! thanks for sharing this.
Thank you Santosh.
Awesome. Too good explanation.
Thank you Suresh.
Awesome!!! Precise and Clear..Thanks
Thank you Siddhesh.
Fantastic!!! Very useful information.
Thanks for the feedback Ashutosh.
Excellent Chandra.
Thanks Venkat.
Thanks!! I was an excellent video for beginners.
Thank you PulidoVPE
Very informative and well explained. Thank you.
Thanks Richard Joy. I'm glad it was helpful
Great tutorial. Thanks Chanda.
Could you please guide me how I can create a docker image and push it to ECR using cloudformation. Thanks in advance
Thanks for the feedback Vikas.
I am not exactly sure I understood your question. If you're trying to automate the process of creating docker image on every code commit and that you want that image be pushed over to the registry, that would be to create a simple or part of CI (Continuous Integration) pipeline. If that's what you're looking to build using CloudFormation and entirely on AWS, here is a high level of what goes into CloudFormation stack definition,
1. Create code repository resource (AWS::CodeCommit::Repository); You would use this as your source repo.
2. Create container image registry resource (AWS::ECR::Repository); This is where you would want the docker images pushed.
3. Create code build project resource (AWS::CodeBuild::Project), the source for this will be CodeCommit repo. Environment of CodeBuild project is where you would define the instructions to build docker image and push to registry. You will probably need to define buildspec.yml in the root of your code repo with necessary instructions to build & push. You may configure code build project to trigger on every code commit.
I haven't used CodeBuild but if I were to create a CI on AWS, this is how I would go about. Now you can use Github code repository instead of CodeCommit in which case the Source definition of CodeBuild project will require some additional inputs.
I hope this helps.
If you do more videos on devops for beginners it will be more helpful for us.thank you, I hope u ll
Yes I will Shivanad. I haven't been able to publish in a while but I will try publishing the next one as soon as I can.
@@ChandraShettigar I know Linux commands very well is it okay for devops, but I don't anything abt software development. Help me please
Thanks for tutorial. How to auto-populate RDS hostname in docker-compose yml file, instead of waiting and pasting?
This is not a recommend approach for a real production stack but if you are just trying to automate the process of what is being done in this tutorial, one quick option would be to use CloudFormation Output variable.
You can define CloudFormation output, value of which will be "!GetAtt DatabaseInstance.Endpoint.Address". Once the stack is created, you could write a small shell script to describe stack (using AWS CLI - aws cloudformation describe-stack ..), grep the output value and assign it to an env variable. In the compose file app.yml, read the database endpoint from that environment variable.
I don't think you can avoid waiting but have the script wait until the RDS instance is ready before starting docker app.
As I mentioned in this short tutorial, you could use CloudFormation itself to automate your stack (infra & app). Eg: To define and automate tracking of completion state of resources and then deploy the application. ECS is one good option where you don't deal with docker compose but define all the container services in CloudFormation template.
Hi. Your videos are very helpful.
I am referring those for my daily tasks.
Could you please make a video on CloudFormation for EMR with "Bootstrap Scripts(Postgres/MySql DB installation) & Step Execution(Spark Job)"
Thanks
Thank you Prashant. I appreciate your feedback and request.
When time permits I will look into making a tutorial on EMR but I can't really promise now. If I can't make the video, I will try to post some code examples on github.com/devteds .
amazing and brillaint .
Thanks for the tutorial, during the last step connecting with docker, it says the target machine refused the connection
I had one question in docker.How the developers work culture would be using containerization concept like check in and checkouts and how the developer environment is to be created on demand to test their own code etc
In general containerized development environment helps the developers in several ways,
1. Consistent runtime, libraries, database versions - that means developers don't have to worry about installing, upgrading software versions and focus on application development. No tooling blockers and focus on getting the things done.
2. Consistent OS - this is a big plus. Developers can use any OS of their choice in the local environment. The basic OS & libraries that is needed for application is packaged in the container. This helps when you have a mix of developers that have preference for their primary OS.
With the application dockerized, setting up environments on demand or the first time one a developer machine is all quick an easy. You probably need to write some scripts to seed data to the environment you spin up.
If the application is not run as containers in production you can still containerize in development and CI environments to use the exact same versions of OS, libraries and softwares. That still helps.
Sometimes it is hard to introduce containerization or docker in a dev team. For some it is some new tool to learn which takes back seat. But once introduced the team will notice improvement in productivity.
I hope that answers your question.
Thanks
Awesome , good job chandra
Thank you Maheeza Sharma
Using your config, I got an error while the task was creating the EC2 instance. "The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request."
Likely there is no default VPC. If there isn't one already, you might want to create a default vpc and try again. FYI - To create default on aws management console - under services, go to VPCs and from 'Actions' dropdown you should be able to create a new default vpc. Make sure to delete the failed stack before you try running 'aws cloudformation create ... " command. Hope this helps.
really super one!!
Thank you.
awesome
Thank you Satish.
well done.
Thank you Orest Ivasiv.
Hi sir,,, am unable to install Jenkins via cloud formation... Plz help with script for installing Jenkins via cloud formation
hi man, thanks for sharing. I have a question that what should I do/ follow/ learn if I want to be an DevOps? --thx a lot
Thanks for asking my suggestion but I don't think I can be of much help providing guidance for a devops career path. But I will send you some information and links that might help.
hi your video very nice and helpful, could you please help me to install Java 8 and maven and tomcat on the ec2 instance using cloud formation. thanks in advance please create a video or share any documents.
Thank you Kashish. I am glad you found the videos useful. I don't do much Java but I have that in my list of videos to make. I can't be sure of timeline as when I can publish one on Java. In the meantime if I come across any tutorials or blogs on Java 8 + Maven + Cloudformation, I will send them to you.
How can add extra root volume to the instance when it is creating
Thimma Reddy, Did you mean EBS block device mapping or create some directories to mount to the container? If it's about creating directories to mount to containers, you can add those commands in the UserData. If you're looking to add EBS kinda stuff, you might want to refer the CloudFormation EC2 reference documentation & examples. Following links might help -
docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html
docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#cfn-ec2-instance-blockdevicemappings
docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ec2.html
Chandra Shettigar I am creating an Beanstalk Env by using cloud formation, when it is launching instance it has to add 20 GB root volume to it
How do I access the terminal that you are talking about?
use Putty. then SSH
If you are using mac, you may use the "Terminal" app. The one I use on mac is called "iTerm". If you use windows 10, I think there are some good options such as "Bash on Ubuntu on Windows 10" and OpenSSH client. In the older windows, I don't know if there many options besides Putty or Cygwin.
I keep receiving the below error at cloudformation and the stack gets rolled_back everytime:
The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.
Client Request Token:Console-CreateStack-fe29ceb9-82d9-4f4a-9f06-2a3402fd8999
The following resource(s) failed to create: [AppNode]. . Rollback requested by user.
Sorry for the delayed reply. I hope you were able to resolve the issue by now. If not, please refer my response to ddubs123's (one of the viewers) question.
This is a terrible way to handle passwords for your database.
James, I agree that using environment variables for database passwords without any encryption is not secure. Scope of this video was very minimal and is a get-started type for cloud-formation & container application. Besides the database password (or in general secret configs - using some encryption or secret management solutions) there are many other topics that the application team need to consider before the application and infrastructure is production-ready.
@@ChandraShettigar Storing passwords in your code might not be the best way to get started. I know password management is difficult, but it's also fundamental to the practice.
@@JamesTharpe I agree with you on the dev practice. In this case I had to narrow the scope to focus more on CloudFormation basics. May be I should have left a comment about securing database password. Thanks for the feedback.