Configure GitLab CI Runner with Docker executor using AWS EC2
ฝัง
- เผยแพร่เมื่อ 25 มิ.ย. 2024
- In this tutorial, I will show you how to Configure your own GitLab Runner with a Docker Executor using AWS EC2.
🔻 Click on SHOW MORE 🔻
This is useful if you are trying to use GitLab.com and are running run out of minutes or simply need more power.
I will show you how you can run your own GitLab Runner on AWS and cover:
- creating a new virtual machine (VM) on AWS using EC2
- installing the GitLab Runner on the VM
- configuring the GitLab Runner to work with GitLab.com
--
⭐️ Video contents ⭐️
⌨️ 00:00 - Overview
⌨️ 01:33 - Create a VM on AWS EC2
⌨️ 05:28 - Install dependencies (Git, Docker)
⌨️ 08:46 - Download & install the GitLab Runner
⌨️ 09:40 - Register the GitLab Runner with GitLab
⌨️ 15:42 - Test the GitLab Runner
⌨️ 18:32 - Test Docker build and push
⌨️ 21:28 - Enable the privileged mode
⌨️ 27:07 - Conclusion & next steps
⭐️ Commands & resources mentioned ⭐️
A Brief Guide to GitLab CI Runners and Executors
/ a81b9b8bf24e
A list of the most important commands used in this tutorial:
Update dependencies:
sudo yum -y update
Install Git:
sudo yum install -y git
Install Docker:
sudo amazon-linux-extras install -y docker
Add the ec2-user user to the docker group:
sudo usermod -aG docker ec2-user
Ensure the Docker daemon starts automatically:
sudo systemctl enable docker
Restart your EC2 instance:
sudo reboot
Download the GitLab Runner installation package:
curl -LJO "gitlab-runner-downloads.s3.am..."
Install the GitLab Runner:
sudo rpm -i gitlab-runner_amd64.rpm
Register the GitLab Runner:
sudo gitlab-runner register
Edit the config.toml file:
sudo vim /etc/gitlab-runner/config.toml
The GitLab CI pipeline config used to test the runner:
gitlab.com/-/snippets/2300614
⭐️ 🎓 GitLab CI/CD ONLINE COURSE ⭐️
Want to learn more about CI/CD pipelines? Check my GitLab CI online course.
Get it at a special price and help support this TH-cam channel:
www.vdespa.com/courses
--
⭐️ 💬 I have a question. ⭐️
I do my best to answer all comments here on TH-cam but I cannot make any guarantees.
If you are asking a question, make sure to include as many details as possible.
--
⭐️ 💡 I have a video idea ⭐️
Do you want me to create a video on a specific topic? Just fill out the form below:
forms.gle/uWEzXFQ2viJtZtvZ7
--
⭐️ 📺 Learn GitLab on TH-cam ⭐️
Collection of video tutorials:
▸ Learn GitLab CI:
• Gitlab
___
⭐️ 📕 Free resources ⭐️
📌 Official Gitlab CI documentation
docs.gitlab.com/
📌 DevOps with Valentine
/ devops-with
--
⭐️ 📩 Contact & Imprint ⭐️
👉 Follow me on Twitter:
/ vdespa
👉 Follow me on Medium:
/ vdespa
👉 Follow me on Facebook:
/ valentin-despa-1130381...
👉 Let's connect on LinkedIn:
/ vdespa
vdespa.com/imprint
__
After wasting several hours trying to get the my pipeline working - was failing on the "dind" service - this video saved the day. Thanks a lot Valentin - love your presentation skills and technical insights 👍
Thanks, Tony. I'm glad this saved you a few hours ;)
Thank you, not only does it work but you also nicely explained it!
Glad to hear that. You are welcome. Thank you for leaving a comment.
Thank you so much man! The privilege error fix helped me a lot!
You are welcome. Thank you for leaving a comment.
that video is absolutely gold
Glad you liked it. You are welcome. Thank you for leaving a comment.
The best vídeo of all youtube to learn CI CD !!!
Glad that it helped. Thank you for leaving a comment.
Your Tips and Tricks have helped me so much. as a result of which I have now Powerful automation sitting in Jenkins . You are Fabulous. God Bless You
Great to hear! I am glad that it was helpful.
Thanks, It was very helpful.
You are welcome. Thank you for leaving a comment.
Really good tutorial.
You helped me a lot.
Thanks.
Glad it helped! You are welcome. Thank you for leaving a comment.
Very clear and concise tutorial.
Glad you think so! Thank you for leaving a comment.
Great tutorial, subscribed
Awesome, thank you!
Thank you valentin. I was stick in this place from past one week it made me happy. Also i liking your teaching style and showing us what happing in background with error this may helpful for every one when they follow this tutorial if they get any error like this. Error and pactice is make way for us to learn more. Please create this type of content that may help us to learn new things.
I suggest to you to create docker full tutorial for beginners to get into devops practice.
Thank you so much for sharing your knowledge to us.☺️
You are welcome. Thank you for leaving a comment.
Awesome video. Keep going.
You are welcome. Thank you for leaving a comment.
Thanks, It was very helpful. Still waiting for your new videos.
You are welcome. Thank you for leaving a comment.
You made my day bro......Thanks Buddy......
My pleasure! You are welcome. Thank you for leaving a comment.
Thank you Valentin!
You are welcome. Thank you for leaving a comment.
Great instructor
Glad you think so! Thank you so much.
Wow first to comment 🤪... Great start I think will be great tutorial
Thank you so much 😁
help me alot, I bought some of your curses and on the gitlab course I was looking for this information, thanks
Great to hear!
You are welcome. Thank you for leaving a comment.
@@vdespa one doubt, im doing it into a internal lab using the gitlab community for study. My pipelines is getting stuck initializing the git repository. We have to manually configure the git client on runner?
@@GabrielPyron Yes, the runner needs to have access to Git.Is the runner on Windows or Linux?
@@vdespa thanks for reply, I found the solution. The main problem is the installed version of gitlab thats conflicts with gitlab runner. So I updated the gitlab and it back to work as expected.
Thanks
Thankyou dude
No problem. You are welcome. Thank you for leaving a comment.
Thanks!
You are welcome. Thank you for leaving a comment.
Thanks bro
You are welcome. Thank you for leaving a comment.
Thanks, it is working for me. Can you also do the same thing for deployment on EKS after building the image?
thanks!!
You are welcome. Thank you for leaving a comment.
thanks a lot..
Welcome 😊
Thank you so much for this. What would be a production ready setup?
Mark, this largely depends on your organization needs. For personal projects, this setup is enough to get you started. However, for an organization, let the IT operations team handle this.
Valentine, thank you for the excellent tutorial; it has been incredibly informative. However, I'm encountering an issue while attempting to execute Docker in detached mode using the command 'docker run -d ' and utilizing Docker-in-Docker as a service with the Docker executor. Although the job appears to complete successfully within the pipeline logs, I'm unable to find the running container when using the 'docker ps' command on the GitLab-runner host. Is there a method to run containers in detached mode using the Docker executor with Docker-in-Docker?
Thanks Valentin for the great tutorial. I'm a bit confused when it comes to Docker in Docker image. I see when you registered the runner, you mentioned alpine image and when you ran the test job you mentioned alpine image as well, but in build job you mentioned "docker" image and in services section you used "docker:dind". I don't understand which image is used at which stage and is the docker container ran on the gitlab runner server (EC2 instance) or inside another container. Can you please explain which image is used at which stage and which image value overrides the others and where the container is ran.
Check the job logs, they will tell you which image is being used. docker:dind is a service that allows you to interact with the Docker daemon. Try removing it to see what happens.
Thanks for doing this. Would you mind doing DB migrations with Sql server , flyway, and GitLab CICD pipeline?
Thanks for your suggestion. I will look into it. Is there something in particular that does not work for you?
hi, and about security groups, we should give permission to all IPs ?
Please show the command in terminal docker images?
Hey Valentin, These Tutorials have helped me a lot in terms of build docker image and testing it(I've also set up my runner on my ec2 instance with a docker executor), I was now wondering how would I go about deploying my react web app(which is in an docker container) from my GitLab pipeline to an ec2 instance? would I have to ssh into said instance, pull container from GitLab registry and then run the docker build? If so I am not sure how to do this.
Thanks.
You can deploy a static web app directly in S3 (without Docker) or using ElasticBeanstalk with Docker. I don't think you want to use EC2 directly for this.
@@vdespa I see what if my web app had a backend? Would it still be advisable to do it on EB(Since s3 can only host static web pages)? I am just trying to understand what would be the pros or cons of using EC2 vs. the other software's. And also asked this questions since I was asked to directly deploy to EC2 instance
Thanks.
@@momopatel9054 EB is a managed service. If one of your container dies, EB will try to restart it. On EC2, you are the one who needs to manage everything, including software updates & co.
Hey valentine was wondering if you could do a video on how to deploy a MERN Stack App on AWS ec2 Using GItLab CI
Thank you, Valentin! This tutorial is extremely interesting and helpful for me, and my job.
I have a question, how come we installed “git”? I could not find any command using git.
How about cloning the project files using Git?
@@vdespa I get it. That’s right. I was just wondering if I can remove installing git if my team doesn’t need to clone during the CI. Thank you for answering my question!
Hey Valentin Thankyou very much for the simplified explanation...I was having one doubt ....What if I want to Pull Docker image from Azure Container Registry......Do I need to Specify Image path of container registry during Runner registration step and in the .gitlab.ci.yaml file
You need to specify that in your YAML file.
created a gitlab cicd pipeline to build the docker image and push it to container registry in gitlab and when I run the pipeline it shows following error. Copy failed file not found in build context or excluded by .dockerignore:stat pkg/demo.war:file does not exist. Can u help in getting through this error
Can your show something that one should do for production? You said in the beginning that one should not do this for production..
Good point, Bill. What "production" means largely differs from project to project. There is no single definition. However I understand the point you are making and I am considering more advanced tutorials that would be a better fit for most production situations where reliability and scalability are a must. Thanks!
@@vdespa that would be great. There are so many tutorials on stuff up to actual deployment, but then actually deploy something, perhaps a project for a small business (one might want to do some small freelancing for), how to deploy (not with heroku 😁), would be really good. 👍
Hello Sir,
I have connected my GitLab with remote rdc with specific runners (tag )installed at remote windows workstations.
not able to trigger scripts from .yml file on remote workstations.(windows 10)
(.yml file is successfully building pipeline by passing maven command for Windows)
Thanks
I am sorry this is not working. What is the error you are getting?
Can i do this with ubuntu and another cloud provider to configure gitlab runner ?
Yes. The process is the same, you just need the package for Ubuntu/Debian.
Hi Valentin,
Video is very informative and helpful. But I am facing one issue.
I have followed all the steps mentioned in the video and it worked for me, my pipeline started working, but once job is successful and I tried to ran another job, the runner went offline..
Can you help me how can make online again?
Did you try checking the status of the runner?
yes, I have checked the runner status and it was running.
@Valentin Now I am facing some weird issue, the CI/CD job is stuck in-between after running fine for 6-7 mins, and when I go to my EC2 instance that also stops responding. Any idea why this is happening?
I have checked the space on the instance, and it has enough space to run the job.
I got this error
I'm trying from a lxd container
when i change to privileged = true
ERROR: Job failed (system failure): Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply caps: operation not permitted: unknown (docker.go:403:1s)
How do i solve this error when i input"sudo rpm -i gitlab-runner_amd64.rpm", it shows that error: gitlab-runner_amd64.rpm: not an rpm package (or package manifest):
Are you sure that file is not empty?
Can somebody here please clarify what course of actions i have to perform to realize this what i read on someone's gitlab project i want to download, compile and run. It has just 3 short lines of text: "Getting started. The first time, run these commands to download afivo, particle_core etc.:
git submodule init
git submodule update". Then compile the code". First obvious question immediately comes: is just downloading and using ZIPped archive with the code available there not enough as usually ?
Just downloading the files usually is enough. However, since you need to run additional Git commands, you need to clone this repo with git.
@@vdespa Thanks Valentin. But why this was done? Cloning means that i will make one more gitlab place (my own clone of original one) with absolutely not clear reason why i have to do that as i do not need to keep there anything. The original place has sources Zipped, and i just can download them without bothering with useless cloning. Additionally the commands "git submodule init, git submodule update" etc are cryptic because obviously are incomplete (they show no address from which address to which address will be performed init or update. What i do not see here?
Getting this error
sh-5.2$ sudo amazon-linux-extras install -y docker
sudo: amazon-linux-extras: command not found
Can you suggest
Please increase terminal font size while recording
Thanks for raising this point. Are you watching on a mobile device?
@@vdespa yes i am watching on mobile, thanks for noticing my comment 🙂.
Increasing terminal font will help to see it clearly
@@highway9691 Got it. Thanks for letting me know.
I ran the command "sudo rpm -i gitlab-runner_amd64.rpm" then got the following error messages:
error: Failed dependencies:
gitlab-ci-multi-runner conflicts with gitlab-runner-15.0.0-1.x86_64
gitlab-ci-multi-runner conflicts with (installed) gitlab-runner-15.0.0-1.x86_64
Please help, I did not how to fix it. Thanks in advanced
It seems that it is already installed.
@@vdespa Thanks Valentin, do I need to create one EC2 instance for each project?
@@dongpham7789 No. This tutorial shows you how to create a group runner. In a group you can have multiple projects.
Where is push u docker image? Dockerhub?
Sorry for the late reply. You can push to AWS, Dockerhub or GitLab.
g8 video. Like + Sub here !
Thank you!
i set up an vm in GCP and do everything like you but when run pipline, it show "This job has not started yet
This job is in pending state and is waiting to be picked by a runner"
Did you configure a runner?
@@vdespa yes, i have registered my vm, set correct tags, give gitlab-runner user sudoer permission, install docker, check status of gitlab-runner and docker service but i still not work after i type "sudo reboot" in that vm - vm on GCP
@@QuangPham-bc7lc did you configure it to run jobs without tags?
yes
@@vdespa