Thanks a lot again !! Maybe you could do video about parallel builds, how to write them compact in declarative pipeline but still keep the "rules" , or generally explain how parallel works.. Thanks in advance, anyway you work is already very appreciated !!
Hello Szymon, Please do a video on multi build pipeline integration with nexus or artifactory. Just imagine there are 5 or six projects in one repository which is 4 of them are maven projects and 5of them are nodejs projects. User only needs to build the maven projects and upload the build artifacts in nexus or artifactory via one single code. I think this will be useful to most of the people.
Thanks for another great comment, Shan! Yeah, I'm aware that multi-project repositories are quite common for some teams. I can tell you that in most cases I try to keep a single project per repository when I work with the teams. Just for the sake of simplicity - you push a commit and the new artifact gets created. I was working with a few multi-project repositories and it was often problematic. Imagine having multiple Java microservices (Maven projects) in one repository. Even if you put separate Jenkinsfile per microservice, when you push a commit to that repository, every pipeline reacts because of the fact of the common repository. There are some workarounds, but I never found a perfect one. Anyway, I agree with you that this topic is important and I will do some research to help people who are working with such an environment. Thanks again, have a good day!
Hey Man.. Thanks might not be the word to express gratitude.. your videos are top-class and the way you explain the concepts are simply awesome.. Please keep up this good work.. Literally you are making changes in ppl life.. Small doubt. This jenkins file will create the docker container in the same host. If I need to create the docker agent in remote server. What is the syntax. where can I get those info>. One last request. Please put some video on how to refer the documentation when ppl stuck somewhere. As actual working env will be always challenging.. If you can guide us on how to refer the material to include the steps in the declarative pipeline.. It would be of great help..
Thanks for your kind words, Prathap. Regarding your question - the docker container gets created on the node that matches a specific label. In this example, I used the "docker" label, which is assigned to my Jenkins host for the sake of simplicity. However, in a production server scenario, you assign the label like "docker" to the remote hosts you use to run pipelines. Jenkins then starts the job and delegates building the docker image and running the container to one of the remote nodes matching the "docker" label. Also, I want to thank you for the video idea suggestion. I added it to my backlog to work on it one day. It might be more difficult than it sounds - each problem is different, and it highly depends on a specific context. Troubleshooting is a discipline based on personal experience, and the more hands-on experience we have, the more effectively we solve the problems. I created the "5 pipeline common mistakes" video some time ago (th-cam.com/video/aFRjn_4nb-Q/w-d-xo.html). It was an attempt to share some of the common pipeline mistakes I've seen throughout my career. I will think about your suggestion and see if there is anything valuable I could offer in such a video. Feel free to brain dump any follow up thoughts on that topic - it will help a lot. Thanks again, and have a good day!
Thanks for the amazing video !! Your tutorial delivery is awesome and top notch. Could you please make a video on Jenkins shared library and enforce the same pipeline by referencing to the shared library.
Thanks for your kind words, Raju! Yes, many people ask for the Jenkins Shared Library and I plan to create one in the upcoming weeks. Consider subscribing to the channel to get notification when it happens. Have a good day!
Thanks for your kind words, Milind! I can't promise anything - I haven't use Jenkins for more than a year, so creating any Jenkins related content these days will be more difficult. But we'll see :) Take care, and have a good day!
Hi Stepniak, Thank you very much for detailed explanation. Can you please create more projects on Jenkins pipeline integrating with Ansible and Docker and deploying the docker container to AWS ECS .
Hi Shitiz! Glad it was helpful! :) Is there anything else in the area of Jenkins Pipeline + Maven (or similar) that you would like to see on the channel in the future? Thanks in advance, have a good day!
I like the way you explain ! Hats off to you , Please make more videos. If possible help us with "Jenkins server to build a python project." Thank You !
Hi, Abishek! Thanks for the kind words. Regarding your request - I have no experience in building Python projects in Jenkins. If I gain some in the future, I will definitely share it here on TH-cam. Take care!
Hi Renato! I have zero experience with .NET Core, and I don't work on Windows machine, so that will be hard. I hope you will find a good video tutorial on other TH-cam channels that explains such a use case in depth. Take care, and have a good day!
Thanks for your kind words, Ankita! I will publish a dedicated video on using Docker with Jenkins soon, including Docker Compose scenario. Stay tuned, and have a good day!
I think "docker: not found" means that you have no docker client installed on the node you are using to run the dockerized pipeline. Make sure that the agent you use to run it have docker installed and ready to use. Good luck, and have a good day!
Hello and thank you for the tutorial, but I have an error when I launch the build here is the error in question: 'Jenkins' does not have label 'docker'
Hi Elisee! The example I shown in the video uses convention that there is a node with the label "docker". To overcome this error you can add label "docker" to e.g. your Jenkins master server (go to Manage Jenkins -> Computers & Nodes -> select "master" and add docker label. Good luck, have a good day!
Thank Szymon, A great content.. If i need to build a docker image(opnejdk..) with jar file that generated out of maven image,, and push the openjd to dockerhub, how do i do that.. Do i need to install Docker on the maven image to generate a build.. or any way that we have to bring the jar file to Jenkins host and build the docker file for open Jdk.. What is the best approach, please suggest
Hi there! Thanks for your kind words. Regarding your question, I think having everything in one image is the most convenient way to handle it. You could consider using Maven Wrapper (similar to Gradle Wrapper) so you don't need to install Maven inside this image (and re-use volume that stores the downloaded version of Maven.) Good luck, and have a nice day!
Hi Szymon, Thanks for sharing this informative session. Can you please clarify below points a) Are you using on-fly docker container just to build/compile the code ? b) So the artifacts will be copied back to mounted volume of the host , and will be used to deploy to DEV / QA / any containers for use ? The way I am setting-up my build environment is : 1) One virtual machine as a master (load balanced) 2) Then pre-configured containers as nodes . like one for java maven builds, one for dot-net core builds , and one for node.js .. this is to isolate the run times. Here the containers are pre-configures to the need of applications we have. 3) master will have all the build definitions which have a label to roue to container agents. This is where I am struggling whether my approach is efficient or not. Can you please provide your inputs around how you expect the Jenkins set-up to be in real world applications. Thank you again. Regards, Pradeep
Hi, Pradeep! Thanks for the insightful comment! Ad. a) It depends. I usually utilize docker containers for almost every part of the CI/CD pipeline, but it doesn't mean I use the same container for every stage. Ad. b) Yes, that is correct. Jenkins mounts your current workspace to the container and sets container's workspace to that specific folder. That way whatever get created during the pipeline run (test reports, artifacts, etc.) gets persisted in the workspace outside of the container. I don't see anything wrong with the Jenkins setup you described. There are a few different "schools" of setting up the environment, and the two I have seen most often are: a. non-specific nodes with docker installed, so you can do whatever you want with them, b. specific nodes (maven, nodejs, php-7, etc.) that provide specific environment setup (something similar to your example.) Both have their ups and downs, so it is always the question what works for you better. In the first case, you don't care much about what is the environment setup on the node apart from having docker installed. The pipeline runs the container it needs with the tools and settings it requires. It is more flexible, but it may also become problematic - not every container utilizes the same amount of resources. In the second case, you prepare predefined environments with specific version of Java, Maven, PHP, nodejs, and so on, and you just tell your users to run the pipeline on: agent { label "php-7" } if they want to build their PHP application etc. It moves the responsibility of the env setup from the team to the dedicated Jenkins admins/devops, so it may be more convenient for some people, and a bit limiting for others. I hope it helps, Pradeep. Thanks again for very interesting comment, have a good day!
@Szymon I have two questions: 1. What in case if we would like to use two docker containers? The First one for building, but the second one for a database to connect from the build container to make tests that require DB connection. And without publishing DB port number, because in case of multiple building at the same time this is not possible to publish the same port number. 2. How to publish a test report (e.g. Jacoco) to Jenkins from a docker building container? Should I map some directory or docker plugin that can support it?
Hi, Krzysztof! Thanks for the very interesting questions. ad.1) The simplest way that allows you to avoid docker-in-docker image is to use the so-called "sibling" container. This is a container that re-uses the host's docker.lock, so any container that is started inside this container is actually started on the host. If you run tests using e.g. Testcontainers, it should be OK, since it will support random ports. Take a look at this example: $ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -it maven:3.6.0-jdk-13 bash This is an example of a Maven-based container (from the official image) that has no docker client pre-installed - that is why we mount the client from the host. Then, you can run something like this inside this container: $ docker run -d -e POSTGRES_PASSWORD=test postgres:alpine and the postgres starts. I will create a video about this topic soon, but before that, you might also want to check this blog post that explains the difference between sibling containers and docker-in-docker one jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ In case of a Jenkins pipeline, you will need to add those two volume mount parameters to the container run arguments. ad.2) You don't need to mount any volumes. You can use jacoco/junit steps directly from the pipeline code. Here's a simple example of post step action that publishes test reports in one of my Maven projects: gist.github.com/wololock/c820673a15f251a2f3db426f79273619 I hope it helps :-)
@@szymonstepniak very well-done video! Thanks! In particular I was about to ask about a docker-in-docker situation but this answer clarifies that! I look forward to such an example :)
Thank you a lot for the great video, now a question about 'label' : does this require docker swarm to be installed? How could we label a node in docker? Does it require any addtional plugin to install in jenkins? Thank you
As far as I remember, you need to have Docker Plugin installed. The "label" option is not related to docker - it determines which Jenkins node to use to execute the pipeline stages. There is one convention that uses "docker" label on Jenkins nodes that have docker client/server installed. The idea behind it is that you don't want to run the whole pipeline on the Jenkins master node, but you want to delegate execution to a different node, and thus the label "docker" is used (by convention) to select the node (one from many) that has docker available. The label is configured on Jenkins master - go to "Manage Jenkins" -> "Manage Nodes and Clouds" and when you click on the config icon for a given node (if there are no slave nodes you will see only "master" node on that list) you will see a text field "Labels" where you can put coma-separated list of labels for that specific node. This is just a convention. In the same way you could e.g. add "windows" label to the slave nodes that are running on Windows machine to run some Windows-specific stages (something like e2e tests that have to be run on a Windows machine.) Hope it helps. Take care and have a good day!
Can I assume the docker agent Dockerfile approach by default finds the Dockerfile in the same source location as JenkinsFile in source repo? Cause this would not allow us to make a docker image of our code unless we can use another Dockerfile.
Yes, the default configuration expects the Dockerfile to be located in the same directory as the Jenkinsfile. But you can configure custom filename (or path) if you need one - www.jenkins.io/doc/book/pipeline/syntax/#agent-parameters Take care, and have a good day!
Hi Yogesh! I would suggest searching for articles (blog posts) solutions to this specific problem. I think it could be described on the official Jenkins blog already. Good luck, and have a good day!
@@ScrolltheNature I think what you are looking for is GDSL. This works with IntelliJ IDEA and allows you to get autocompletion based on your current Jenkins setup. Follow these instructions - gist.github.com/arehmandev/736daba40a3e1ef1fbe939c6674d7da8
Hey Symon, great video! Congratulations and thank you for the content, it helped me! I have one question: I have my Jenkins server running as a docker container. I tried to mount the volume as you said to prevent the download of dependencies every time my pipeline is executed but it didn't work. Do you know why I'm not able to do this? Do you have any recommendation to solve this problem?
Hey Marcus! I would check if the folder you mount exists in the Jenkins instance container, and if it is owned by the user that runs the Jenkins. Docker will create the folder if it does not exist, and it will assign it to the root user. In such case, the container that Jenkins starts won't have permissions to write to this folder. That's my first guess, it doesn't have to be the root cause. Shoot me an email with more details if needed, I will do my best to help you.
@@szymonstepniak Thank you for answering me! You're right, the problem was related to user permissions. Mounting the volume with the z option did the trick. Ref: github.com/carlossg/docker-maven/issues/63#issuecomment-359272506
Hi, Richard! Thanks for the comment. I have never did anything in .NET unfortunately. If I gain any experience in building pipelines for .NET in the future, I will definitely share it with the video tutorial. Have a good day!
Hi Szymon, amazing explain of how to use Jenkins and Docker!! I have a question: I am running your example (jenkins_02) but i have this message "[Pipeline] withDockerContainer Jenkins does not seem to be running inside a container " and later, in the same log it tells me "java.io.IOException: Failed to run image 'maven:3.6.0-jdk-13'. Error: docker: Error response from daemon: the working directory 'C:/WINDOWS/system32/config/systemprofile/AppData/Local/Jenkins/.jenkins/workspace/test/' is invalid, it needs to be an absolute path." Because I am trying to run your example in Windows 10 pro, exist some additional configuration in jenkins that I should revise?
Hi Christian! Thank you for your kind words! Regarding the issue you mentioned - I never run Jenkins (and Docker) on Windows, so this problem is new to me. I'm guessing that the problem with the working directory path might be caused by the fact that the host runs on Windows, while the Maven docker image is built on the Linux base system. I made a quick googling and I found those two resources that might help you solve the problem in one way or another: github.com/docker/cli/issues/2204 superuser.com/questions/1610886/how-to-solve-error-docker-error-response-from-daemon-the-working-directory-is Good luck, and have a good day!
@@szymonstepniak Hi Szymon thank's for your reply, I think I resolved it, I believe that the WSL 2 needs to work with a linux kernel, so, If I am working with jenkins on Windows I will have this kind problem, but if I install a linux distro on my Windows (Ubuntu for example) and I create a Jenkins container by my 'new linux terminal', I won't have this problem again. In summay, I will need 2 containers if I want execute docker from jenkins on windows, one for the CI tool , and other for the Maven project for example
Hey Szymon, great stuff, just one question please. I keep running into this error here Could not create local repository at /var/jenkins_home/.m2/repository Any Tips?
Hi NIkola! Thanks for your kind words. The error you mentioned is probably caused by missing permissions or something similar. I would suggest googling something like "docker cannot create /var/jenkins_home/.m2" - I'm guessing there is an answer to this problem posted on Stack Overflow already. Good luck and have a good day!
Hi, how would you create an Automated CI/CD Pipeline in Jenkins with Java 14 or later using Selenium Webdriver in a Maven Project with TestNG? am Not interested on how to setup Jenkins with Java 8 or 11!
Hi, I have an automation suit which runs on nodejs and now I want to setup Jenkins declarative pipeline for the same (with docker). I have docker file and docker yml part of my git repo. I have certain customized node variables like email and password to log into application which are set on my machine and test cases uses these variables and I want my Jenkins pipeline to have access to them as well. How could I do that, any suggestions ? Thanks.
Hi, Rohit! What have you tried so far? Did you run into some specific issue? I can at least try to help you solving specific issue, but I don't implement pipelines on request. Have a good day!
@@szymonstepniak thank you for reply. Well I wasn't asking to implement pipeline as I have already done that but I was missing some configuration which was causing an issue. Anyway I fixed it.
Hi @szymon thanks for the amazing tutorial, please help I am getting the below error while running the job, prnt.sc/wmnjso I trid manually by pulling images from jenkins user and it got successfully pulled, but from jenkins jobs it is giving error. thanks
Hi Ayush! It looks like your Jenkins is running on a user that has no access to the docker socket. Double-check if that user is a member of the "docker" group, and if not, add it to that group. This might help you - docs.docker.com/engine/install/linux-postinstall/ Good luck and have a good day!
Thank you for these explanations. However, when I tested the code which is in GitHub, I have the following error: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 2: Invalid agent type "dockerfile" specified. Must be one of [any, label, none] @ line 2, column 13. agent { dockerfile true } How to add the dockerfile agent? (Docker plugin already installed)
It looks like you are missing Docker Pipeline Plugin - plugins.jenkins.io/docker-workflow/ Go to "Manage Plugins" section and search for "docker pipeline" and install the plugin. Good luck!
@@szymonstepniak Thank you for your reply, and it solved the issue. However, I have a new error: [Pipeline] sh + docker build -t d6a1dad24c6fefb114c534a9d9d9b1ed6a2af1ec -f Dockerfile . /home/jenkins/workspace/Example2@tmp/durable-f1fb6e07/script.sh: line 1: docker: command not found It seems that the docker was not installed in the container. How did you manage to make it work?
![ตีสิบเดย์ [FULL] | ตัวตึง! ปากกล้า! ท้ารบ! "ทนายไพศาล เรืองฤทธิ์"](http://i.ytimg.com/vi/pU3dXTsaBVc/mqdefault.jpg)
💾 Source code from the video: github.com/wololock/jenkine-pipeline-maven-demo
⏳ Timestamps: 👇
00:00 Introduction
00:17 Demo Maven project introduction
01:12 Creating Jenkinsfile
02:44 Global Tools Configuration
04:23 Using "docker" agent type
07:02 Fixing Maven dependencies download cache
09:25 ssh: command not found
11:20 Switching to "dockerfile" agent type
16:12 Summary
master node labeled with docker ?!
Just saw video till 4 mins and could not help myself to pause and hit the like button
Thank you so much for your kind words, Ganesh! Take care and have a good day!
I like the way you explain! Great job, keep it up!
Thank you, Gordon! I'm happy to hear you liked it! Happy coding! 😉
Thanks a lot again !!
Maybe you could do video about parallel builds, how to write them compact in declarative pipeline but still keep the "rules" , or generally explain how parallel works..
Thanks in advance, anyway you work is already very appreciated !!
Thank you for this video! It helped me to define the Maven container to use my Jenkins repo with his libs in it. You save the day!
Thanks for your kind words, I'm happy to hear it helped you, Viktor! :) Take care, and have a good day!
Could you please make more videos on adding other tools like sonar cube , git difference etc to build a complete enterprise level pipeline
I can't promise anything, but we will see :) Take care, and have a good day!
Good works bro! The way you introduce the problem & the solution is clear! The calm music is great!
Thank you so much for your kind words! Take care, and have a good day!
Explained clearly.... Can you please make a video on Jenkins Pipeline NPM build (For Node.JS) as well
Thanks for suggesting the topic for the video, Siva! I think I will record something like that in the future. Thanks!
the way you explain the concept is great
please make a video for jenkins 'diclarative directive generator'
Thank you for your kind words, Venkat! I wrote down your suggestion in my backlog. Have a good day!
this video save my life. My respect to you
Thank you for your kind words! Take care, and have a good day!
It has been really helpful. Thank you!
Glad it was helpful!
Hello Szymon, Please do a video on multi build pipeline integration with nexus or artifactory. Just imagine there are 5 or six projects in one repository which is 4 of them are maven projects and 5of them are nodejs projects. User only needs to build the maven projects and upload the build artifacts in nexus or artifactory via one single code. I think this will be useful to most of the people.
Thanks for another great comment, Shan! Yeah, I'm aware that multi-project repositories are quite common for some teams. I can tell you that in most cases I try to keep a single project per repository when I work with the teams. Just for the sake of simplicity - you push a commit and the new artifact gets created. I was working with a few multi-project repositories and it was often problematic. Imagine having multiple Java microservices (Maven projects) in one repository. Even if you put separate Jenkinsfile per microservice, when you push a commit to that repository, every pipeline reacts because of the fact of the common repository. There are some workarounds, but I never found a perfect one. Anyway, I agree with you that this topic is important and I will do some research to help people who are working with such an environment. Thanks again, have a good day!
Hey Man.. Thanks might not be the word to express gratitude.. your videos are top-class and the way you explain the concepts are simply awesome.. Please keep up this good work.. Literally you are making changes in ppl life.. Small doubt. This jenkins file will create the docker container in the same host. If I need to create the docker agent in remote server. What is the syntax. where can I get those info>. One last request. Please put some video on how to refer the documentation when ppl stuck somewhere. As actual working env will be always challenging.. If you can guide us on how to refer the material to include the steps in the declarative pipeline.. It would be of great help..
Thanks for your kind words, Prathap. Regarding your question - the docker container gets created on the node that matches a specific label. In this example, I used the "docker" label, which is assigned to my Jenkins host for the sake of simplicity. However, in a production server scenario, you assign the label like "docker" to the remote hosts you use to run pipelines. Jenkins then starts the job and delegates building the docker image and running the container to one of the remote nodes matching the "docker" label.
Also, I want to thank you for the video idea suggestion. I added it to my backlog to work on it one day. It might be more difficult than it sounds - each problem is different, and it highly depends on a specific context. Troubleshooting is a discipline based on personal experience, and the more hands-on experience we have, the more effectively we solve the problems. I created the "5 pipeline common mistakes" video some time ago (th-cam.com/video/aFRjn_4nb-Q/w-d-xo.html). It was an attempt to share some of the common pipeline mistakes I've seen throughout my career. I will think about your suggestion and see if there is anything valuable I could offer in such a video. Feel free to brain dump any follow up thoughts on that topic - it will help a lot.
Thanks again, and have a good day!
Very nice, think I have learned the thing or two from this tutorial. Thanks!
Glad it was helpful! Take care, and have a good day!
Crystal clear explanation!!!!
Thank you!
The next one please about Gradle!
Thank you for a good job!
Thanks for the suggestion, Krzysztof! I wrote it down to my notebook with future video ideas. 👍
Excellent 👌 thank
Thank you!
I like the way you explain ! Great job
Thank you so much for your kind words, Yassine! Have a good day!
You're doing a great job. I really enjoy watching your videos and reading your blog. Thank you for all your efforts!
Thank you so much, Aston! You are very kind.
Thanks a lot!
You're welcome!
Thanks for the amazing video !!
Your tutorial delivery is awesome and top notch. Could you please make a video on Jenkins shared library and enforce the same pipeline by referencing to the shared library.
Thanks for your kind words, Raju! Yes, many people ask for the Jenkins Shared Library and I plan to create one in the upcoming weeks. Consider subscribing to the channel to get notification when it happens. Have a good day!
I really liked the video
Thank you! 🙏 I'm glad to hear you liked the video. Cheers! 👍
This is awesome! can you please create videos on shared library using declarative pipeline
Thanks for your kind words, Milind! I can't promise anything - I haven't use Jenkins for more than a year, so creating any Jenkins related content these days will be more difficult. But we'll see :) Take care, and have a good day!
Hi Stepniak, Thank you very much for detailed explanation. Can you please create more projects on Jenkins pipeline integrating with Ansible and Docker and deploying the docker container to AWS ECS .
Hi! Yes, there I will be working on dedicated Jenkins + AWS series soon. Have a good day!
A very nice detailed video, thanks a lot :)
Hi Shitiz! Glad it was helpful! :) Is there anything else in the area of Jenkins Pipeline + Maven (or similar) that you would like to see on the channel in the future? Thanks in advance, have a good day!
This video is very useful and very clear explanation. Thank you very much for the awesome tutorial. Keep it up my friend!
Thank you so much, Shan! I'm happy to hear you find it useful. Have a good day!
Many thanks for your videos!
Thank you so much for the kind words, Valentyn! I'm glad to hear you find my videos useful. Have a good day!
Dziękuje
I like the way you explain ! Hats off to you , Please make more videos. If possible help us with "Jenkins server to build a python project." Thank You !
Hi, Abishek! Thanks for the kind words. Regarding your request - I have no experience in building Python projects in Jenkins. If I gain some in the future, I will definitely share it here on TH-cam. Take care!
Thanks for the great video, I am definitely going for the dockerfile approach. Can you show how you would tackle a .NET Core project in Jenkins?
Hi Renato! I have zero experience with .NET Core, and I don't work on Windows machine, so that will be hard. I hope you will find a good video tutorial on other TH-cam channels that explains such a use case in depth. Take care, and have a good day!
Great video.
Can you please explain how to use docker-compose in the pipeline?
Thanks for your kind words, Ankita! I will publish a dedicated video on using Docker with Jenkins soon, including Docker Compose scenario. Stay tuned, and have a good day!
Hey! Thanks for the tutorial!
I am getting the ":docker: not found" exception when trying to use docker agent type. WHat can it be realted to?
I think "docker: not found" means that you have no docker client installed on the node you are using to run the dockerized pipeline. Make sure that the agent you use to run it have docker installed and ready to use. Good luck, and have a good day!
Hello and thank you for the tutorial, but I have an error when I launch the build here is the error in question: 'Jenkins' does not have label 'docker'
Hi Elisee! The example I shown in the video uses convention that there is a node with the label "docker". To overcome this error you can add label "docker" to e.g. your Jenkins master server (go to Manage Jenkins -> Computers & Nodes -> select "master" and add docker label. Good luck, have a good day!
Thank Szymon, A great content.. If i need to build a docker image(opnejdk..) with jar file that generated out of maven image,, and push the openjd to dockerhub, how do i do that.. Do i need to install Docker on the maven image to generate a build.. or any way that we have to bring the jar file to Jenkins host and build the docker file for open Jdk.. What is the best approach, please suggest
Hi there! Thanks for your kind words. Regarding your question, I think having everything in one image is the most convenient way to handle it. You could consider using Maven Wrapper (similar to Gradle Wrapper) so you don't need to install Maven inside this image (and re-use volume that stores the downloaded version of Maven.) Good luck, and have a nice day!
Hi Szymon,
Thanks for sharing this informative session. Can you please clarify below points
a) Are you using on-fly docker container just to build/compile the code ?
b) So the artifacts will be copied back to mounted volume of the host , and will be used to deploy to DEV / QA / any containers for use ?
The way I am setting-up my build environment is :
1) One virtual machine as a master (load balanced)
2) Then pre-configured containers as nodes . like one for java maven builds, one for dot-net core builds , and one for node.js .. this is to isolate the run times.
Here the containers are pre-configures to the need of applications we have.
3) master will have all the build definitions which have a label to roue to container agents.
This is where I am struggling whether my approach is efficient or not. Can you please provide your inputs around how you expect the Jenkins set-up to be in real world applications.
Thank you again.
Regards,
Pradeep
Hi, Pradeep! Thanks for the insightful comment!
Ad. a) It depends. I usually utilize docker containers for almost every part of the CI/CD pipeline, but it doesn't mean I use the same container for every stage.
Ad. b) Yes, that is correct. Jenkins mounts your current workspace to the container and sets container's workspace to that specific folder. That way whatever get created during the pipeline run (test reports, artifacts, etc.) gets persisted in the workspace outside of the container.
I don't see anything wrong with the Jenkins setup you described. There are a few different "schools" of setting up the environment, and the two I have seen most often are:
a. non-specific nodes with docker installed, so you can do whatever you want with them,
b. specific nodes (maven, nodejs, php-7, etc.) that provide specific environment setup (something similar to your example.)
Both have their ups and downs, so it is always the question what works for you better. In the first case, you don't care much about what is the environment setup on the node apart from having docker installed. The pipeline runs the container it needs with the tools and settings it requires. It is more flexible, but it may also become problematic - not every container utilizes the same amount of resources. In the second case, you prepare predefined environments with specific version of Java, Maven, PHP, nodejs, and so on, and you just tell your users to run the pipeline on:
agent { label "php-7" }
if they want to build their PHP application etc. It moves the responsibility of the env setup from the team to the dedicated Jenkins admins/devops, so it may be more convenient for some people, and a bit limiting for others.
I hope it helps, Pradeep. Thanks again for very interesting comment, have a good day!
@@szymonstepniak thank you much for your inputs. waiting for your next video on end to end pipeline to depict production ready process.
@Szymon I have two questions:
1. What in case if we would like to use two docker containers? The First one for building, but the second one for a database to connect from the build container to make tests that require DB connection. And without publishing DB port number, because in case of multiple building at the same time this is not possible to publish the same port number.
2. How to publish a test report (e.g. Jacoco) to Jenkins from a docker building container? Should I map some directory or docker plugin that can support it?
Hi, Krzysztof! Thanks for the very interesting questions.
ad.1) The simplest way that allows you to avoid docker-in-docker image is to use the so-called "sibling" container. This is a container that re-uses the host's docker.lock, so any container that is started inside this container is actually started on the host. If you run tests using e.g. Testcontainers, it should be OK, since it will support random ports.
Take a look at this example:
$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -it maven:3.6.0-jdk-13 bash
This is an example of a Maven-based container (from the official image) that has no docker client pre-installed - that is why we mount the client from the host. Then, you can run something like this inside this container:
$ docker run -d -e POSTGRES_PASSWORD=test postgres:alpine
and the postgres starts. I will create a video about this topic soon, but before that, you might also want to check this blog post that explains the difference between sibling containers and docker-in-docker one jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ In case of a Jenkins pipeline, you will need to add those two volume mount parameters to the container run arguments.
ad.2) You don't need to mount any volumes. You can use jacoco/junit steps directly from the pipeline code. Here's a simple example of post step action that publishes test reports in one of my Maven projects: gist.github.com/wololock/c820673a15f251a2f3db426f79273619
I hope it helps :-)
Hi @@szymonstepniak , many thanks for your answers! Appreciated :)
@@szymonstepniak very well-done video! Thanks! In particular I was about to ask about a docker-in-docker situation but this answer clarifies that! I look forward to such an example :)
Thank you a lot for the great video, now a question about 'label' : does this require docker swarm to be installed? How could we label a node in docker? Does it require any addtional plugin to install in jenkins?
Thank you
As far as I remember, you need to have Docker Plugin installed. The "label" option is not related to docker - it determines which Jenkins node to use to execute the pipeline stages. There is one convention that uses "docker" label on Jenkins nodes that have docker client/server installed. The idea behind it is that you don't want to run the whole pipeline on the Jenkins master node, but you want to delegate execution to a different node, and thus the label "docker" is used (by convention) to select the node (one from many) that has docker available. The label is configured on Jenkins master - go to "Manage Jenkins" -> "Manage Nodes and Clouds" and when you click on the config icon for a given node (if there are no slave nodes you will see only "master" node on that list) you will see a text field "Labels" where you can put coma-separated list of labels for that specific node. This is just a convention. In the same way you could e.g. add "windows" label to the slave nodes that are running on Windows machine to run some Windows-specific stages (something like e2e tests that have to be run on a Windows machine.) Hope it helps. Take care and have a good day!
Can I assume the docker agent Dockerfile approach by default finds the Dockerfile in the same source location as JenkinsFile in source repo? Cause this would not allow us to make a docker image of our code unless we can use another Dockerfile.
Yes, the default configuration expects the Dockerfile to be located in the same directory as the Jenkinsfile. But you can configure custom filename (or path) if you need one - www.jenkins.io/doc/book/pipeline/syntax/#agent-parameters Take care, and have a good day!
@@szymonstepniak Thanks for reply.
Hi Szymon, Its a great video, my requirement is to build and push the jar files as docker image and push it to docker hub, can you please guide me ?
Hi Yogesh! I would suggest searching for articles (blog posts) solutions to this specific problem. I think it could be described on the official Jenkins blog already. Good luck, and have a good day!
Great Video , but can you explain where to refer these DSL keywords so that it could be easily used accordingly ?
Can you explain what DSL keywords you have in mind?
@@szymonstepniak Thanks, i am new to jenkins pipeline, also shared library are being used here & all pipelines are coming from single repo e.g. keywords being used are env.HEALTH_CHECK_URL, globalMavenSettingsConfig , a sample stage of pipeline is below.
---
stage('App Build & Sonar') {
steps {
withMaven (
maven: 'MAVEN_3.3.9',
globalMavenSettingsConfig: 'DevopsGlobalMavenSettings'
) {
dir('module') {
script {
//sh "mvn build-helper:parse-version versions:set -DnewVersion=\\\${parsedVersion.majorVersion}.\\\${parsedVersion.minorVersion}.${BUILD_NUMBER} -DgenerateBackupPoms=false"
if (gitlabTargetBranch == 'dev') {
sh "mvn build-helper:parse-version versions:set -DnewVersion=\\\${parsedVersion.majorVersion}.\\\${parsedVersion.minorVersion}.${BUILD_NUMBER} -DgenerateBackupPoms=false"
env.sonarBuild = true
} else {
sh "mvn build-helper:parse-version versions:set -DnewVersion=\\\${parsedVersion.majorVersion}.\\\${parsedVersion.minorVersion}.\\\${parsedVersion.nextIncrementalVersion} -DgenerateBackupPoms=false"
env.sonarBuild = true
}
pom = readMavenPom file: 'pom.xml'
env.version = pom.version
env.groupid = pom.getGroupId()
env.artifactid = pom.getArtifactId()
sh 'mvn install -e'
//sh "mvn ${MVN_SONAR7_GOAL_REVIEW} -Dsonar.branch.name=${gitlabSourceBranch} -e"
dir('target') {
stash includes: '*.jar', name: 'jar-artifact'
}
dir('docker') {
stash includes: '**/*', name: 'dockerfiles'
}
}
}
}
}
}
@@ScrolltheNature I think what you are looking for is GDSL. This works with IntelliJ IDEA and allows you to get autocompletion based on your current Jenkins setup. Follow these instructions - gist.github.com/arehmandev/736daba40a3e1ef1fbe939c6674d7da8
Hey Symon, great video! Congratulations and thank you for the content, it helped me!
I have one question:
I have my Jenkins server running as a docker container. I tried to mount the volume as you said to prevent the download of dependencies every time my pipeline is executed but it didn't work. Do you know why I'm not able to do this? Do you have any recommendation to solve this problem?
Hey Marcus! I would check if the folder you mount exists in the Jenkins instance container, and if it is owned by the user that runs the Jenkins. Docker will create the folder if it does not exist, and it will assign it to the root user. In such case, the container that Jenkins starts won't have permissions to write to this folder. That's my first guess, it doesn't have to be the root cause. Shoot me an email with more details if needed, I will do my best to help you.
@@szymonstepniak Thank you for answering me! You're right, the problem was related to user permissions.
Mounting the volume with the z option did the trick. Ref: github.com/carlossg/docker-maven/issues/63#issuecomment-359272506
@@MrBondezan I'm happy to hear you solved the problem. Good luck, Marcus!
can you also make a similar video tutorial for dotnet core technology?
Hi, Richard! Thanks for the comment. I have never did anything in .NET unfortunately. If I gain any experience in building pipelines for .NET in the future, I will definitely share it with the video tutorial. Have a good day!
Are you able to do one for Kubernetes plugin?
If we are to run Jenkins in a kubernetes cluster, the above video context will need to be switched
Hi, Kyrogue! Thanks for the comment. Yes, "Jenkins & Kubernetes CI/CD Setup" video is in my backlog. Have a good day!
Hi Szymon, amazing explain of how to use Jenkins and Docker!!
I have a question: I am running your example (jenkins_02) but i have this message "[Pipeline] withDockerContainer Jenkins does not seem to be running inside a container
" and later, in the same log it tells me "java.io.IOException: Failed to run image 'maven:3.6.0-jdk-13'. Error: docker: Error response from daemon: the working directory 'C:/WINDOWS/system32/config/systemprofile/AppData/Local/Jenkins/.jenkins/workspace/test/' is invalid, it needs to be an absolute path."
Because I am trying to run your example in Windows 10 pro, exist some additional configuration in jenkins that I should revise?
Hi Christian! Thank you for your kind words! Regarding the issue you mentioned - I never run Jenkins (and Docker) on Windows, so this problem is new to me. I'm guessing that the problem with the working directory path might be caused by the fact that the host runs on Windows, while the Maven docker image is built on the Linux base system. I made a quick googling and I found those two resources that might help you solve the problem in one way or another:
github.com/docker/cli/issues/2204
superuser.com/questions/1610886/how-to-solve-error-docker-error-response-from-daemon-the-working-directory-is
Good luck, and have a good day!
@@szymonstepniak Hi Szymon thank's for your reply, I think I resolved it, I believe that the WSL 2 needs to work with a linux kernel, so, If I am working with jenkins on Windows I will have this kind problem, but if I install a linux distro on my Windows (Ubuntu for example) and I create a Jenkins container by my 'new linux terminal', I won't have this problem again. In summay, I will need 2 containers if I want execute docker from jenkins on windows, one for the CI tool , and other for the Maven project for example
I am getting error docker lable not found any solution for this ?
hey sir, how i can add specific settings.xml when i use docker agent ? thank you for help
Hey Szymon, great stuff, just one question please.
I keep running into this error here
Could not create local repository at /var/jenkins_home/.m2/repository
Any Tips?
Hi NIkola! Thanks for your kind words. The error you mentioned is probably caused by missing permissions or something similar. I would suggest googling something like "docker cannot create /var/jenkins_home/.m2" - I'm guessing there is an answer to this problem posted on Stack Overflow already. Good luck and have a good day!
@@szymonstepniak I think it was that in the end.
Thanks!
@@jaxparrow1794 How did you end up fixing this? Having the same problem
@@ImZenkas Sorry Mate. Can't remeber. It's been a while.
Good luck!
Hi, how would you create an Automated CI/CD Pipeline in Jenkins with Java 14 or later using Selenium Webdriver in a Maven Project with TestNG? am Not interested on how to setup Jenkins with Java 8 or 11!
Thank you for your comment. Have a good day!
0:42 What kind of template did you use to create Jenkins project? Is it freestyle or Maven or any other? Please help me
It was a regular "Pipeline" job type, no specific template. Have a good day!
ey buddy, i would like to ask you, what terminal's skin are you using ?
Hi Arnaldo! I use this ohmyz.sh/ Have a good day!
Hi,
I have an automation suit which runs on nodejs and now I want to setup Jenkins declarative pipeline for the same (with docker). I have docker file and docker yml part of my git repo.
I have certain customized node variables like email and password to log into application which are set on my machine and test cases uses these variables and I want my Jenkins pipeline to have access to them as well. How could I do that, any suggestions ?
Thanks.
Hi, Rohit! What have you tried so far? Did you run into some specific issue? I can at least try to help you solving specific issue, but I don't implement pipelines on request. Have a good day!
@@szymonstepniak thank you for reply. Well I wasn't asking to implement pipeline as I have already done that but I was missing some configuration which was causing an issue. Anyway I fixed it.
@@rohit_tyagi Thanks for elaborating on that. I misunderstood your question. Good to hear you already solved your issue. Have a good day!
nice video but please remove the background music, its little bit distracting.
Thanks for your kind words and suggestion, Lakshmi! Take care, and have a good day!
Hi @szymon thanks for the amazing tutorial,
please help I am getting the below error while running the job,
prnt.sc/wmnjso
I trid manually by pulling images from jenkins user and it got successfully pulled, but from jenkins jobs it is giving error.
thanks
Hi Ayush! It looks like your Jenkins is running on a user that has no access to the docker socket. Double-check if that user is a member of the "docker" group, and if not, add it to that group. This might help you - docs.docker.com/engine/install/linux-postinstall/ Good luck and have a good day!
Thank you for these explanations. However, when I tested the code which is in GitHub, I have the following error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 2: Invalid agent type "dockerfile" specified. Must be one of [any, label, none] @ line 2, column 13.
agent { dockerfile true }
How to add the dockerfile agent? (Docker plugin already installed)
It looks like you are missing Docker Pipeline Plugin - plugins.jenkins.io/docker-workflow/ Go to "Manage Plugins" section and search for "docker pipeline" and install the plugin. Good luck!
@@szymonstepniak Thank you for your reply, and it solved the issue. However, I have a new error:
[Pipeline] sh
+ docker build -t d6a1dad24c6fefb114c534a9d9d9b1ed6a2af1ec -f Dockerfile .
/home/jenkins/workspace/Example2@tmp/durable-f1fb6e07/script.sh: line 1: docker: command not found
It seems that the docker was not installed in the container. How did you manage to make it work?
Thank you!!!!
You're welcome! :-)