Automate your Docker Build/Test/Deploy pipelines!

This is my conference talk from Docker Community All Hands in March 2022. I show off different Docker build/push/run workflows (pipelines) and walk through GitHub Action examples
GitHub Repo: github.com/BretFisher/allhands22
Chapters
================
00:00 Intro
00:38 What is this about?
01:29 State of DevOps automation
02:43 Why GitHub Actions?
04:42 What should we automate with Docker?
05:31 Basic automation diagram
06:35 Linting PSA
07:34 Intermediate automation diagram
09:46 Advanced automation diagram
12:14 01 Basic Docker Build
15:00 02 Add BuildKit Cache
16:57 Add multi-platform builds
18:05 Add metadata to images
19:52 Add comments with image tags to PRs
20:50 Add CVE scanning
23:26 Add CVE security reporting
25:37 Add unit testing
26:47Add integration testing
28:59 Add Kubernetes smoke tests
31:51 Add job parallelizing for mucho speed
32:50 What's next?
Bret Fisher
=========
- Support this channel on Patreon! It's the #1 way to support this channel, my podcast, and open source / bretfisher
- Join the discussion on our Discord chat server devops.fan
- Get the best coupons for my Docker and Kubernetes courses at www.bretfisher.com/courses
- Listen to my podcast of this show's past episodes at www.bretfisher.com/podcast
#docker #ci/cd #automation

[@BretFisher]

I'll be live during this Premier, come join us and ask CI/CD questions!

[@aiiaznsk8er]

Really top-notch production! Great job dude.

[@BretFisher]

@@aiiaznsk8er 😍

[@ryancomia4500]

this is exactly what I was looking for a month now.. Awesome stuff!

[@pnddesign]

This is gold !! Thank you 👊

[@chrisjames278]

awesome video. Great information from you as always, thanks Bret.

[@SaitopLam]

THIS IS GREAT! Thank you!

[@ahmedalmulki]

Amazing videos as usual

[@marcelomg1985]

Thanks for the amazing information. Saludos!!!!😀

[@hoang-himself]

TIL the envsubst command

[@christianibiri]

Thank you!!!

[@amitabhyadav1562]

I really needed this, at least I have a better perspective. I'm new to this and am building a CI pipeline for a hardware code base (with really, really specific version constraints) all of which has to be run in docker. It's all over the place. :sigh:

[@amitabhyadav1562]

Any good linter suggestions for VHDL code?

[@xxdivinexx90]

Hey Bret! Great video! I was wondering about a few things;
- Shouldn't we be using image promotion rather than building the image twice, once in the testing pipeline and once after the PR? Jbaruch has a great talk about this. For the testing tools we can build another image that will use the image for promotion as a base image?
- You didn't touch application versioning, how would you advise to implement versioning?

[@BretFisher]

I've seen jfrog/Jbaruch talk on promotion, and I agree that their tool is *one* (expensive) way to fix the problem. To me, promotion isn't a feature, it's an idea around wayst to solve the Problem. The Problem being "how do I ensure the thing I tested is 100% the thing in production". I never recommended rebuilding an image from scratch. My examples (and also how I put it in my DockerCon talk from this week th-cam.com/video/Z0lpNSC1KbM/w-d-xo.html) show how you can use the previous build as a cache for your new build, and if you setup multi-stage Dockerfiles correctly, the "final" build after testing is just using the exact same layers as previously built, and only changing metadata before pushing to (any) registry.
Also, this is why we pin versions of *everything*. I don't want some "image promotion" feature in a proprietary registry to be the only way I know how to move images through the automation solution.
One thing he may not point out in those videos is if you pull and push a image to a different registry, it's a different sha, so there's no easy way to ensure an image in two registries is indeed the same one.
Lastly, you version your app any way you like. If you're talking about "app versions in your image tags", then yes, I've seen many types that I like. I recommend seeing all the ways you can tag (many tags on the same image build) with BuildKit, described in the Docker Metadata Action: github.com/marketplace/actions/docker-metadata-action#tags-input
Good luck!

[@xxdivinexx90]

@@BretFisher Hey Bret! Thanks for the reply, I totally agree about the promotion, I meant it not as a feature of a tool rather as a good practice.
But wouldn't it be more air-tight to simply change the image tag rather than rebuild with cache?
About versioning, are there any flows you recommend? For example if I'm using semantic versioning and bump2version to bump my versions, how would you recommend to manage the bumps. (Git branch naming or tags to trigger bumping patch, minor or major?)
Thanks again!

[@BretFisher]

@@xxdivinexx90 Yes, retagging is fine, but most teams I work with need additional items in their testable image for unit/integration testing that we don't want in production, so that's why my recommendations are to have the test-stage layer on *top* of the prod-stage, so then when you "rebuild" the production image, all your doing is making a new manifest.json of the same prod-stage layers (no layer rebuild).
On versioning, I've seen them all, and prefer the Docker Metadata design of having auto-labeling and auto-tagging in a CI step (in this case GitHub Action). It tags pr's with numbers, it tags branch names, it can tag on semver if you use git tags, etc. I've done auto date-based with it as well. All using "tag rules". I wish this functionality was built into docker automatically (you can do some of it with "docker bake" I believe github.com/marketplace/actions/docker-metadata-action#tags-input

[@BretFisher]

​@@xxdivinexx90 So examples:
- If app has no real versions, just use "keyword + date" for unique prod tag on each merge to main/master branch: bretfisher/example:stable-20220513123222
- I also tag that as :latest for developer/casual use convenience. Obviously never used in testing/pipelines/prod.
- If you use git tags to version in GitHub, then the Docker Metadata Action can put that tag in the image tag.
- If you have app versions, and also want to version the "deployment/image/chart", you could do bretfisher/example:v1.23.2-2.34.1 (and I believe this is still semver compliant)
- Lastly, semver still allows beta/alpha after numerical so if you merge to master and pass automated testing, but you still have more manual approvals before image is truly blessed, then the first tag after main commit could be v1.23.2-beta and then once manually blessed/approved the -beta is removed.

[@devopslp]

Any suggestions for Superlinter alternatives for other CI/CD Tools or Code Editor Plugins?

[@BretFisher]

mega-linter supports other CI's. VS Code has individual extensions for each language linter.

[@devopslp]

@@BretFisher oh, this is great. Support for CI/CD Tools in all of my projects. Lint everything ✊ - here I come

[@j.q.6208]

may you please share this yml file, you not have github url in definitions.

[@BretFisher]

It's all in the GitHub Repo listed in the Description: github.com/BretFisher/allhands22

[@sams1348]

I am a learner... What does PR stands for?

[@akashjss]

PR is pull request in github, similar to merge request in gitlab. It is a way of asking for a peer review before a change is done ✔

[@sams1348]

@@akashjss Thanks