We use Docker Official Images as the base images for several of our services. It's a popular choice, and there are good reasons for this. A dedicated team reviews them at Docker and gets actively maintained by experts from their respective communities, including the security community.

When we first started building these images, we struggled to integrate the process used to release fixes to the community. I can summarize our problems into two high-level buckets:

  • We were missing security and bug fixes. By missing, I mean that we weren't applying them quickly, or we weren't applying them at all.
  • Our builds were breaking mysteriously. Maybe all build breakages are mysterious, but this class of breakages was really bizarre!

It wasn't that patches weren't flowing in. The maintainers were doing a great job. It turned out that we needed a better approach for managing our Dockerfiles. We needed a Docker policy that could take advantage of the security and bug fixes that were reliably streaming in. And that was something we were missing entirely.

Background

To ground this discussion in something real, we'll use the official openjdk image as an ex. The lessons are applicable to any of the image repositories (debian, node, nginx, or redis for example).

Let's start with a refresher on how images are named and compare these two names:

  • openjdk@sha256:46285dcf00260440e00dcb68b5432cbed8f5fea734470be6ffb8ac7cc052024b
  • openjdk:11-jre-slim-buster

At the time of writing, these two refer to the same thing. At the time of reading, they most likely do not. The maintainers of images move the tags all the time. In a sense, that's what they do. Today, openjdk:11-jre-slim-buster might point at openjdk@sha256:46285dcf00260440e00dcb68b5432cbed8f5fea734470be6ffb8ac7cc052024b, but then a security fix is released, and it moves to point at a new digest.

It can be confusing to know when to use each type of name. My colleague has written an excellent post on this subject. The key takeaway is that image names with digests (e.g. those containing sha256:....) are great for builds, deployments, and any other process that should be repeatable. On the other hand, image names with tags are great when you're trying to keep up with changes. But how do we combine these two things? How do we achieve repeatability without losing the benefits of staying up to date?

When tags move

The maintainers of official images release fixes by updating (pushing) tags. We found it helpful to demystify this process for ourselves, so I'll quickly review how this actually works.

For the library of official images, all of the action occurs right out in the open. Anyone can follow along in the Docker Official Image library on GitHub.  Each official image maps to a library here in this directory.  The openjdk library, for example, is maintained here.

These library files provide the maintainers with a spec to define metadata such as:

Here's an example entry in the library file for openjdk:

Tags: 11.0.11-9-jre-slim-buster, 11.0.11-jre-slim-buster, 11.0-jre-slim-buster, 11-jre-slim-buster, 11.0.11-9-jre-slim, 11.0.11-jre-slim, 11.0-jre-slim, 11-jre-slim
Architectures: amd64, arm64v8
GitCommit: e1db2c7a51d718978eea6ee608e1dc687627fba4
Directory: 11/jre/slim-buster

This tells us where to look for the Dockerfile, which tags to push for a new build, and which platform architectures are supported by this manifest. Note that these tags all move together. You can think of them as aliases for one another (if you've ever wondered why there are so many tags that seem to point at the same thing, here's the reason!)

If we follow that GitCommit and Directory, we'll find the current Dockerfile for this particular set of openjdk images. The automation that watches these repositories, executes docker builds, and pushes the resulting tags, is a set of maintained Jenkins jobs. Here's the job for openjdk.

So that's the essence of how it all comes together. Fortunately for us, we don't need to interact with these systems at this level.

We just need to know when tags move because that's precisely when we fall behind.

In the picture above, we're watching the 11-jre-slim-buster tag. Something from either the base debian image, or the openjdk distribution has been updated. In this particular case, it was an openjdk patch, where a new version 11.0.11_9 was released. The maintainer pushed an update to the GitHub repo, an automation kicked off, and the tag got pushed. It now points at a new manifest.

So what?

Well, we want those fixes.

At the start of the blog, I talked about two problems — random build breakages and missing important fixes. These problems boil down to how we name things and how we keep track of external changes. We need to introduce a policy with the following characteristics:

  1. Watch all Dockerfiles in your repos. Raise pull requests whenever a FROMline references a tag. Replace the tag with a digest (read more on pinning).  This makes the builds of our checked in code reproducible.
  2. Continue to watch these tags. When they move, raise a pull request to update the digest. This ensures we don't fall behind.
  3. Watch for Dockerfiles that suddenly find themselves using tags that are no longer supported. Notify owners (we recommend failing a GitHub CheckRun)

These items work together to form a policy for keeping on top of critical fixes (especially security-related fixes). The first item is helpful, but it can't work alone. If you're going to pin something, you need to know when to move it. And for that, what could be better than a pull request?

Here's an example of the GitHub pull request that we raise:

Besides updating the FROM line, the pull request includes the changelog from the maintainers and a comparison between the current and the proposed base images. Our pull requests also have vulnerability data to give the developer additional context for evaluating the change. It's important to know that this change will also remove vulnerabilities. All of which gets presented as a pull request comment.

Nested Fixes

No official images can be derived from, or depend on, non-official images

This quote is pulled from the Docker Official Images README.md.  The quote actually goes on to add "with the notable exceptions of ....", but still, it raises an important point. Your base images might themselves have base images. For example, when I select an openjdk image tag like 11-jre-slim-buster, I am not just selecting a major revision of openjdk. I am also selecting a stable version of debian (as well specifying that I'd like the slim version with as few packages as possible).

This distribution of debian might be called "stable", but fixes are still flowing in.  It's a bit confusing that we are receiving our debian fixes by updating an openjdk image, but this is just how it is.  In the future, we may find ourselves working more with distroless images. For now, these nested fixes just further highlight the need to stay up to date.

Thank you

The maintainers of the official images are doing a great job. When we first started building this Docker policy, we were surprised by just how much activity is going on behind the scenes. And then, once we had the policy turned on, we see several updates per month.

It's an excellent feeling of confidence to know that we are building on top of solid, well-maintained images. We know that we aren't falling behind when important fixes roll out.

If you want to try out our new Docker policy, you can get early access and enable it on your repos by installing our GitHub app.  Please keep in mind that this is early access, and we're gathering feedback every day to make it better, and your comments and suggestions matter. Let us know what you think -- we're eager to hear from you.