GitHub recently transitioned its support for container images from its original offering called Docker registry to a new one called GitHub Container registry. Besides the namespace change from docker.pkg.github.com to ghcr.io, it looks like a lot of the confusing aspects of the Docker registry have been re-worked into something that is a lot more intuitive.

We prefer to integrate with a new registry product using the Docker v2 APIs, so we decided to take a closer look at how GitHub Container registry supports these. There's already plenty of documentation on using https://ghcr.io with the docker cli, and with GitHub Actions. However, we had to hunt a bit more to figure out the current state of their v2 API support.

Authentication

Having built a fair number of Docker registry integrations recently, our team has learned that registry authentication is often surprisingly fiddly. Fortunately, GitHub was relatively simple, and predictable.

curl \
  -u username:<personal-access-token> \
  https://ghcr.io/token?service=ghcr.io&scope=repository:<repo>:pull&client_id=atomist

A response with status 200 will have an application/json body with a token key.

Reading a V2 Manifest

This mostly aligned with our expectations. You would expect the manifests to be queryable with:

curl \
    -H "Authorization: Bearer <token>" \
    -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    https://ghcr.io/v2/<org>/<repo-name>/manifests/<tag-or-digest>

And it works exactly how one would expect. It's great that the <repo-name> above does not have to be a git repository name. The Container registry namespace is now separate from the git repository namespace — you can push images into container repositories with names like org/x1 and org/x2 using code from a GitHub repo named something completely different (e.g. org/something-else). In my opinion, this is a really nice change and aligns much more effectively with how teams use these registry products.

On the other hand, we did not anticipate that the bearer <token> in the above expression was not going to support GitHub app installation tokens. The token that we created in the first step can only be created using a personal access token. Our GitHub app installations already have authorized read/write package access so it's disappointing to have to ask users to provide an additional personal access token. We second-guessed ourselves a lot here, but this does really seem to be the case.

Docker-Content-Digest headers

There's a surprising amount of entropy here between registry providers. GitHub again does what you expect here. When you read a manifest, the response will have a Docker-Content-Digest with the content digest SHA.

The Location headers on the https://ghcr.io/v2/<org>/<repo-name>/blobs/<digest> are also exactly how you'd expect. I apologize to those of you who don't know why I'm being so specific about this. API integrations can be a frustrating activity, and it's easy to lose a lot of time figuring out the small differences between providers.

/v2/_catalog

The catalog API is not implemented yet. We assume this is on its way, but for now, we've resorted to the v4 Graphql API to find available container repositories for an organization.

Searching Tags in a Repository

This one is back to aligning with expectations.

curl <https://ghcr.io/v2/><org>/<repo-name>/tags/list
    -H "Authorization: Bearer <token>"

So initial scanning and discovery still required some workarounds without a working catalog API, and the requirement to use a personal access token was certainly unexpected, but overall the experience of using the v2 API was good.

Webhook Support

Push notifications are an important part of our product. When users install our GitHub application, we can start scanning new container images as soon as they're pushed.

Webhooks fire as expected. However, there is one edge case. GitHub Container registry supports pushing container images that do not have a "Repository Source." In other words, users can push an image to a GitHub registry using some process that does not end with GitHub knowing anything about the source code used to build that image. When GitHub doesn't know anything about the repository source, webhooks do not fire. This makes some sense because GitHub app installations are scoped to git repositories. We wondered whether a GitHub app installation that is installed for "All Repositories" would receive these webhooks, but they do not.

GitHub will check the label metadata for each image that is pushed. If the label metadata references the "source" of the image, then the "Repository source" will be set, and webhooks will fire as expected.

org.opencontainers.image.source=https://github.com/<org>/<repo-name>

We already recommend that users build with both org.opencontainers.image.source and org.opencontainers.image.revision labels (among others), so this requirement is not a big deal. It's important to be able to go back to the git SHA that was used to build an image. Making this part of the label metadata, coupled with the fact that this can't be changed without altering the content digest, already makes this a compelling practice.

GitHub also automatically sets the repository source whenever an image is built using a GitHub action workflow. The git repository containing the workflow is assumed to be the repository source. This also makes perfect sense.

Overall, webhook events for GitHub App installations are working well for us. However, teams that are not using GitHub Actions, or not building with images with org.opencontainer.image.source labels, will probably think that they are missing events.

Conclusion

It was a good time to take another look at GitHub's Container registry support. There were a few surprises with the Docker v2 APIs, but the overall user experience was a big improvement. In fact, we're currently migrating our image scanning tutorial to default to the GitHub Container registry because we felt it was such a nice starting experience for users.