Comparing Docker base image layers to find vulnerabilities
It's a fantastic moment when the code that you're writing starts to do something useful.
The data in the backend has coalesced, and the web app has two shiny new filters. Select images by base image and show only application layer vulnerabilities. I tentatively reach out and select the JVM base image that all our Clojure API services use. Next, I carefully toggle (these are early days, you understand) to show just application layer vulnerabilities, removing the common noise from the base image that all these services share.
What do we see?
We see signal.
As these services share our JVM Docker image layer we would also expect them to have some fairly similar dependencies. As such we want to see them clustered together in terms of vulnerabilities, but we have a huge outlier. Why does org-service have almost four times more vulnerabilities than a well-behaved service like Steelix?
To find out, I click to see the detail view for the image digest and zoom into the critical vulnerabilities.
I think we have a suspect.
Org-service is using an old and vulnerable
9.2.x version of jetty-http. This is great! The feature is working! No. Wait. This is bad! jetty-http sounds an awful lot like something that would be internet-facing, and those vulnerabilities look both applicable and intimidating.
I dutifully create an issue against org-service with the very screenshots you see above. Tied somewhere between excitement that the idea came together and concern that we might be exposing known vulnerabilities to the world, I wait with bated breath.
Fortunately, at this point, our hero enters. Neil is a developer that cannot resist the chance to update a dependency. Do you see the cluster of 9 services with absolutely identical vulnerabilities and thus dependencies in the first image? It's no surprise that these are nine services that Neil regularly works on.
Mere seconds after I raise the bug, a comment arrives:
Neil is ever the professional, but really his thought process was more like, "I spent all that time making and rolling out a common HTTP library based on http-kit, and this service has the absolute audacity to use jetty?! Not on my watch, mate. Oh, wait, it doesn't. Something else is pulling it in..." Well, actually, the real thought process involved more of the symbols above the number keys, but even I'm too professional to include those.
A few seconds later:
The dependency is excluded. The image is rebuilt and deployed into staging and production. We rescan the image, and it returns to the middle of the vulnerability pack where it belongs. There are a few more services in our list that are showing signs of independence too. Not to fear. We'll soon have them crushed back into conformity.
Grouping images together via a common Docker base image layer is just one way of using Atomist to look at your vulnerability data. It is a very useful technique as it provides a large amount of signal very quickly. In the data we've explored, we found that similar services cluster around similar Docker base image layers — and if they don't, it's probably a sign that they should.
Docker base image layers often have more vulnerabilities than the application layer on top of them, so looking at total vulnerabilities can hide problems. While both base and application layer vulnerabilities need to be addressed, the way we do so (updating dependencies vs. updating or patching base images) is different, and it's helpful to look at them separately.
If you're interested in seeing how this looks in your code then head over to dso.atomist.com and sign up to get started with the beta. It's completely free to try and we have support for Docker Hub, Elastic Container Registry, Google Container Registry and Github Container Registry.