Secrets are a fact of life in modern software development. If you are developing with any type of API (Stripe, Twilio, etc.) or using modern tools like GitHub and Slack or running on AWS, Azure, Google Cloud or another cloud, you've got secrets. Specifically, you have the kind of secrets that give you and your code access to talk to other services. Whether they are API tokens, access keys or good old passwords, they are secrets. Okay, so we've established that everyone has secrets.

It is far too easy to accidentally leak secrets. When you're developing, it's easy to inadvertently put those secrets in places where they can be leaked. Sure, public repositories are an obvious place to avoid putting secrets. Still, sometimes it happens before you realize it. Private repositories may give you the illusion of secret security, but you still need to be careful with collaborator access and downstream artifacts that get published elsewhere. If a secret is embedded in code or config that makes its way into a package or image on a public registry, it's now public too. The best policy is the "no secrets in any repos" policy.

Last year, our app had two private key leaks that resulted in a $250k bill for fraudulent cloud usage. You bet we have Secret Scanner turned on.

—Dave, an Atomist user
Sequence diagram of secret scanning
Secret leak prevention requires a defense-in-depth solution

It should be just as easy to prevent leaking secrets as it is to leak them in the first place

Secret security nirvana is a state where someone or something intercepts anything that could possibly be a secret and removes it before it ever escapes the privacy of your editor, terminal or remote clone. That turns out to be difficult to do because you need a way to check for secrets at each step of the process.

  • 💻  Locally — Safety starts at home. Make it a practice to manage secrets through environment variables and vaults instead of in code or config. Run local secret scanning tools like truffleHog or git-secrets manually or add them as a pre-commit git hook to run on every git commit.
  • 🙌  On push — Ideally, your code gets scanned before it ever hits a repository where others can see it. This requires a pre-receive scan on the upstream remote side — e.g. GitHub, GitLab or other Git service you use.  GitHub provides a secret scanning service to catch common secrets before your push is accepted. Currently, the service only supports a fixed set of secrets— you cannot add your own. It runs on public repositories and is in beta for private repositories.
  • 📥  After push — If you're not using a Git service that has secret scanning or you have secrets that are not on your provider's list of scanned secrets (GitHub currently scans for 36 secret types), you also need to scan when code hits the repository, ideally as fast as possible. Bots are watching public commits for secrets, so you want to scan early and quickly so that you can remediate by invalidating the secret and removing it from repository history. The Secret Scanner skill in the Atomist catalog automatically scans for a list of popular secrets on every push and you can add custom secret patterns.

🗝  Custom Secrets: Whether you have your own private type of secrets that your app uses, or you use secrets that aren't scanned for by your provider, you'll need to be able to configure custom patterns to scan for. Given the fixed set of secrets supported by available scanners, virtually all of us will eventually need support for custom secrets. That's why we made it part of the Secret Scanner skill.

What to do when a secret has been leaked

Immediate revocation often makes sense. GitHub has created a program with several service providers to automatically report the secret to the service provider for revocation. Sometimes revoking a secret might need to be managed so that it does not disrupt your app or service. You may prefer to issue a new secret and configure your service to use it before revoking the leaked secret.

Regardless of the remediation path, you definitely want the right people to be notified at high priority as quickly as possible. Who should be notified and by which channel? We use Secret Scanner, which adds a failed GitHub Check to the commit when a secret is detected and sends Slack alerts to the committer and the appropriate chat channel.

Get safe 🔒

Keeping your secrets under wraps requires a defense-in-depth strategy.

  • Find a tool that works for your local development flow, whether IDE-integrated, using git-commit hooks or fastidiously running it from the command-line yourself
  • Use the secret scanning feature of your Git provider if it has such support
  • Enable Secret Scanner for custom secret scanning and post-push scans

If you have questions or suggestions, let me know in our Atomist community Slack or @jryanday.