Persist Kubernetes resource changes to Git

Atomist makes deploying to Kubernetes easy in a way that is already largely consistent with GitOps best practices:

  • It is programmatic, not manual
  • It is based on a separation of concerns
  • It is not driven by CI and/or scripts

There was one major part of GitOps missing: having the desired state of your system versioned in Git.

Maintaining the state of your software system in Git is desirable for a few reasons. First, you maintain a historical record of what has been running in your cluster. Second, it allows you to more easily replicate your environment.

Maintaining a historical record of what is running in your cluster is important for troubleshooting and being able to restore a previously known good state. While Kubernetes deployments maintain a history of their activity, most other resources do not. Recording changes in Git allows you to roll back to a known good ingress configuration, pod security policy, secret value, etc.

In an ideal world, you won't have to replicate your environment, but things are not always ideal. If something goes wrong with your cluster or you need to spin up a separate environment to troubleshoot some issue, having all the Kubernetes resource specs sitting in a Git repository can be quite helpful.

Version 1.5 of the Kubernetes Software Delivery Machine (SDM) extension pack addresses this shortcoming.

SDM GitOps

The SDM GitOps flow runs both ways: the SDM can persist changes it makes to a Git repository and watch that Git repository for changes you and/or your CI/CD system make to it, applying those changes against the Kubernetes API endpoint for the cluster.

Persisting SDM changes

To enable persisting application deployments made by an SDM to a Git repository, make sure your SDM is using a recent version of @atomist/sdm-pack-k8s:

$ npm install --save @atomist/sdm-pack-k8s@latest

and then add the following section to your SDM client configuration:

export const configuration = {
   …
   sdm: {
       …
       k8s: {
           options: {
               sync: {
                   repo: {
                       name: "REPO_NAME",
                       owner: "REPO_OWNER"
                   }
               }
           }
       }
   }
};

replacing REPO_NAME with the GitHub.com repository name and REPO_OWNER with the GitHub.com repository owner, i.e., user or organization. That's it! Once you start an SDM with that configuration, it will start committing the Kubernetes resources specs it uses when deploying an application to the configured GitHub.com repository.

Example contents of a GitOps repo synchronized with an SDM

Applying your changes

If you already have a GitOps flow worked into your CI pipeline that updates Kubernetes resource specs in a Git repository, you can configure your SDM to listen for changes in that repository and apply changes to your cluster. This is accomplished using the standard SDM goal setting on a push to the configured GitOps repository. If you've configured repo synchronization as described above, there is nothing more to do to get this goal-setting capability. Reading and applying both YAML and JSON Kubernetes resource specs is supported.

Keep it secret, keep it safe

The worst kept secret in Kubernetes is that secrets aren't secret. Kubernetes secret data values are Base64 encoded, not exactly an unbreakable code. Storing sensitive information in Git repositories, whether public or private, is not a recommended practice, even if they are Base64 encoded. To allow storing of secrets in your GitOps repository while keeping sensitive information safe, the Kubernetes SDM extension pack supports encrypting secret data values prior to committing secret resource specs to the repository. The data values are then decrypted before applying them to the cluster.

To enable encryption of secret values, just add an encryption key to your SDM k8s configuration options.

export const configuration = {
   …
   sdm: {
       …
       k8s: {
           options: {
               sync: {
                   repo: {
                       name: "REPO_NAME",
                       owner: "REPO_OWNER"
                   },
                   secretKey: "ENCRYPTION_KEY"
               }
           }
       }
   }
};

When the SDM persists the secrets it creates to the GitOps repository, it will encrypt the data values using AES-256. The secret key can be any suitably long, suitably random string. When applying the changes you or your CI system makes to the GitOps repository, it will use that key to decrypt the data values before applying changes to secret resources. The Atomist CLI can be used to encrypt and decrypt the secret data values yourself:

$ atomist kube-encrypt|kube-decrypt --secret-key=KEY --literal=TEXT

replacing KEY with the same encryption key you provide in the SDM configuration and TEXT with the Base64-encoded secret data value you would like to encrypt or decrypt.

Separate separation of concerns

The SDM Kubernetes extension pack uses Kubernetes role-based access controls (RBAC) to securely access the Kubernetes API and perform its operations. While this is unchanged from previous versions of the extension pack, this new mode of operation may require additional permissions. Previously, the SDM only required read/write access to the resources it managed when deploying your applications, e.g., deployments, services, and ingresses. If you plan on having the SDM apply resources you or your CI system add to your GitOps repository, read/write access to those resources will need to be added to the role bound to the SDM's service account.

If you have a large organization that maps in a complicated way to your Kubernetes clusters and namespaces and are reluctant to give one pod access to too many resources, you can always run multiple SDMs, each having only the permissions it needs to perform its tasks. You can configure an SDM to only manage a single namespace or some subset of namespaces. You can use multiple GitOps repositories, or multiple branches in the same repository, with different SDMs watching each and only managing the resources defined in its repository/branch.

The flexibility of an event-based system connected to real, running code gives you everything you need to satisfy even the most complex delivery flows.

Why Kubernetes GitOps with Atomist?

Using Atomist to manage your GitOps flow in Kubernetes has several advantages:

  • There are no additional file formats or templating languages to learn: it is just plain-old Kubernetes resource specs
  • There is no lock-in: the data persisted to Git are fully-realized Kubernetes resource specs usable by kubectl or any Kubernetes API client
  • Two-way support of changes so your Git repository is always up to date: changes you commit to the Git repository will be applied by the SDM and the SDM will persist the changes it makes to the Git repository
  • It is secure: the Atomist approach exposes no endpoints, and you can even disable external access to the Kubernetes API endpoint if you want

Persistence pays off

If you've wanted to start using GitOps but weren't sure how, it is now easier than ever. If you already have an SDM managing your Kubernetes deployments, just add the sync configuration pointing to your GitOps repo. Otherwise, create your own SDM, add the sync configuration, and add a test deployment to your development Kubernetes cluster!