Managing a GitOps Flow on Kubernetes
Using GitOps on Kubernetes means all your changes are reviewed and recorded, so you can move more quickly because everything is automated and you know all changes are easy to undo. Exposing your GitOps flow in chat means all the information you need is at your fingertips. How do you GitChatOps, or is it ChatGitOps? Here's how we do it.
Spoiler alert: we use Atomist.
In previous posts, we have explored getting started with GitOps, orchestrating GitOps, and ChatOps using various features of the Atomist Software Delivery Machine (SDM) framework and Atomist API for Software. In this post, we will present how we at Atomist use Atomist to orchestrate our own GitOps flow for delivering our back-end services, SDMs, client SDKs, and web sites.
One to many
We have used Kubernetes, even managing our back-end services with a simple GitOps flow, since the 1.1 release. We initially automated creation and provisioning of small sets of single-availability-zone Kubernetes clusters in AWS using custom command-line tools and services. As our use of and the tooling around Kubernetes matured, we migrated from federated multipurpose clusters, i.e., multiple clusters all running the same resources, that we managed on AWS to purpose-segregated multi-availability-zone clusters, i.e., different clusters for development, integration, production, customer resources, etc., on GKE.
As we migrated to segregated clusters, we found that we were managing identical core resources across these clusters. For example, we run external-dns to manage DNS records for our ingress resources in all our clusters. Using the Atomist CLI's
kube-fetch command, we were able to capture the resource specifications for these resources and add them to our GitOps repository for each cluster.
The release of the synchronization feature in our SDM Kubernetes pack allowed us to gain more insight and control compared to our initial GitOps implementation. After adding the Kubernetes synchronization configuration to each of our SDMs, all deployments managed by an SDM were persisted to the GitOps repository and any changes made to the repository, e.g., tweaking resource requests and limits via PR, were converged into the Kubernetes cluster by the SDM. Using the built-in encryption for secrets data values, we can reliably and safely maintain even our sensitive information in the GitOps repositories. We can view our builds & deployments and even approve production deployments in Slack using Atomist's ChatOps capabilities.
As mentioned above, we have multiple Kubernetes clusters. Some organizations prefer to segregate resources by purpose within a single cluster using namespaces. We found there was enough contention for core services like the cluster DNS that a single cluster approach proved problematic, especially when load testing in pre-production environments. Plus, we like the idea of testing changes to cluster configuration and core services in a non-production cluster before rolling it out in production. Perhaps you feel the same. For these reasons, we segregate resources between clusters for a variety of reasons. The two main drivers are "environment" and "users". We have different clusters for different environments, i.e., we have different clusters for testing and production resources. We also have different clusters for different user segments, i.e., we do not run customer resources in the same clusters we run our own resources.
Our Kubernetes SDM, k8s-sdm, is the workhorse of our GitOps flow. An instance of k8s-sdm is deployed in every one of our Kubernetes clusters. Since k8s-sdm is able to manage the entire GitOps flow, it is the only resource we have to provision when spinning up a new Kubernetes cluster. Therefore, we have integrated its provisioning into our Kubernetes cluster provisioning application. As we provision a new Kubernetes cluster and instance of k8s-sdm, we provide k8s-sdm its configuration via a Kubernetes secret. The k8s-sdm secret contains the necessary credentials to connect to the Atomist API and the synchronization repository configuration. We map each Kubernetes cluster to its own GitOps repository. Some organizations prefer using a single GitOps repository with a different branch for each cluster. This allows these organization to manage promotion from lower environments to higher productions via PRs. We prefer the more automated and flexible SDM goal approach for promotion and Drift Management to drive promotion of resources through environments.
Once the k8s-sdm instance is provisioned it begins synchronizing the resources in its GitOps repository to the cluster. It also subscribes to changes in its GitOps repository and Kubernetes deployment goals assigned to it. These subscriptions ensure the GitOps changes make it into the cluster. We also have it set to periodically resynchronize the resources in the GitOps repository with the Kubernetes clusters. This ensures any changes that fail for one reason or another eventually converge and ensure any manual changes to the cluster get reverted. If you want it in the cluster, go through the proper process!
Our k8s-sdm instances work in coordination with our other SDMs to deploy our services. The atomist-sdm manages the delivery for most of our open source repositories. The atomist-internal-sdm manages the delivery for our private repositories. Our documentation is built by docs-sdm. These other SDMs subscribe to pushes to the repositories they manage, analyze the repository & its changes, and schedule the appropriate delivery goals for each push. If the goal set includes a Kubernetes deployment, once the Docker build is completed, the managing SDM schedules a Kubernetes deployment goal with the appropriate k8s-sdm, i.e., the k8s-sdm running in the cluster where it should be deployed, selected to fulfill the deployment. The k8s-sdm in the cluster is subscribed to Kubernetes deployment goal events where it is selected for fulfillment so it receives that goal event and fulfills it, calling the appropriate Kubernetes APIs to create or patch the resources and then persisting the changes to the GitOps repository.
The above chat screenshot shows a goal set for a push to master to the aspect-sdm repository. The delivery of this repository is interesting in that the Docker image that is built is deployed in two different modes to each of the deployment targets, staging and production. All of the goals for this push are scheduled by the atomist-sdm. The atomist-sdm executes all of the goals up through the Docker build itself. It then schedules the two staging deployments but they are executed by the k8s-sdm instance running in the staging Kubernetes cluster. Once the staging deployment has been approved, the atomist-sdm schedules the production deployments and this time they are executed by the k8s-sdm instance running in the production Kubernetes cluster. Concurrent with the production deployments, the atomist-sdm executes the other goals associated with a production release.
Atomist + GitOps = Rest easy
With Atomist managing your GitOps flow on Kubernetes, you can put away the pagers and rest easy. Kubernetes will keep your stuff running and Atomist will keep it up to date. If anything happens to your cluster, you can just spin up another one, deploy k8s-sdm, and be back in business!