Kubernetes Apply vs. Replace vs. Patch

Editor's note: This post was originally published in May 2020. Since then, Atomist has evolved and updated its platform and product offerings. For up-to-date information, check out the Atomist product page.

Kubernetes offers several ways to update resources: apply, edit, patch, and replace. Unfortunately, there seems to be some confusion about what each does and when to use them. When searching Google for ‘kubernetes apply vs replace’, the highlighted answer provided from Stack Overflow is wrong. When searching for ‘kubernetes apply vs patch’, the first entry of the results is the kubectl patch documentation, which does not include a comparison of apply and patch. This post explains the various approaches and when to use each.

Update: This post is now the first site returned by Google when searching for ‘kubernetes apply vs replace’, but the highlighted answer is still wrong. When searching for ‘kubernetes apply vs patch’, this post is the highlighted answer!

Sometimes during the lifecycle of a Kubernetes resource, e.g., a service, deployment, or ingress, specific properties of the resource need to be changed, added, or deleted. For example, an annotation needs to be added or the desired number of replicas of a deployment needs to be increased or decreased.

The Kubernetes CLI

If you interact with your Kubernetes clusters using the kubectl command-line interface (CLI), you are probably familiar with the apply and edit subcommands. The apply command reads resource specs from a file and “upserts”, i.e., inserts/creates a resource if it does not exist or updates it if it does, those resources in the Kubernetes cluster. The edit command reads the resource from the Kubernetes API, writes the resource specification, i.e., “spec”, into a local file, and opens that file up in a text editor for you. You can then edit and save the file and kubectl will send the changes you made back to the Kubernetes API, which takes care of effecting those changes upon the resource.

Likely less familiar are the patch and replace subcommands of kubectl. The patch command allows you to modify part of a resource spec, providing just the changed part on the command line. The replace command behaves kind of like a manual version of the edit command. You have to download the current version of the resource spec, e.g., using kubectl get -o yaml, edit it, and then use kubectl replace to update the resource using the modified spec. If any changes have occurred between reading and replacing the resource, the replace will fail.

The Kubernetes API

If you interact with your Kubernetes clusters using a Kubernetes client library for your programming language of choice, you are probably familiar with methods like CoreV1().Pods().Update(), replaceNamespacedService or patch_namespaced_deployment. Underlying these client library methods are HTTP requests using the PUT and PATCH HTTP request methods. The “update” and “replace” methods use PUT and the “patch” methods use, not surprisingly, PATCH.

It is worth noting that kubectl also interacts with your clusters using the Kubernetes API. In other words, kubectl is a command-line wrapper for the Go Kubernetes client library, while also providing a significant amount of sugar on top of the capabilities of the Kubernetes API. For example, you may have noticed no “apply” API methods were mentioned above. At present, all of the kubectl apply logic, i.e., the logic that creates non-existent resources and patches existing resources, lives entirely in the kubectl code base. There is an effort underway to move the apply logic into the Kubernetes API, but those efforts are still in beta. More on that below.

By default, patch

When you want to update a resource, it is best to default to patching it. This is true whether using the Kubernetes client libraries or kubectl.

Be strategic

Under the hood, the kubectl apply, edit, and patch commands all use the PATCH HTTP request method to update an existing resource. More specifically, they all use the strategic-merge patching approach when updating resources, although patch can use other approaches (more on that below). The strategic-merge approach attempts to “do the right thing” when combining the provided spec with the existing spec. More specifically, it attempts to merge both objects and arrays, meaning changes tend to be additive. For example, providing a patch that contains a single, new environment variable in a pod container spec results in that environment variable being added to the existing environment variables, not overwriting them. To delete a property with this approach, you need to specifically set its value to null in the provided spec. So among the updating kubectl subcommands, which is best to use?

If you create and manage your resources using kubectl apply, it is best to always use kubectl apply when updating so kubectl can manage the configuration and properly track the requested changes from apply to apply. The advantage of always using apply is that it keeps track of the previously applied spec, allowing it to know when spec properties and array elements are explicitly removed. This allows apply to remove properties and array elements when a normal strategic merge would not. The edit and patch commands do not update the annotation kubectl apply uses to track its changes, so any changes made by edit and patch, while tracked and made manifest by the Kubernetes API, are invisible to subsequent apply commands, meaning that apply will not remove them even if they do not appear in the apply input spec^[1].

If you are not using kubectl apply, you can use both edit and patch interchangeably, selecting whichever command suits the specific change you are trying to make. When adding and changing spec properties, both approaches are more or less equivalent. When removing spec properties or array elements, edit behaves like a one-time-use apply, i.e., it keeps track of what the spec was before and after you edit it so it can explicitly remove properties and array elements from the resource. Using patch, you have to explicitly set the value of a property to null in the patch spec to remove it from the resource. Removing an array element using a strategic merge patch is more complicated, requiring the use of merge directives. See the other patching approaches below for more reasonable alternatives.

To have the “patch” client library methods behave similarly to the above kubectl patching subcommands, you should set the content-type header in the request to application/strategic-merge-patch+json. Like with kubectl patch, if you want to remove spec properties, you will need to explicitly set their values to null. If you want to remove array elements, you will need to include merge directives in your patch spec or use a different patching approach.

The Kubernetes documentation states that edit and patch do update the annotation used by kubectl apply, but in practice that is not the case. ↩︎

Other patching approaches

Kubernetes supports two other patching approaches: JSON merge patch and JSON patch. Like the strategic-merge approach, the JSON merge patch approach accepts a partial Kubernetes spec as input and supports merging objects. The JSON merge patch approach differs from the strategic-merge approach in that it only supports replacing arrays. This includes the containers array in a pod spec. That means when using the JSON merge approach, you need to supply complete specs for all of the containers if you want to change any property of any container. As such, the JSON merge patch approach is useful if you want to remove elements from from an array in a spec. On the command line, you can select JSON merge patch using kubectl patch --type=merge. When using the Kubernetes API, JSON merge patch is used when accessing a Kubernetes resource endpoint using the PATCH request method and setting the content-type request header to application/merge-patch+json.

Rather than providing a partial resource spec, the JSON patch approach uses a JSON representation of the changes you want to make to a resource. A JSON patch is an array where each element of the array is a description of a change to make to the resource. JSON patch is a more flexible and powerful way to express the changes you want to make, but comes at the cost that you are no longer sending a partial resource spec, but a set of changes you want made in a distinct, non-Kubernetes-spec format. On the command line, you can select JSON patch using kubectl patch --type=json. When using the Kubernetes API, JSON patch is used when accessing a Kubernetes resource endpoint using the PATCH request method and setting the content-type request header to application/json-patch+json.

When you need to be certain, replace

There may be certain instances when you want to be sure that no changes have been made to a resource between the time you read the resource and when you are updating it. Put another way, you need to make sure all changes to a resource are atomic. This is the use case for updating a resource using replace. For example, if you have a ConfigMap with a counter that is updated by multiple sources, you may want to make sure that two sources do not attempt to update the counter simultaneously, causing you to “lose” an update. To illustrate, imagine this sequence of events using the patching approach.

A and B get the current state of the resource from the Kubernetes API.
Both A and B locally update the spec, incrementing the counter from n to n+1 and appending “A” and “B”, respectively, to the “updated-by” annotation.
A is a bit quicker and patches the resource.
B patches the resource.

The result is that the A update is lost. The last patch operation wins and the counter ends up only being incremented by one instead of two and the value of the “updated-by” annotation ends with “B” and has no “A”. Compare the above to what happens when the updates are done using the replace approach.

A and B get the current state of the resource from the Kubernetes API.
Both A and B locally update the spec, incrementing the counter from n to n+1 and appending “A” and “B”, respectively, to the “updated-by” annotation.
A is a bit quicker and replaces the resource.
B attempts to replace the resource but the update is rejected by the API because the resource version in the replacement spec does not match the current resource version in Kubernetes because the resource version was incremented when the A replace operation was made.

In the above case, source B will have to fetch the resource again, make its changes against the new state, and attempt to replace again. The result will be that the counter will have been incremented by two and the “updated-by” annotation will end with “AB”.

Implicit in the above example is that replacing a resource is just that: replacing the entire resource. The spec supplied in the replace request should be the fully formed resource spec, not a partial spec and not just the parts required when using kubectl apply. In particular, you must include the current resourceVersion in the spec metadata. If you do not include the resourceVersion or the version you provide is not the current one, the replacement will be rejected. As such, the best approach for using replace is to read the resource, update it, and replace it, immediately. Using kubectl, this might look like

$ kubectl get deployment my-deployment -o json \
    | jq '.spec.template.spec.containers[0].env[1].value = "new value"' \
    | kubectl replace -f -

That being said, you may have noticed that you can successfully run these two commands in sequence

$ kubectl create -f deployment.yaml
$ kubectl replace -f deployment.yaml

as long as the deployment.yaml does not specify the .metadata.resourceVersion property. That seems to directly contradict what I said above, i.e., “you must include the current resourceVersion in the spec metadata.” So is that statement wrong? No, it is not. What is happening under the hood is that kubectl notices you have not supplied the resourceVersion so it reads the resource from Kubernetes, adds the current resourceVersion to the spec you provided, and only then performs the replace request. Since that magic, which is dangerous if you are relying on the atomic nature of replace, resides entirely in kubectl, you cannot rely on it when using a Kubernetes API client library. In that case, you will have to GET the current resource spec, update it, and then issue the PUT request.

When you can't patch, forcibly replace

Sometimes you need to make changes to a resource that are not allowed by the Kubernetes API. In these instances, you can forcibly replace the resource, effectively deleting and re-creating the resource. This is done using kubectl replace --force on the command line. This forcibly and immediately deletes the resources and then re-creates it with the provided spec. There is no “force replace” API endpoint. To perform a similar operation using the Kubernetes API, you have to perform two separate operations. First, you must delete the resource, setting gracePeriodSeconds to zero (0) and propagationPolicy to “Background”, and then re-create the resource using the desired spec.

This approach can be disruptive and result in an inconsistent state!

Server-side apply

As mentioned above, the Kubernetes developers are working on implementing the apply logic in kubectl within the Kubernetes API. The beta 2 server-side apply logic is available in Kubernetes 1.18 via kubectl apply --server-side on the command line and on resource API endpoints using the PATCH request method with the content-type header value set to application/apply-patch+yaml.

Note: JSON is valid YAML, so you can send a JSON spec when content-type is application/apply-patch+yaml.

Aside from simply making kubectl apply logic available to all clients via the API, the server-side apply implementation tracks who is responsible for which fields in the spec, allowing multiple “field managers” to safely edit different portions of a spec without unknowingly conflicting. In other words, when server-side apply is more generally available, it will provide a mechanism for different clients, e.g., kubectl, Pulumi or Terraform, GitOps, and a script you wrote using a Kubernetes client library, to all manage the same resources safely.

Wrapping up(dates)

Hopefully this brief trip through the various ways to update resources in your Kubernetes clusters has been helpful. We have learned that the combatants are not just apply vs. replace, you can update a resource with apply, edit, patch, or replace. And really, it is not Kubernetes apply vs. replace or Kubernetes apply vs. patch, each approach has specific use cases in mind. If you need atomic updates, using kubectl replace or a PUT/replace/update API call is the way to go. Otherwise, using a strategic-merge patch via kubectl apply or a PATCH API call is usually the best choice. At the very least, I hope you have learned that you should not trust Google or Stack Overflow when searching for ‘kubernetes apply vs replace’. That is, until this post replaces the current answer. If you have any questions, ping me on Twitter or in the Atomist community Slack.

Regardless of how you are updating your Kubernetes resources, take the burden off your developers and operators by monitoring your cluster with the Kubernetes pod health monitor skill.