Lesson 6 of 11 · 10 min · Running workloads well
Rolling updates and rollbacks
What you will learn
- How a rolling update works internally
maxSurge,maxUnavailableand the Recreate strategy- Watching, pausing, restarting and rolling back
- What a failed rollout looks like (it does not roll back by itself)
- What you must configure for a genuinely zero-downtime update
In simple words
To release a new version, Kubernetes doesn't stop everything and start again. It starts a few new Pods, waits until they are ready, removes a few old ones, and repeats, so users are served the whole time. If the new version is bad, one command takes you back.
Words to know
- Rolling update: replacing old Pods with new ones a few at a time.
- Rollout: one update in progress, from the old version to the new one.
- Rollback: returning to the previous version.
- Readiness: whether a Pod is ready to receive traffic yet.
What triggers a rollout
A Deployment rolls out when its Pod template changes: a new image tag, an environment variable, a resource limit, a label. Changing replicas does not (that is just scaling). Each template version gets its own ReplicaSet. During an update the Deployment scales the new ReplicaSet up and the old one down step by step, so the application keeps serving throughout.
kubectl set image deployment/web nginx=nginx:1.27-alpine # or edit the YAML and kubectl apply
kubectl rollout status deployment/web
kubectl get rs -l app=web
web-57b5dddfb4 0 0 0 <- old ReplicaSet kept (scaled to 0) for rollback
web-9896776d7 3 3 3 <- new ReplicaSet
The strategy
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
maxSurge: how many Pods above the desired count may exist during the update (extra capacity).maxUnavailable: how many Pods may be below the desired count (missing capacity).- Both are numbers or percentages; the defaults (25% each, as our cluster showed) mean with 4 replicas one extra and one fewer may exist during the rollout.
Typical choices:
| Settings | Behaviour |
|---|---|
maxSurge: 1, maxUnavailable: 0 |
never lose capacity: start one new Pod, wait until it is Ready, then stop one old one. Safest, needs room for one extra Pod |
maxSurge: 0, maxUnavailable: 1 |
no extra resources, one Pod fewer during the update |
maxSurge: 100% |
start all new Pods first (fast, like blue-green, needs double the resources) |
The other strategy is Recreate: stop all old Pods, then start the new ones. There is downtime, but it avoids running two versions at once (needed by some stateful or exclusive-lock applications).
We watched a rollout with maxSurge: 1, maxUnavailable: 0 (the line shows every Pod's status and readiness over a few seconds):
Running:1/1 Running:1/1 Running:1/1 ContainerCreating:0/1
Completed:0/1 Running:1/1 Completed:0/1 Running:1/1 Running:1/1 ContainerCreating:0/1
Running:1/1 Running:1/1 Running:1/1
At every moment three Pods were ready (Running 1/1); a fourth was being created, and old ones went to Completed (terminating) only after replacements were Ready.
Readiness decides everything
The Deployment treats a Pod as available only when its readiness probe passes. Without a readiness probe a container counts as ready the moment it starts, long before the app can serve, and the rollout proceeds into a failure. Define one (probes lesson). Add minReadySeconds to require a Pod to stay ready for some seconds before counting it.
When a rollout fails
Kubernetes does not roll back automatically. We updated to an image tag that does not exist:
kubectl set image deployment/web nginx=nginx:9.99-nonexistent
kubectl rollout status deployment/web --timeout=5s
After the progress deadline:
error: deployment "web" exceeded its progress deadline
State at that moment:
READY 3/3 UP-TO-DATE 1 AVAILABLE 3 <- the 3 OLD pods still serve traffic
1 pod 0/1 ErrImagePull <- the single new surge pod cannot start
Progressing=False ProgressDeadlineExceeded
Because maxUnavailable was 0 and the new Pod never became Ready, no old Pod was removed: users saw no impact. This is the safety of rolling updates: a bad version stalls instead of taking you down, provided readiness is correct. The Deployment reports Progressing=False after progressDeadlineSeconds (default 600 s; we set 25 s). A pipeline should run kubectl rollout status --timeout=... and, on failure, roll back and fail the job.
Rolling back
kubectl rollout history deployment/web
kubectl rollout undo deployment/web # to the previous revision
kubectl rollout undo deployment/web --to-revision=1
kubectl rollout status deployment/web
After the undo the image was back to nginx:1.27-alpine and the rollout completed successfully. Kubernetes keeps revisionHistoryLimit (default 10) old ReplicaSets for this. The CHANGE-CAUSE column of the history comes from the annotation kubernetes.io/change-cause, which you can set in your pipeline (kubectl annotate deployment/web kubernetes.io/change-cause="release 1.4.2") to make history readable. Useful extras:
kubectl rollout history deployment/web --revision=1 # shows that revision's pod template (image)
kubectl rollout pause deployment/web # make several changes without triggering rollouts
kubectl rollout resume deployment/web
kubectl rollout restart deployment/web # new Pods, same spec (for config changes)
In our test, with the Deployment paused, changing the image created no new Pods; after resume one rollout applied everything.
GitOps note: if Argo CD or Flux manages the Deployment, kubectl rollout undo is overwritten by the next sync from Git. Roll back by reverting the commit instead.
Zero-downtime checklist
A rolling update only avoids dropped requests if the application cooperates:
- Readiness probes that reflect real readiness (including connections to dependencies you need).
- Graceful shutdown: when a Pod is terminated Kubernetes sends SIGTERM, waits
terminationGracePeriodSeconds(default 30), then SIGKILL. The app must handle SIGTERM: stop accepting new requests, finish in-flight ones, exit. (Docker lesson: exec-formCMDso the signal reaches the app.) - Endpoint removal races: the Pod is removed from Service endpoints at the same time it is signalled, and some nodes learn of it slightly later. A short
preStopsleep (lifecycle.preStop.exec.command: ["sleep","5"]) keeps the Pod serving while endpoints update, avoiding errors during termination. - Enough replicas and
maxUnavailable: 0/small, and a PodDisruptionBudget for voluntary disruptions such as node drains. - Backward-compatible changes (database schema, APIs), since old and new versions run together (CI/CD course, release lesson).
- Resource headroom for surge Pods, or Pods will stay Pending (scheduling lesson).
For finer control (canary, blue-green with automated analysis) use Argo Rollouts, Flagger or a service mesh; the plain Deployment gives rolling updates only.
Common mistakes
- No readiness probe, so bad versions roll out fully.
- Assuming failed rollouts roll back.
- Ignoring SIGTERM (requests fail, or Pods are killed after 30 s).
maxUnavailabletoo high on a small deployment.- Changing config without restarting, then wondering why the update "did nothing".
- Using
kubectl rollout undounder GitOps and watching it be reverted. latesttags, wherekubectl rollout restartmay pull something unexpected.
Try it yourself
- Create a 3-replica Deployment, update its image and watch
kubectl get pods -wduring the rollout. - Update to a non-existent tag, observe the stalled rollout and intact old Pods, then undo.
- Set
maxSurge: 0, maxUnavailable: 1and compare the behaviour on the Pod list. - Add a readiness probe to a Deployment and a SIGTERM handler to its app, then verify no requests fail during an update with a
curlloop.
Check yourself
What do maxSurge and maxUnavailable control?
How many extra Pods may exist above the desired count, and how many may be missing below it, during a rolling update.
A rollout stalls at 1 new Pod in ImagePullBackOff. Are users affected?
Normally not: the old Pods keep serving because the new Pod is not Ready and maxUnavailable prevents removing old ones. You must fix or roll back; Kubernetes will not do it automatically.
How do you return to the previous version?
kubectl rollout undo deployment/<name> (or --to-revision), or revert the commit under GitOps.
Why does a Pod need a SIGTERM handler for zero-downtime updates?
Pods are stopped with SIGTERM followed by SIGKILL after a grace period. Handling SIGTERM lets the app finish in-flight requests and close cleanly instead of dropping them.
Interview questions
How do you ensure zero-downtime deployments on Kubernetes?
Rolling update with maxUnavailable: 0 or small, accurate readiness probes, graceful shutdown with a preStop delay, enough replicas and a PodDisruptionBudget, backward-compatible releases, and a pipeline that checks rollout status and rolls back on failure.
What is the difference between a rolling update and Recreate?
Rolling replaces Pods gradually with both versions running; Recreate stops all old Pods before starting new ones, causing downtime but never running two versions together.
How would you do a canary release with plain Kubernetes objects?
Run a second small Deployment of the new version with the same labels behind the same Service (traffic splits by replica count), or use an ingress controller or service mesh for weighted routing, or Argo Rollouts for automated canary analysis.
Summary
- Changing the Pod template creates a new ReplicaSet and shifts Pods gradually, controlled by
maxSurgeandmaxUnavailable. - A failed rollout stalls (old Pods keep serving) but is not rolled back automatically: check
rollout statusand userollout undo. - Readiness probes, graceful SIGTERM handling, preStop delay, enough replicas and compatible releases make updates zero-downtime.
- Under GitOps, roll back through Git.
Going deeper
- Performing a rolling update: the official tutorial.
- Deployment: updating, rolling back and strategy: all the details and fields.
- Pod lifecycle: termination of Pods: SIGTERM, grace period and preStop.
- Pod disruption budgets: protecting availability during voluntary disruptions.
See how much of this you can explain
A free mock interview asks you new questions on Kubernetes for DevOps Engineer and scores every answer, with what to fix.