Lesson 7 of 8 · 11 min · Releasing safely

Deployment strategies: rolling, blue-green and canary

Covers: nginx 1.24 with two Python backends (blue v1, green v2) on Ubuntu 24.04. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Why the way you release matters as much as what you release
  • Recreate, rolling, blue-green and canary strategies and their trade-offs
  • A tested zero-downtime blue-green switch, and a 90/10 canary split
  • What must be true of your application and database for any of this to work
  • How to choose, and how automation decides to promote or abort

In simple words

Replacing a running application is like changing a tyre on a moving car. You can swap everything at once and accept a short stop, replace it piece by piece, run old and new side by side and switch, or try the new one on a few users first. Each way trades risk against cost.

Words to know

  • Deployment: putting a new version of the software onto the servers.
  • Rolling update: replacing the old copies with new ones a few at a time.
  • Blue-green: running the old and new versions side by side, then switching all users at once.
  • Canary: giving the new version to a small share of users first, to see whether it is safe.

The goal: release without users noticing

A deployment can fail in three ways: it breaks (errors), it goes down (outage during the switch), or it is subtly worse (slower, wrong results). A good strategy limits both the blast radius (how many users a bad version touches) and the recovery time (how fast you go back).

1. Recreate

Stop the old version, start the new one. Simple, and there is downtime between the two. Acceptable for internal tools, batch jobs, or when two versions cannot run together. Nothing else here is cheaper.

2. Rolling update

Replace instances gradually: take one (or a few) out of the load balancer, update, health-check, put back, repeat. At any moment the old and new versions run side by side, and capacity never drops to zero. This is the default in Kubernetes Deployments (maxUnavailable, maxSurge), ECS services and auto-scaling groups.

  • Pros: no extra full environment, no downtime, simple.
  • Cons: two versions serve traffic at once (they must be compatible, see below); rollback means rolling back again, which takes as long as the rollout; a bad version reaches users as it spreads unless health checks stop it.

Needed: health or readiness checks that truly reflect ability to serve, and connection draining so in-flight requests finish before an instance stops.

3. Blue-green

Run two identical environments: blue (current production) and green (the new version). Deploy and test green while blue serves users. When green is ready, switch all traffic at once at the load balancer, DNS or ingress. Blue stays idle but intact, so rollback is just switching back.

We built it with nginx and two backends, v1 (blue) on port 9001 and v2 (green) on 9002. The upstream pointed at blue; while 200 requests were running, we changed one line to green, tested the config and reloaded:

upstream app { zone app 64k; server 127.0.0.1:9001; }     # blue
# change to :9002 (green), then:
#   nginx -t && nginx -s reload

Result during the switch under load:

status codes:   200 200          (every one of the 200 requests succeeded)
versions seen:  49 v1 (blue)   151 v2 (green)

No request failed and no connection was dropped: nginx started new workers with the new configuration and let the old workers finish their requests (reload is graceful). After the switch all requests returned v2 (green). Rollback was the same one-line change back: the next request returned v1 (blue).

In real platforms the switch is a load-balancer listener rule or target group swap (AWS ALB, CodeDeploy blue/green), a Kubernetes Service selector or Ingress change, a Route 53 weighted record (slower, because of DNS caching), or a tool such as Argo Rollouts.

  • Pros: instant switch and instant rollback; full pre-production testing on production infrastructure; no mixed versions at the user level.
  • Cons: needs double the capacity during the release (cost); stateful parts are hard (databases, sessions, caches, uploaded files must work for both); all users move at once, so a bug affects everyone until you switch back.

4. Canary

Send the new version a small slice of real traffic first, watch it, and increase gradually (1% → 10% → 50% → 100%). Named after the canary in the coal mine: a few users detect the problem before everyone is exposed.

Our nginx test used weights 9 and 1:

upstream app { zone app 64k; server 127.0.0.1:9001 weight=9; server 127.0.0.1:9002 weight=1; }

Of 100 requests, 90 went to v1 and 10 to v2, exactly the split. (The shared-memory zone is what makes the counters global across nginx's worker processes; without it a low-traffic test can look uneven, as seen in the load-balancing lesson.)

What makes a canary valuable is the analysis: compare the canary's error rate, latency and business metrics with the stable version over a time window, and promote only if it is not worse. Automated systems (Argo Rollouts, Flagger, Spinnaker/Kayenta, AWS CodeDeploy canary configurations) do this and abort automatically.

  • Pros: smallest blast radius; real production evidence before a full rollout; automated promote/abort.
  • Cons: needs good metrics and enough traffic to be statistically meaningful; slower; two versions at once; complexity of routing (by percentage, by header, by user group, by region).

Variants: dark launch / shadow traffic (copy production requests to the new version and discard its responses), ring deployments (employees → beta customers → everyone), A/B testing (compare behaviour of user groups for product decisions, not safety).

Comparison

Strategy Downtime Extra capacity Rollback speed Blast radius Complexity
Recreate yes none slow (redeploy) all users lowest
Rolling none small slow-medium (roll back) grows as it spreads low
Blue-green none 2x during switch instant all users after switch medium
Canary none small fast (route to 0%) a few users at first high

Many organisations combine them: rolling updates as the default, canary for risky services, blue-green for big switches.

What your application must support

Any strategy except recreate runs two versions at the same time, at least briefly. That requires:

  1. Backward-compatible database changes with the expand and contract pattern. Never drop or rename a column in the same release that stops using it. Step 1 (expand): add the new column/table, code writes both and reads the old. Step 2: migrate data. Step 3: code switches to the new. Step 4 (contract, a later release): remove the old column. Every release must work with the database schema before and after its migration, so that rolling back the code does not break.
  2. Compatible APIs and messages: old clients must work with the new server, and old and new servers must understand each other's messages and cache entries (version your APIs and event formats).
  3. Stateless servers: sessions and uploads in shared stores, not local memory or disk (otherwise blue-green's switch logs people out).
  4. Idempotent, retry-safe operations.
  5. Graceful start and shutdown: readiness checks, SIGTERM handling and draining.
  6. Observability: version-labelled metrics and logs so you can compare the versions (app_version label), plus alerts that trigger rollback.

Automating the decision

A pipeline that deploys safely typically does:

  1. Deploy the new version to a small portion (canary) or to green.
  2. Run smoke tests against it.
  3. Observe for a fixed window: compare error rate and latency (for example: 5xx rate not more than 1.5x baseline, p95 latency not more than 20% worse).
  4. If healthy, promote (increase weight or switch); if not, roll back automatically and notify.
  5. Keep the old version available until the new one has been healthy long enough.

Treat deployments themselves as measured: every release should record version, time, who/what triggered it and the result, so incidents can be correlated with changes ("what changed at 14:02?" is the first question in an outage).

Common mistakes

  • Incompatible database migrations, making rollback impossible.
  • Blue-green with sticky local state, so the switch logs users out or loses data.
  • Canary without metrics, so nobody can tell if it is working.
  • A canary too small to detect anything (1% of 100 requests an hour).
  • No automatic abort, relying on someone watching a dashboard.
  • Removing the old environment immediately, losing the rollback path.
  • Health checks that return 200 while the app cannot serve (so rolling updates continue into a failure).

Try it yourself

  1. Reproduce the nginx blue-green switch with two Python servers; run a curl loop through the switch and prove no request failed.
  2. Convert it to a canary with weights and verify the ratio over 100 requests.
  3. Describe an expand-and-contract plan for renaming a database column used by a live service.
  4. Write the metrics and thresholds you would use to decide whether to promote a canary of a web API.

Check yourself

What is the main advantage of blue-green over rolling?

An instant switch and instant rollback, with the new version fully tested in the production environment before it takes traffic, and no mixed versions on user requests. The cost is duplicate capacity.

How does a canary release reduce risk?

It exposes only a small share of real traffic to the new version first, compares its metrics with the stable version, and promotes or aborts based on evidence, limiting the number of users affected by a bad release.

Why must database changes be backward compatible?

During rolling, canary and blue-green releases (and rollbacks) old and new code run against the same database. Each version must work with the schema before and after the migration. Expand and contract achieves this.

What makes nginx -s reload suitable for a zero-downtime switch?

It validates the new configuration, starts new worker processes with it, and lets the old workers finish their in-flight requests before exiting, so no connections are dropped.

Interview questions

How would you release a risky change to a service handling 1,000 requests per second?

A canary with automated analysis: deploy the new version to a small share, compare error rate and latency to the baseline over a window, increase in steps, abort automatically on regression, and keep the old version ready. Make sure schema changes are backward compatible and use a feature flag for the risky behaviour.

What is the difference between blue-green and canary?

Blue-green switches all traffic between two full environments at once, with fast rollback. Canary shifts traffic gradually and judges the new version on real metrics at each step.

How does Kubernetes do rolling updates?

A Deployment creates a new ReplicaSet and scales it up while scaling the old one down, bounded by maxSurge and maxUnavailable, and only counts a pod as available when its readiness probe passes. kubectl rollout undo reverts to the previous ReplicaSet.

Summary

  • Choose a strategy to limit both blast radius and recovery time: recreate (downtime), rolling (gradual), blue-green (instant switch, instant rollback), canary (gradual with metric analysis).
  • The nginx tests showed a graceful blue-green switch with zero failed requests and a 90/10 canary split.
  • Running two versions at once needs backward-compatible schema and APIs, stateless servers, health checks and draining.
  • Automate promote and abort decisions on error rate, latency and business metrics; keep the old version until the new is proven.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.