Lesson 8 of 8 · 12 min · Releasing safely

Feature flags and rollbacks

Covers: Python 3 for the flag example; PostgreSQL 16 for the expand-and-contract migration. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Deploy versus release, and what a feature flag is
  • Kinds of flags: release, ops (kill switch), experiment, permission
  • A working percentage rollout with stable bucketing
  • Managing flags so they do not become debt
  • How to roll back: previous artifact, revert, GitOps, and database migrations

In simple words

Putting code on the servers and switching a feature on for users are two different things. A feature flag is a switch inside the code: the feature is installed but stays hidden until you turn it on, for everyone or for a few. And when a release goes wrong, you need a quick way back.

Words to know

  • Feature flag: an on/off switch for a feature that you can change without releasing again.
  • Kill switch: a flag used to turn a troublesome feature off quickly.
  • Rollback: going back to the previous working version.
  • Migration: a change to the database's structure. It is the hardest part to undo.

Separate deploying from releasing

Deploying puts new code on production servers. Releasing makes a feature available to users. Traditionally they were the same moment, which made every deployment a risk and encouraged big, rare releases. A feature flag (feature toggle) is a condition in the code that decides at run time whether a feature is on:

if flags.is_on("new-checkout", user):
    return new_checkout(cart)
return old_checkout(cart)

With flags you can merge unfinished work into main daily (trunk-based development) with the flag off, deploy it dark, turn it on for staff first, then for 1%, 10%, 100% of users, and turn it off in seconds if something goes wrong, with no deployment at all. This is the fastest rollback there is.

Kinds of flags

Kind Purpose Lifetime
Release flag hide an unfinished or risky feature; progressive rollout short: days to weeks; remove after full rollout
Ops flag / kill switch turn off a heavy or unreliable feature under load or during an incident (disable recommendations, switch to read-only mode) long-lived by design
Experiment flag A/B tests: which variant gives better results until the experiment ends
Permission flag features only for a plan, tenant or beta group long-lived (entitlements)

A working rollout implementation

The rules of a good percentage rollout: the decision must be stable per user (the same person does not flip between old and new on every request) and evenly distributed. Hash the flag name and the user id into a bucket from 0 to 99 and compare with the percentage:

import hashlib

FLAGS = {"new-checkout": {"enabled": True, "percent": 25, "allow": {"asha@example.com"}}}

def is_on(flag, user):
    cfg = FLAGS.get(flag)
    if not cfg or not cfg["enabled"]:
        return False
    if user in cfg["allow"]:
        return True
    bucket = int(hashlib.sha256(f"{flag}:{user}".encode()).hexdigest(), 16) % 100
    return bucket < cfg["percent"]

We ran it over 1000 synthetic users:

share on: 24.6 %                       (target 25%)
same user, 3 calls: [False, False, False]   (stable)
allow-listed: True                     (the team member always sees it)
after kill switch: 0                   (enabled = False turns it off for everyone)

Including the flag name in the hash means different flags select different 25% groups, so the same unlucky users are not the guinea pigs for every experiment. Raising percent from 25 to 50 keeps the original 25% in (bucket < 25 is a subset of bucket < 50), so users do not flip.

In production use a flag service rather than a dictionary in code: OpenFeature (a vendor-neutral standard API), LaunchDarkly, Unleash, Flagsmith, GrowthBook, AWS AppConfig. They provide a UI, targeting rules (by user, region, plan), audit logs, SDKs that cache flags locally, and instant updates. Evaluate flags locally with a cached ruleset so a flag-service outage cannot take your app down, and choose a safe default when the service is unreachable.

Flag hygiene

Flags are powerful and dangerous. Every flag adds a branch of behaviour, and combinations multiply (10 flags = 1024 combinations, of which you test very few).

  1. Give every flag an owner, a purpose and an expiry date. Release flags should be deleted soon after 100%: remove the flag and the old code path.
  2. Keep flag logic shallow: check it at one place (a router, a factory), not scattered through the code.
  3. Test both states, and the transitions.
  4. Default to off for new features, and design the off path to be safe.
  5. Don't use flags for permissions or configuration by accident; separate concerns.
  6. Audit changes: who flipped what and when. A flag change is a production change and belongs in the incident timeline.
  7. Keep an inventory and review stale flags monthly.

Rollbacks

Things will go wrong. The faster and more routine rollback is, the less scary releases are. Choose the fastest safe mechanism, in this order:

  1. Turn off the flag (seconds), if the problem is a flagged feature.
  2. Shift traffic back (blue-green switch, canary weight to 0%): seconds to minutes.
  3. Redeploy the previous known-good artifact. Because artifacts are immutable and tagged by commit, this means deploying shop-api:sha-8e5b2f1 again instead of sha-1a2b3c4. In Kubernetes: kubectl rollout undo deployment/shop-api; in Compose or plain servers: re-run the deploy job with the old tag. Make "deploy a specific version" a button in the pipeline (a workflow_dispatch input as in the GitHub Actions lesson).
  4. Revert the commit and let the pipeline ship the revert: git revert <sha> creates a new commit that undoes the change, safe on shared branches (Git lesson). It is slower than redeploying the old artifact but leaves the repository consistent with production.
  5. Roll forward: sometimes the quickest fix is a small corrective change shipped through the normal fast pipeline. Valid when the fix is obvious and the pipeline is quick, but do not guess under pressure; prefer rolling back to a known state first.

GitOps tools (Argo CD, Flux) make rollback a Git operation: revert the commit in the deployment repository and the cluster converges to it.

Rollback needs preparation

  • Keep the previous artifacts (retention policies must not delete the last good versions).
  • Keep configuration versioned with the code, so the old version runs with its old config.
  • Have health checks and alerts that detect the failure, and a clear threshold ("error rate above 2% for 5 minutes: roll back").
  • Practise it (game days). A rollback procedure that has never been run will fail when you need it.
  • Record every deployment and rollback with a timestamp for the incident record.

The hard part: databases and irreversible changes

Code is easy to roll back; data is not. If version 2 changes a column and writes new data, rolling back to version 1 can break it, and a destructive migration cannot be undone. Use expand and contract so every version works with the schema before and after each step. Here is a PostgreSQL walk-through (tested). Starting with customers(id, fullname):

-- 1. EXPAND: add the new column; old code ignores it
ALTER TABLE customers ADD COLUMN display_name text;
-- 2. BACKFILL, while new code writes both columns
UPDATE customers SET display_name = fullname WHERE display_name IS NULL;
-- the old version still works against the expanded schema:
SELECT fullname FROM customers WHERE id = 1;          -- returns Asha Rao
-- 3. release code that reads display_name (rollback to the old code is still safe)
-- 4. CONTRACT, in a LATER release, once nothing reads fullname:
ALTER TABLE customers DROP COLUMN fullname;

If you jump straight to step 4 and any old version is still running (or you must roll back), it fails:

ERROR:  column "fullname" does not exist

Rules of thumb: never drop or rename in the same release as the code change; make new columns nullable or with defaults; do large backfills in batches; keep migrations in version control, run by the pipeline, and take a backup or snapshot before risky ones; for truly irreversible operations (deleting data) require a separate, deliberate release. Also remember side effects that cannot be rolled back: emails sent, payments charged, messages published. Design them to be idempotent and reversible by compensating actions.

Putting it together: a safe release

  1. Merge small changes to main behind flags.
  2. The pipeline builds one tagged artifact, tests, scans and deploys to staging.
  3. Deploy to production by canary; automated analysis promotes or aborts.
  4. Roll out the feature by flag: staff, then 1%, 10%, 100%, watching metrics.
  5. If anything regresses: flip the flag off, or roll back the artifact, depending on the cause.
  6. Delete the flag and the old code.

Common mistakes

  • Flags that live forever and rot into untested code paths.
  • Flipping a flag in production without recording it.
  • Rollback plans that depend on a destructive migration.
  • Not keeping previous images/artifacts.
  • A flag service as a hard dependency with no safe default.
  • Per-request random decisions instead of stable bucketing.
  • Never rehearsing rollbacks.

Try it yourself

  1. Implement the rollout function above and measure the share and stability over 10,000 users. Raise the percentage and show that no existing user flips off.
  2. Deploy two tagged versions of a container, then roll back by redeploying the old tag; time it.
  3. Practise git revert of a merged change on a test repository and describe how the pipeline deploys it.
  4. Write an expand-and-contract plan for splitting a name column into first_name and last_name.

Check yourself

What is the difference between deploying and releasing?

Deploying puts the code on production. Releasing exposes the feature to users. Feature flags separate the two, so code can be deployed dark and released gradually.

Why hash the user id and flag name for percentage rollouts?

To make the decision stable for each user and evenly distributed, and to make different flags select different user groups.

Name four ways to roll back and their relative speed.

Flag off (seconds), traffic switch or canary to 0% (seconds to minutes), redeploy the previous immutable artifact (minutes), revert the commit and run the pipeline (longer, but keeps Git consistent).

Why is a database migration the hardest part of a rollback?

Data changes cannot simply be undone, and old code must still work with the new schema. Expand and contract keeps both versions compatible; destructive steps wait for a later release.

Interview questions

How do feature flags support trunk-based development and CI/CD?

They allow unfinished features to be merged and deployed in small increments with the flag off, decoupling deployment from release, enabling progressive rollout and instant disable without redeploying.

What is a kill switch?

An operational flag that turns off a feature or dependency at run time during an incident or overload, to protect the system without a deployment.

A release causes errors in production at night. Walk through your response.

Confirm via metrics, stop the rollout, use the fastest rollback available (flag off, traffic switch, or redeploy the previous artifact), verify recovery, communicate, then investigate, fix forward through the pipeline, and write a post-incident review with actions to prevent a repeat.

Summary

  • Feature flags separate deploying from releasing: progressive rollout, instant disable, trunk-based development.
  • Use stable hash bucketing; use a flag service with local evaluation; give flags owners and expiry dates and delete them.
  • Roll back by the fastest safe route: flag off, traffic switch, previous artifact, revert commit; practise it.
  • Make schema changes backward compatible with expand and contract so rollback is always possible.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on CI/CD pipelines and releases for DevOps Engineer and scores every answer, with what to fix.