Lesson 5 of 8 · 10 min · State and teamwork

Remote state, locking and environments

Covers: Terraform 1.17, S3 backend with native lockfile. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • Why teams need a remote backend, and what a good one provides
  • Configuring the S3 backend with state locking
  • What the lock looks like when two people collide
  • State versioning and recovery
  • Workspaces and directory-per-environment layouts
  • Bootstrapping the backend itself

In simple words

On your own laptop the state notebook is one local file. In a team, everyone needs the same notebook, and two people must never write in it at the same moment. So it is kept in a shared, protected place, with a lock that lets only one person change it at a time.

Words to know

  • Remote backend: a shared place where the state is stored, such as an S3 bucket.
  • State locking: a guard that stops two runs from changing the state at the same time.
  • Workspace: a separate state for the same code, used for separate environments.
  • Environment: one full copy of your system: dev, staging or production.

What a backend gives you

A backend decides where state is stored and how operations run. A good remote backend provides:

Need Why
Shared storage CI and every engineer see the same state
Locking two applies at once cannot corrupt state or double-create resources
Encryption at rest and TLS state holds secrets
Versioning recover from a bad apply or a corrupted file
Access control only the pipeline (and few people) can read or write production state

Options: Amazon S3 (the most common on AWS), Google Cloud Storage, Azure Blob Storage, HCP Terraform / Terraform Enterprise (HashiCorp's managed service, with remote runs, policy and UI), GitLab-managed state, Consul, Postgres. The concepts are the same.

The S3 backend, tested

terraform {
  required_version = ">= 1.10"
  backend "s3" {
    bucket       = "tf-state-demo"
    key          = "shop/dev/terraform.tfstate"
    region       = "ap-south-1"
    use_lockfile = true
    encrypt      = true            # server-side encryption (add kms_key_id for your own key)
  }
}

(To follow along with a local AWS emulator, add dummy credentials and an endpoint override; on real AWS you omit those and authenticate through the environment, a profile or an assumed role.) terraform init configures it:

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

After an apply there was no local state file: the directory contained only main.tf. The state lives in the bucket at shop/dev/terraform.tfstate, and every command (plan, state list, output) reads it from there:

null_resource.slow
random_pet.name
"select-iguana"

The key is the path inside the bucket: give each stack its own (shop/prod/network/terraform.tfstate). Backend blocks cannot use variables; pass the settings at init time for flexibility: terraform init -backend-config=prod.s3.tfbackend.

Locking

Since Terraform 1.10 the S3 backend supports native locking with use_lockfile = true: while an operation runs, Terraform writes a small terraform.tfstate.tflock object next to the state, and removes it afterwards. Older setups used a DynamoDB table (dynamodb_table = "tf-locks") for the same purpose; both work, the lockfile removes the extra table to manage. We watched it happen. While one apply was running (a 25-second provisioner), the bucket listing showed:

2026-11-12 14:41:54        226 terraform.tfstate.tflock

and a second terraform plan from another machine failed immediately:

Error: Error acquiring the state lock
Lock Info:
  ID:        98c19655-62c4-7293-183c-ed0a07cd9499
  Operation: OperationTypeApply
  Who:       root@0f1979b24b80

The error tells you who holds the lock, since when, and what operation. Wait for it to finish. If a run was killed (CI runner died) and left a stale lock, confirm nobody is applying and then terraform force-unlock <ID>. Do not use -lock=false on shared state.

Versioning and recovery

Enable bucket versioning: every state write creates a new version. After our two writes, list-object-versions showed 2 versions of the state object. If a bad apply or an accidental state rm corrupts the state, restore the previous version of the object from S3. Add lifecycle rules to expire old versions after a while, and block public access to the bucket.

Bootstrapping

The backend bucket must exist before Terraform can use it, a chicken-and-egg problem. Common solutions: create it once with a small separate Terraform configuration using local state (then keep or migrate that state), or by hand/CLI/CloudFormation, with versioning, encryption, public-access block and a restrictive policy; keep this account-level foundation separate and rarely changed. Use a dedicated bucket per account or environment, and restrict who may write production state.

Migrating state

Changing the backend block and running terraform init offers to copy existing state to the new backend (-migrate-state); answer carefully and verify with terraform state list before deleting the old copy.

Environments: workspaces or directories?

You need the same configuration in several environments (dev, staging, prod) with separate state.

Workspaces let one configuration keep multiple state files:

terraform workspace list
terraform workspace new staging
terraform workspace select staging

With the S3 backend each workspace's state is stored under env:/<workspace>/<key>, which we saw in the bucket:

env:/staging/shop/dev/terraform.tfstate
shop/dev/terraform.tfstate

Use terraform.workspace in code to vary names. Workspaces are quick, but have downsides for serious environments: it is easy to forget which workspace you are in and run against production; all environments share the same backend permissions; and the code is identical, so differences are awkward. HashiCorp itself recommends separate configurations for strongly isolated environments. (HCP Terraform workspaces are a different, stronger concept: one per environment with its own variables and permissions.)

Directory-per-environment (or per-account) layouts are explicit and safest for production:

infra/
  modules/            # reusable building blocks
    network/
    app/
  envs/
    dev/      main.tf  backend.tf  terraform.tfvars
    staging/  main.tf  backend.tf  terraform.tfvars
    prod/     main.tf  backend.tf  terraform.tfvars

Each environment has its own backend key (or bucket and account), its own variables and its own pipeline permissions, and calls shared modules. A change is promoted by applying it to dev, then staging, then prod. Tools like Terragrunt reduce the repetition. Pick one structure per organisation and document it.

Reading another stack's outputs

Separate states need a way to share values (the VPC id from the network stack to the app stack):

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "tf-state-demo"
    key    = "shop/dev/network/terraform.tfstate"
    region = "ap-south-1"
  }
}

# data.terraform_remote_state.network.outputs.vpc_id

The reading stack needs read access to that state (which contains secrets): often better to publish selected values to SSM Parameter Store or look the resource up by tag with a data source, keeping state private.

Common mistakes

  • Local state for shared environments.
  • No locking (or disabling it) on shared state.
  • No bucket versioning, with no way back after corruption.
  • Public or broadly readable state buckets.
  • Forgetting the current workspace and applying to the wrong environment.
  • Everything in one state and one backend with one set of permissions.
  • Force-unlocking while an apply is really running.

Try it yourself

  1. Create a bucket with versioning and encryption, configure the S3 backend with use_lockfile, and run terraform init. Confirm no local state file appears.
  2. Start a long apply and run plan from a second terminal to see the lock error.
  3. List object versions of the state in the bucket, and describe how you would restore one.
  4. Create a second workspace and see where its state is stored; then try the directory-per-environment layout and compare.

Check yourself

Why should shared infrastructure use a remote backend with locking?

So everyone and CI use one authoritative state, concurrent applies cannot corrupt it, and it can be encrypted, versioned and access-controlled.

What does state locking protect against?

Two Terraform operations modifying the same state at the same time, which could corrupt state or create duplicate or conflicting resources.

What do you do when a lock is left behind after a crash?

Verify nobody is running Terraform, then run terraform force-unlock <lock-id>. Do not disable locking as a habit.

Why are separate directories or accounts often better than workspaces for production?

They make the target environment explicit, allow separate credentials, permissions and backends, and reduce the chance of applying to the wrong environment.

Interview questions

How do you set up Terraform for a team of ten on AWS?

S3 backend with versioning, encryption and a restrictive bucket policy, native locking (or DynamoDB), state split per environment and layer, CI that plans on pull requests and applies from the main branch with assumed IAM roles via OIDC, and no one holding long-lived admin keys.

How do two stacks share values?

Outputs read via terraform_remote_state, or values published to a parameter store or looked up with data sources. Prefer data sources or parameters to avoid exposing the whole state.

What can you do if the state file is corrupted?

Restore the previous version from bucket versioning, or from a backup of terraform state pull; as a last resort, rebuild state with import blocks.

Summary

  • Remote backends give shared, locked, encrypted, versioned state; S3 with use_lockfile = true is a common AWS choice.
  • Lock contention produces a clear error with the holder's details; use force-unlock only for stale locks.
  • Workspaces keep several states per configuration; separate directories/accounts are safer for production.
  • Split state by environment and layer, bootstrap the backend separately, and restrict access.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Infrastructure as Code with Terraform for DevOps Engineer and scores every answer, with what to fix.