Terraform state for a small team: backends, locks, blast radius

How a small team can keep Terraform state safe: a versioned GCS backend, locking you can trust, and a state layout that limits the damage of a bad apply.

Published

Updated

—

Reading time

10 min

Take a team of three engineers running a product on Google Cloud: a VPC and DNS, a handful of projects with IAM and buckets, and a few Cloud Run services and jobs. Nobody owns Terraform full time, both people and CI need to run it, and the infrastructure is small enough that a bad apply is a bad afternoon rather than an outage report. This is a design review of the state setup for that system, from requirements to the places it breaks. The example is hypothetical, but each mechanism is Terraform's or Google Cloud's own.

Terraform state maps the resources in your code to the real objects in your cloud project, and every plan compares your configuration, that map and the live API. Lose it and Terraform no longer knows what it owns. Corrupt it and the next apply may try to recreate things that exist. Let two people write it at once and you get both. The design below keeps it boring.

What the design has to satisfy#

  • The setup survives a month of nobody touching it.
  • Humans and CI can both plan, and only one party applies at a time.
  • A change to a Cloud Run service does not share a plan with the network.
  • An interrupted apply or a wrong resource move can be recovered without heroics.
  • The bill is close to zero, because remote state is a handful of small objects.

Committing state to git fails the first two requirements: there is no locking, so two people planning from the same commit both believe they are first, and state can contain sensitive values in plain text, which turns repository history into a secrets store (see HashiCorp's guidance on sensitive data in state). A local ignored file fails everything. That leaves a remote backend, and for a team already on Google Cloud the GCS backend is the least to operate: a bucket, with locking supported by the backend. A managed platform such as HCP Terraform gives you runs, policy and history built in, at a price you should check against your own usage.

One bucket, several state objects. The prefix is the unit of locking and the unit of blast radius.

One versioned bucket, created outside Terraform#

The state bucket is the one resource created by hand, because it must exist before Terraform can store anything. Never let a configuration manage the bucket that holds its own state, since a careless destroy would take the safety net with it. Keep it in a dedicated project, with uniform bucket-level access and Object Versioning on; the Terraform documentation strongly recommends versioning so that state can be recovered. The names below are placeholders.

terminalbash
gcloud storage buckets create gs://example-tf-state \
  --project=example-tf-admin \
  --location=europe-west4 \
  --uniform-bucket-level-access
 
gcloud storage buckets update gs://example-tf-state --versioning
 
gcloud storage buckets describe gs://example-tf-state \
  --format="default(versioning_enabled)"

The last command confirms versioning is on rather than assuming it. Bucket names are global, so choose one unlikely to collide. See using Object Versioning and creating buckets.

Versioning keeps every overwritten generation, so the bucket grows slowly forever unless a lifecycle rule bounds it. Setting isLive to false in a lifecycle condition matches noncurrent versions, and numNewerVersions counts how many newer ones exist. This rule deletes a noncurrent version once 30 newer ones exist; choose a number that covers a few weeks of your own apply frequency:

lifecycle.jsonjson
{
  "rule": [
    {
      "action": { "type": "Delete" },
      "condition": { "isLive": false, "numNewerVersions": 30 }
    }
  ]
}
terminalbash
gcloud storage buckets update gs://example-tf-state \
  --lifecycle-file=lifecycle.json

Three states, split by what a mistake costs#

A single state for everything is simple until the first mistake. Split by blast radius, using one backend prefix each:

texttext
infra/
  network/prod/     # VPC, DNS, shared load balancer
  platform/prod/    # projects, IAM, buckets, Artifact Registry
  apps/prod/        # Cloud Run services and jobs

The two questions are how often a layer changes and how bad it is when wrong. Network configuration changes rarely and an error can take everything offline, so it deserves its own plan that people read slowly. Application deployments change daily and deserve a small, fast plan. In one state, the daily plan also refreshes and compares the rarely changed, high-impact resources, and a stray diff there looks like noise until the day it is not. The layout of the code around this is covered in one repo, many deployables.

platform/prod/backend.tfhcl
terraform {
  backend "gcs" {
    bucket = "example-tf-state"
    prefix = "platform/prod"
  }
}

The backend block takes a required bucket and an optional prefix; for a non-default workspace the object is named after the prefix and workspace, ending in .tfstate (GCS backend reference). Backend blocks cannot reference variables, so teams often leave the bucket out and pass it as partial configuration with terraform init -backend-config="bucket=example-tf-state".

Splitting has a price in cross-state references. Prefer reading stable values through data sources, such as looking up a network by name, over terraform_remote_state, which needs read access to another state file and therefore to everything in it. Passing an output as a pipeline variable is fine for a handful of values. Splitting by every service instead produces dozens of plans and a tangle of references, and one state for everything means a slow plan, a noisy diff and one lock serialising the whole team. Split where the blast radius genuinely differs.

Who can touch the bucket#

The backend documentation says the credentials in use need the Storage Object Admin role on the bucket. Grant it to a group for humans and to the pipeline's service account, never to individuals one by one:

terminalbash
gcloud storage buckets add-iam-policy-binding gs://example-tf-state \
  --member="group:infra-admins@example.com" \
  --role="roles/storage.objectAdmin"

The backend also supports impersonate_service_account. The engineer authenticates as themselves, holding the Service Account Token Creator role on the named account, and the backend reads the bucket as that service account. That keeps the number of identities with direct bucket access small:

hclhcl
terraform {
  backend "gcs" {
    bucket                      = "example-tf-state"
    prefix                      = "platform/prod"
    impersonate_service_account = "tf-state@example-tf-admin.iam.gserviceaccount.com"
  }
}

Anyone who can read the bucket can read every value Terraform stored, so treat bucket access like production database access. Humans plan locally; applies go through the pipeline.

What the lock does, and what to do when it sticks#

The GCS backend supports state locking, and Terraform locks state for any operation that could write it. If it cannot acquire the lock it does not continue. Ask it to wait instead of failing at once:

terminalbash
terraform plan -lock-timeout=120s

-lock-timeout retries for a period before returning an error. The documentation warns that -lock=false is dangerous if others might run commands against the same workspace, so I ban it in CI and by hand.

A killed run can leave a lock behind. terraform force-unlock LOCK_ID removes it, and the documentation is blunt that unlocking while another party holds the lock could produce multiple writers. The lock ID is printed when acquisition fails and acts as a nonce. The rule: only the person who started the run, or someone who has confirmed from the CI log that the run is dead, force-unlocks. Print the lock ID in the job log so that person can find it.

A pipeline whose applied plan is the reviewed plan#

The goal is that the plan a human approves is the plan that runs. A pipeline that plans on a pull request and then, on merge, creates a fresh plan and applies it does not achieve that: the second plan can differ from the first if main moved or the cloud changed in between. Two properties fix it. First, the apply job applies a saved plan file that an earlier job produced for the exact commit being applied. Second, a human reviews that plan, as a protected environment's required reviewer, before the apply job runs. The pull request plan stays in the design, but as early feedback, not as the thing that is approved.

Terraform's -out writes the plan to a file in an opaque format. The documentation warns that the file holds your full configuration and variable values, including sensitive ones, in plain text, so it is a secret artifact: short retention, private repository, never public. Terraform also refuses to apply a saved plan whose recorded state serial no longer matches the state (the "Saved plan is stale" error), so if anyone applies something between the plan and the apply, the apply fails rather than running a plan built from an old picture.

.github/workflows/terraform-platform.ymlyaml
name: terraform-platform
on:
  pull_request:
    paths: ["infra/platform/**"]
  push:
    branches: [main]
    paths: ["infra/platform/**"]
 
permissions:
  id-token: write
  contents: read
 
concurrency:
  group: terraform-platform
  cancel-in-progress: false
 
defaults:
  run:
    working-directory: infra/platform/prod
 
jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: google-github-actions/auth@v3
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.TF_SA }}
      - uses: hashicorp/setup-terraform@v4
      - run: terraform init -input=false
      - run: terraform plan -input=false -lock-timeout=120s -out=tfplan
      - name: Show the plan for reviewers
        run: terraform show -no-color tfplan >> "$GITHUB_STEP_SUMMARY"
      - if: github.event_name == 'push'
        uses: actions/upload-artifact@v7
        with:
          name: tfplan-${{ github.sha }}
          path: infra/platform/prod/tfplan
          retention-days: 1
 
  apply:
    if: github.event_name == 'push'
    needs: plan
    runs-on: ubuntu-latest
    environment: production-infra   # required reviewers configured on this environment
    steps:
      - uses: actions/checkout@v7
      - uses: google-github-actions/auth@v3
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.TF_SA }}
      - uses: hashicorp/setup-terraform@v4
      - uses: actions/download-artifact@v8
        with:
          name: tfplan-${{ github.sha }}
          path: infra/platform/prod
      - run: terraform init -input=false
      - run: terraform apply -input=false -lock-timeout=120s tfplan

The pipeline authenticates with Workload Identity Federation, described in keyless CI/CD for GitHub Actions. The action majors above (checkout 7, auth 3, setup-terraform 4, upload-artifact 7, download-artifact 8) are the current releases at the time of writing.

What this guarantees: the plan printed in the run summary, which the environment reviewer reads, is byte-for-byte the plan applied, produced from the commit being merged, and Terraform rejects it if state moved. What it does not guarantee: that the pull request plan matches. If main advanced after the PR plan, the merge-commit plan can differ, and the reviewer approving the apply sees the real one. Commit .terraform.lock.hcl and pin the required Terraform version so both jobs run the same providers and CLI, because a plan must be applied with the configuration and providers it was created from.

Refactoring without destroying anything#

Renaming a resource in code looks to Terraform like deleting one object and creating another. A moved block says otherwise:

hclhcl
moved {
  from = google_cloud_run_v2_service.api
  to   = google_cloud_run_v2_service.public_api
}

Terraform looks for an object at the from address in state and treats it as belonging to to, so the rename does not destroy and recreate it. Keep old moved blocks in place: removing them makes configurations that still use the old address plan to delete the object (see refactoring). The imperative terraform state mv edits state outside code review, so I use it only when no moved block can express the move, such as between two separate state files.

Changing a backend block needs terraform init -migrate-state, which copies existing state to the new backend. -reconfigure does the opposite, disregarding existing configuration and preventing migration, and using it by mistake leaves an empty state that wants to create everything again.

Restoring a previous state#

With versioning on, an overwritten state file survives as a noncurrent generation. List the versions and copy one back, which makes the copy the live version:

terminalbash
gcloud storage ls --all-versions gs://example-tf-state/platform/prod/
 
gcloud storage cp \
  'gs://example-tf-state/platform/prod/default.tfstate#GENERATION_NUMBER' \
  gs://example-tf-state/platform/prod/default.tfstate

Quote the source because some shells treat # specially, and list the prefix first because the exact object name depends on the workspace. Make sure no run holds the lock before restoring, and plan straight afterwards to see how the restored state differs from reality. Recovery returns you to an earlier belief about the world, not to the world itself. Rehearse it once on a throwaway state before you need it.

What still breaks under this design#

  • Drift. Someone changes a resource in the console, the next plan shows the difference, and a hurried apply reverts a fix made during an incident. Decide ahead of time whether console changes are allowed during an incident and how they get back into code.
  • Version drift. A teammate on a newer Terraform may write state an older CLI cannot read. Pin the required version in configuration and in CI.
  • Secrets in state and plan files. Both are plain text to anyone who can read them. Limit bucket and artifact access accordingly.
  • Stuck pipeline after a stale plan. If the apply fails as stale, the answer is a new run that produces a new plan, not an override.

If you manage a handful of resources that rarely change, one state in a versioned bucket is enough and splitting it is process you will not use. If nobody on the team will read a plan before applying it, no backend layout helps; build the habit of reading plans first, then add the structure.

Share
All articles →

Why a scheduled job can fail for weeks without anyone noticing, and the retry, idempotency and alerting setup that makes Cloud Scheduler failures loud.

13 min