Secret Manager across GCP projects without leaking access

The first cross-project secret incident I remember was not a leak. It was a deploy that refused to start. An internal admin tool lived in its own project, and it needed the same database password as the production API. The API had been reading that secret for months, so everyone assumed "the secret is shared". The admin service booted, asked Secret Manager for the value, got a permission error, and exited. The fix that someone proposed in the incident channel was to grant roles/secretmanager.secretAccessor on the whole production project to the admin tool's service account. It would have worked in thirty seconds, and it would have let an internal dashboard read every production credential that existed, and every one that would be created afterwards.

That moment is typical of how multi-project secret setups decay. Projects are a good isolation boundary, and Google Cloud encourages splitting staging, production and tooling apart. But the moment one workload needs a secret that lives somewhere else, the quickest path is always a broader grant. This article describes how I lay out secrets when a system spans several projects, how cross-project access actually works, and the Terraform and audit tooling that keep it narrow. It builds on two earlier pieces: configuration and secrets in multi-service Node apps, which covers how a single service consumes secrets, and GCP IAM least privilege, which covers the wider IAM model. Here the focus is the space between projects.

The constraints#

• Several projects per environment. A typical layout has an application project, an admin or back-office project, and sometimes a data project, for both staging and production.
• Some secrets are shared across those projects. A database password, an email provider key or a webhook signing secret may be needed by workloads in two or three of them.
• Staging must never be able to read production. Not through a misconfigured grant, not through a shared project, not through a group that happens to contain both identities.
• Every read must be attributable. When a key leaks, I want to know which identity read which version, and when.
• Rotation must be routine. A credential that is painful to rotate does not get rotated.
• The grant model must be reviewable. Someone reading one file should be able to say who can read what.

How cross-project access works#

Secret Manager has no concept of sharing a secret with another project. Access is plain IAM on the secret resource, and the member in a binding can be any principal, including a service account that belongs to a different project. There is no trust relationship between the two projects, no export step and no shared registry. A binding like this is the entire mechanism:

gcloud secrets add-iam-policy-binding db-password \
  --project=secrets-prod \
  --member="serviceAccount:admin-runtime@admin-prod.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

On the consuming side, Cloud Run references a secret in another project by its full resource name, using the project number rather than the project ID:

gcloud run deploy admin-api \
  --project=admin-prod --region=europe-west1 \
  --service-account=admin-runtime@admin-prod.iam.gserviceaccount.com \
  --update-secrets=DB_PASSWORD=projects/123456789012/secrets/db-password:4

Two consequences follow, and both are what caught the admin tool in the story above.

First, each consumer needs its own grant. The production API's service account holding access says nothing about the admin tool's service account, even when they read the same secret for the same reason. There is no "this secret is available to project X". There are only identities.

Second, the identity that matters is the runtime service account of the consuming workload, not the deployer and not the project. For Cloud Run it is the service's --service-account. For Cloud Build steps that fetch a build-time secret, it is the build's service account. If a workload runs as a default service account, the grant has to go to that default account, which is one more reason to give every workload its own identity.

Where secrets should live: options compared#

The decision#

I use one dedicated secrets project per environment. Production workloads, wherever they run, read from secrets-prod. Staging workloads read from secrets-staging. Nothing holds a Secret Manager role at the project level of either secrets project apart from a small administrators group, and every consumer grant is on an individual secret.

The reasons, in order of weight:

• The environment boundary is a project boundary. No IAM mistake inside secrets-staging can expose a production value, because production values are not in it.
• Administration is concentrated. Only a handful of people need roles/secretmanager.admin, and only on the two secrets projects. Application projects hold no Secret Manager admin grants at all.
• Audit logging is cheap to scope. Data Access logs for Secret Manager are switched on in exactly two projects, not in every project that happens to contain a secret.
• Shared credentials exist once. Rotating the database password means adding one version, not three.

Secrets that are genuinely private to a single workload can still live in that workload's own project. I do not force everything through the central project, but anything read by more than one project goes there.

Implementation#

Naming#

Because the project already encodes the environment, secret IDs do not. db-password in secrets-prod and db-password in secrets-staging are the same logical secret in two environments, which means deploy configuration differs only in the project number. I name secrets after what they are, not who uses them: payments-api-key, not api-service-payments-key. Consumers change, the credential does not.

Labels carry the rest: an owner label naming the team or system that rotates it, and a data-class label if you classify credentials. Labels are searchable through Cloud Asset Inventory and they are the first thing I read during an incident.

Replication and CMEK#

Replication is chosen when a secret is created, and the documentation is explicit that the locations cannot be updated afterwards. Changing it later means creating a new secret, copying the value, moving every consumer and deleting the old one.

• Automatic replication lets Google choose where payload data lives and is billed as a single location.
• User-managed replication pins data to the regions you list. Each listed region is billed as a separate location, and the secret depends on those regions being available.

I use automatic replication unless there is a data-residency requirement. When there is one, I pick two regions inside the required geography and accept the doubled storage line on the bill.

Customer-managed encryption keys are optional. If policy demands them, the CMEK guide spells out the rules that matter: an automatically replicated secret needs a key in the global Cloud KMS location, a user-managed one needs a key in each replica's location, and the Secret Manager service agent needs roles/cloudkms.cryptoKeyEncrypterDecrypter on the key. The operational consequence is the one people forget: if the key is disabled or the service agent loses access, reading any version fails. A KMS mistake then stops every new Cloud Run instance that references the secret. Changing the key later only affects new versions; existing ones are not re-encrypted.

Grants in Terraform#

The grant matrix belongs in code, in one file per environment, reviewed like any other change. The shape I use keeps secrets and their consumers side by side, so a reviewer sees in one place who reads what:

locals {
  secrets = {
    "db-password" = {
      locations = ["europe-west1", "europe-west4"]
      consumers = [
        "api-runtime@app-prod.iam.gserviceaccount.com",
        "admin-runtime@admin-prod.iam.gserviceaccount.com",
      ]
    }
    "payments-api-key" = {
      locations = []
      consumers = ["api-runtime@app-prod.iam.gserviceaccount.com"]
    }
    "smtp-password" = {
      locations = []
      consumers = ["mailer-runtime@app-prod.iam.gserviceaccount.com"]
    }
  }
 
  grants = merge([
    for secret, cfg in local.secrets : {
      for sa in cfg.consumers : "${secret}/${sa}" => { secret = secret, sa = sa }
    }
  ]...)
}
 
resource "google_secret_manager_secret" "this" {
  for_each            = local.secrets
  project             = var.project_id
  secret_id           = each.key
  labels              = { owner = "platform" }
  deletion_protection = true
 
  replication {
    dynamic "auto" {
      for_each = length(each.value.locations) == 0 ? [1] : []
      content {}
    }
    dynamic "user_managed" {
      for_each = length(each.value.locations) > 0 ? [1] : []
      content {
        dynamic "replicas" {
          for_each = each.value.locations
          content {
            location = replicas.value
          }
        }
      }
    }
  }
}
 
resource "google_secret_manager_secret_iam_member" "consumer" {
  for_each  = local.grants
  project   = var.project_id
  secret_id = google_secret_manager_secret.this[each.value.secret].secret_id
  role      = "roles/secretmanager.secretAccessor"
  member    = "serviceAccount:${each.value.sa}"
}

Three details are deliberate. The resources are _iam_member, which add one binding and leave others alone, so a manual emergency grant is not silently wiped (and not silently kept either, because the audit script below will show it). The map key includes the member, so removing one consumer destroys exactly one binding. And deletion_protection stops a refactor from deleting a production secret along with its versions.

Terraform never holds secret values here. Versions are added out of band, by a person with printf '%s' ... | gcloud secrets versions add or by the rotator, so values do not end up in Terraform state.

Rotation notifications#

Secret Manager does not rotate anything itself; it sends reminders. The event notifications guide describes the setup, which has one step people skip: the Secret Manager service agent has to exist and be allowed to publish to the topic.

gcloud beta services identity create \
  --service=secretmanager.googleapis.com --project=secrets-prod
 
gcloud pubsub topics create secret-events --project=secrets-prod
 
gcloud pubsub topics add-iam-policy-binding secret-events \
  --project=secrets-prod \
  --member="serviceAccount:service-123456789012@gcp-sa-secretmanager.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher"

Then attach the topic and a schedule to the secrets that rotate. In Terraform, that is two more blocks on the secret resource:

  topics {
    name = google_pubsub_topic.secret_events.id
  }
 
  rotation {
    next_rotation_time = "2026-11-02T06:00:00Z"
    rotation_period    = "2592000s" # 30 days
  }

The topic receives events for the whole lifecycle, not only rotation reminders: SECRET_ROTATE, SECRET_VERSION_ADD, SECRET_VERSION_DISABLE, SECRET_VERSION_DESTROY and more. Each message carries an eventType attribute and a secretId attribute holding the full resource name. The rotator subscribes, filters on SECRET_ROTATE, creates a new credential with the provider and adds it as a new version:

import { SecretManagerServiceClient } from "@google-cloud/secret-manager";
import { createProviderKey } from "./provider.js";
 
const client = new SecretManagerServiceClient();
 
export async function handle(attributes: Record<string, string>): Promise<void> {
  if (attributes.eventType !== "SECRET_ROTATE") return;
  const secretName = attributes.secretId; // projects/123456789012/secrets/payments-api-key
  const newKey = await createProviderKey();
  await client.addSecretVersion({
    parent: secretName,
    payload: { data: Buffer.from(newKey, "utf8") },
  });
}

The rotator's own service account gets roles/secretmanager.secretVersionAdder on the secrets it rotates, and nothing that lets it read values. It can put a new key in; it cannot take any key out. Pub/Sub delivers at least once, so the handler must tolerate duplicates; the patterns in idempotent Pub/Sub consumers apply directly. A SECRET_VERSION_ADD subscriber is also a good place to open a pull request that bumps pinned versions in each consumer's deploy config.

Data Access audit logs#

Admin Activity logs record grants and version changes and are always on. Reads are different: AccessSecretVersion is logged as a Data Access (DATA_READ) operation, and Data Access logs are off by default for most services. Turn them on in the secrets projects only:

resource "google_project_iam_audit_config" "secret_reads" {
  project = var.project_id
  service = "secretmanager.googleapis.com"
  audit_log_config {
    log_type = "DATA_READ"
  }
}

After that, "who read this secret last week" is a query:

gcloud logging read \
  'protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion"
   AND protoPayload.resourceName:"secrets/db-password"' \
  --project=secrets-prod --freshness=7d \
  --format="table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)"

A read by a principal that is not in the Terraform consumer list is the most useful alert in this whole setup.

An audit script: who can read what#

Terraform describes intent. The live policy is what matters. This script lists every principal that can reach a secret in a project, from secret-level and project-level bindings, and marks which project each service account comes from:

#!/usr/bin/env bash
set -euo pipefail
PROJECT="${1:?usage: audit-secret-access.sh SECRETS_PROJECT_ID}"
 
origin='(capture("@(?<p>[a-z0-9-]+)\\.iam\\.gserviceaccount\\.com").p // "-")'
 
{
  printf 'SECRET\tROLE\tMEMBER\tSA_PROJECT\n'
 
  # Bindings set on individual secrets
  gcloud asset search-all-iam-policies \
    --scope="projects/${PROJECT}" \
    --asset-types="secretmanager.googleapis.com/Secret" \
    --format=json |
  jq -r ".[] | (.resource | split(\"/\") | last) as \$s
    | .policy.bindings[] | .role as \$r | .members[]
    | [\$s, \$r, ., ${origin}] | @tsv"
 
  # Project-level bindings apply to every secret in the project
  gcloud projects get-iam-policy "$PROJECT" --format=json |
  jq -r ".bindings[] | select(.role | test(\"secretmanager|^roles/owner$\"))
    | .role as \$r | .members[]
    | [\"(all secrets)\", \$r, ., ${origin}] | @tsv"
} | column -t -s $'\t'

I read the output for four things: any service account from a staging project in a production secrets project, any user: member, any deleted: member left behind by a removed service account, and any project-level row outside the administrators group. The script does not expand groups, custom roles or grants inherited from folders. For that, Policy Analyzer is authoritative:

gcloud asset analyze-iam-policy --project=secrets-prod \
  --permissions="secretmanager.versions.access" --expand-groups

Run both on a schedule and diff the output against the previous run. New rows are either a reviewed Terraform change or an incident.

Trade-offs and failure modes#

• More bindings. A secret read by four workloads has four bindings. That is the point, but it means Terraform is not optional; nobody maintains this by hand for long.
• Project numbers leak into deploy config. Cross-project references use the project number. Keep it in one environment variable per pipeline, not scattered across service definitions.
• Deleted service accounts do not come back. If a consumer's service account is deleted and recreated with the same name, its old bindings do not apply to the new account; they linger as deleted: members. Re-apply Terraform after recreating identities.
• Propagation delay. A new grant usually works within minutes, not instantly. A pipeline that creates a service account, grants it access and deploys in the same run can fail on its first attempt; retry the deploy rather than widening the grant.
• Replication is permanent. Picking user-managed replication in the wrong regions turns into a migration later. Decide once, per secret, deliberately.
• CMEK adds a dependency. A disabled key breaks every consumer at its next instance start. Alert on key state changes as you would on the secret itself.
• Perimeters. If you use VPC Service Controls, the secrets project and its consumers must be allowed to talk across the perimeter, or reads fail with a policy violation that looks like a permission error.
• Emergency grants. In an incident someone will grant project-level access. The audit script and a SetIamPolicy alert on the secrets project turn that into a visible, time-limited exception instead of a permanent one.

When not to do this#

If your whole system lives in one project per environment, you do not need a separate secrets project; secret-level grants inside that project give you the same isolation with less plumbing. If you have a single environment and a single service, the single-service setup is enough on its own. And if your organization already runs a central secrets platform, such as Vault or a landing zone with its own secrets projects, fit into it rather than building a parallel one. The rules that transfer everywhere are the small ones: grant per secret, grant per consumer, and make every read visible.