Configuration and secrets in multi-service Node apps
Nine traps in Node config and Secret Manager on Cloud Run, from silent fallbacks to a trailing newline, each with the symptom, the cause and the fix.
Configuration bugs rarely look like bugs. A service boots with PORT unset and listens on a default nobody chose. A JWT secret falls back to "dev" in a staging environment that is reachable from the internet. A credential is rotated, one of twelve services still holds the old version, and the only symptom is a trickle of 401s. None of these fails a unit test, because the tests run with the environment the developer already has.
The examples below use Node and Cloud Run with Secret Manager, but the first three traps apply to any platform. Each trap has the same parts: what you see, why it happens, and what I do about it. Table 1 lists them all by symptom.
| What you see | Trap |
|---|---|
| Service starts fine with a variable missing | Config is read where it is used, with fallbacks |
| Validation error prints a secret | The error message echoes values |
| HMAC or password check fails for no visible reason | Trailing newline in the stored secret |
Another service in another project gets PERMISSION_DENIED | Secret access is per identity, not per project |
| New secret version, nothing changes | Env var secrets are resolved at instance start |
| A deploy removed all secrets from the service | --set-secrets replaces the whole set |
| A value split into two variables | Commas in --set-env-vars |
| Production reads a staging value, or the reverse | Environments share a secret store |
Trap: config is read where it is used#
What you see. process.env.PORT in one file, process.env.JWT_SECRET ?? "dev-secret" in another, a feature flag read inside a request handler. The service boots with a variable missing and fails on the first request that needs it, or never fails at all.
Why. Nothing forces the environment to be complete before traffic arrives. The fallback case is the dangerous one: the day the variable is absent in a deployed environment (a renamed secret, a copy-pasted deploy command, a new region), the service starts and signs tokens with a string that sits in your Git history. Anyone who can read the repository can mint valid sessions.
Fix. One file per service is allowed to touch process.env. It reads the environment, validates it against a schema, coerces types and exports a frozen object. If anything is wrong the module throws on import, the process exits, and Cloud Run keeps the previous revision serving because the new one never becomes ready. A lint rule banning process.env outside src/config/env.ts keeps the rest of the codebase honest. In my services this does two jobs: a half-configured service shows up as a failed deploy instead of a real user's request failing at 3 a.m., and the config module becomes the complete, typed answer to "what does this service need to run?", which I could never answer with confidence when any file might quietly read the environment.
I use TypeBox because the schema is also the TypeScript type. This is written against @sinclair/typebox 0.34.
import { Type, type Static } from "@sinclair/typebox";
import { Value } from "@sinclair/typebox/value";
const EnvSchema = Type.Object({
NODE_ENV: Type.Union([
Type.Literal("development"),
Type.Literal("test"),
Type.Literal("production"),
]),
PORT: Type.Integer({ minimum: 1, maximum: 65535, default: 8080 }),
LOG_LEVEL: Type.Union(
[Type.Literal("debug"), Type.Literal("info"), Type.Literal("warn"), Type.Literal("error")],
{ default: "info" },
),
GCP_PROJECT_ID: Type.String({ minLength: 1 }),
PUBLIC_BASE_URL: Type.String({ pattern: "^https?://" }),
// Secrets: required, no defaults, ever.
JWT_SECRET: Type.String({ minLength: 32 }),
DATABASE_URL: Type.String({ minLength: 1 }),
});
export type Env = Static<typeof EnvSchema>;
function loadEnv(source: NodeJS.ProcessEnv): Env {
const raw: Record<string, unknown> = {};
for (const key of Object.keys(EnvSchema.properties)) {
const value = source[key];
// Treat empty strings as missing, and strip a single trailing newline.
if (value !== undefined && value.trim() !== "") raw[key] = value.replace(/\r?\n$/, "");
}
const withDefaults = Value.Default(EnvSchema, raw);
const converted = Value.Convert(EnvSchema, withDefaults);
if (!Value.Check(EnvSchema, converted)) {
const problems = [...Value.Errors(EnvSchema, converted)].map(
(e) => ` ${e.path || "/"}: ${e.message}`,
);
throw new Error(`Invalid environment configuration:\n${problems.join("\n")}`);
}
return Object.freeze(converted as Env);
}
export const env = loadEnv(process.env);Several details are deliberate. The function iterates the schema, not process.env, so a typo such as DATABSE_URL surfaces as a missing /DATABASE_URL instead of being ignored. Value.Convert turns "8080" into 8080 and rejects "eighty". Defaults exist for tuning knobs (PORT, LOG_LEVEL) and never for credentials. The next trap explains why the error text carries paths and not values.
For a small worker, a hand-written required(name) that throws on a missing or blank variable gives you the same property. What matters is that validation happens once, at startup, in one place.
For local development, Node loads .env files itself: --env-file since 20.6 and --env-file-if-exists since 22.9, which does not fail when the file is absent (see the CLI documentation):
{
"scripts": {
"dev": "node --env-file-if-exists=.env --watch dist/server.js"
}
}Git-ignore .env and .env.*, commit a .env.example with placeholders only, compare its keys with the schema in CI so it never goes stale, and keep .env in .dockerignore so it cannot be copied into an image.
One distinction saved me real confusion: some "environment variables" are not runtime configuration at all. Frontend frameworks inline certain values into the compiled bundle at build time. Changing one means rebuilding, not redeploying, validating it at server startup validates nothing, and it is public the moment it ships to a browser. My conventions mark those variables explicitly, and nothing secret is allowed to travel that path.
Trap: the validation error prints the secret#
What you see. A boot failure in the logs that includes the malformed value, for example a pattern mismatch message quoting the string it tried to match.
Why. Startup errors go to your log sink, where retention is long and access is broad. A validator that echoes the offending value has just published the secret there.
Fix. Report the path and the rule, never the value, as loadEnv does above. Related: never log the env object, even at debug level. For redacting credentials from request logs, see the pino configuration in structured logging on a budget.
Trap: the newline inside the secret#
What you see. A webhook signature check or a database login fails, the secret "looks right", and rotating it makes no difference.
Why. echo "value" | gcloud secrets create ... stores value followed by a newline. Secret Manager stores bytes exactly as given, so the consumer receives the newline too, and an HMAC key or password that differs by one byte is a different key.
Fix. Create secrets with printf '%s', which adds nothing:
printf '%s' "$(openssl rand -base64 48)" | \
gcloud secrets create jwt-secret \
--replication-policy=user-managed \
--locations=europe-west1 \
--data-file=-The loadEnv function above strips one trailing newline as a second line of defence, but that is a patch for a stored value that is already wrong; fix the secret as well. User-managed replication in one region is a data-residency choice. Automatic replication is simpler if you do not care where copies live, and the replication policy cannot be changed after creation.
Trap: secret access belongs to an identity, not a project#
What you see. A second service, often in a different project, needs a secret the first service already reads. It deploys, starts, asks Secret Manager for the value and exits with PERMISSION_DENIED. The tempting fix is secretAccessor on the whole secrets project.
Why. Secret Manager has no concept of sharing a secret with a project. Access is plain IAM on the secret resource, and the member in a binding can be any principal, including a service account that belongs to another project. The first service's grant says nothing about the second service's account, even when they read the same secret for the same reason. The identity that matters is the runtime service account of the workload (the --service-account of a Cloud Run service, not the deployer and not the project). A workload running as the default compute service account needs the grant on that account, which is one more reason to give every service its own.
Fix. Grant roles/secretmanager.secretAccessor on each individual secret, to each runtime service account that needs it, and never at project level:
gcloud secrets add-iam-policy-binding db-password \
--project=secrets-prod \
--member="serviceAccount:admin-runtime@admin-prod.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"When the secret lives in another project, the consumer references it by its full resource name using the project number, not the project ID, as described in the Cloud Run secrets documentation:
gcloud run deploy admin-api \
--project=admin-prod --region=europe-west1 \
--service-account=admin-runtime@admin-prod.iam.gserviceaccount.com \
--update-secrets=DB_PASSWORD=projects/123456789012/secrets/db-password:4The diagram shows why CI needs no read access: the deploy contains references such as jwt-secret:3, and only the runtime identity resolves them. The deployer needs permission to create revisions and to act as the runtime service account, nothing on the secrets. Workload Identity Federation for GitHub Actions covers how that CI identity is set up without key files.
A grant created seconds earlier may not be effective yet. A pipeline that creates a service account, grants it access and deploys in one run can fail once and succeed on retry; retry rather than widening the grant.
Where the secrets themselves should live#
If staging, production and admin tooling sit in separate projects, I keep one dedicated secrets project per environment: secrets-staging and secrets-prod. Nothing holds a Secret Manager role at project level there except a small administrators group, and every consumer grant is on an individual secret. The reasons:
- The environment boundary is a project boundary. No IAM mistake inside
secrets-stagingcan expose a production value, because production values are not in it. - Administration and Data Access audit logging are concentrated in two projects instead of every project that happens to hold a secret.
- A credential shared by two workloads exists once, so rotating it means adding one version.
Secrets private to one workload can stay in that workload's project. The grants belong in Terraform, with google_secret_manager_secret_iam_member resources that add one binding each, so a manual emergency grant is neither silently wiped nor silently kept. Secret values never enter Terraform state; versions are added with gcloud secrets versions add. Keep the project number in one pipeline variable rather than scattering it across service definitions. For the wider IAM model, see GCP IAM least privilege.
Trap: you added a version and nothing changed#
What you see. A new secret version is enabled, the old credential is revoked upstream, and the service starts failing, or the service keeps using the old value after you expected rotation.
Why. Cloud Run exposes a secret in two ways, and they behave differently:
- Env var. The value is resolved when an instance starts. A new version does nothing until instances are replaced. Google's documentation recommends pinning a specific version for this mode. With
latest, two instances of the same revision can start minutes apart and hold different values. - Volume mount. The secret appears as a file. A mount that references
latestis read at file-read time, so rotation works without a redeploy, but only if your code re-reads the file instead of caching it forever. Mounts also suit multi-line material such as PEM keys.
gcloud run deploy auth-api \
--image=europe-west1-docker.pkg.dev/my-project/services/auth-api:1.42.0 \
--service-account=auth-api@my-project.iam.gserviceaccount.com \
--set-env-vars=NODE_ENV=production,GCP_PROJECT_ID=my-project \
--set-secrets=JWT_SECRET=jwt-secret:3,DATABASE_URL=database-url:7,/secrets/signing/key.pem=signing-key:latestFix. Pin versions for env var secrets, so every rotation is an explicit deploy that can be rolled back. Use a volume mount with latest only where you need rotation without a deploy, and write the code to re-read the file.
Secret Manager can send rotation reminders to a Pub/Sub topic, but it does not rotate anything; you write the handler. The safe order for a pinned secret:
- Add the new version.
- If the upstream system accepts two credentials at once, enable both.
- Deploy each consumer pinned to the new version, one service at a time.
- Watch error rates. If they rise, roll back the revision; the old version is still enabled.
- Revoke the old credential upstream, then disable the old version, then destroy it after a grace period.
For signing keys, the verifier should accept both old and new keys (a key ID in the token header makes this clean) until every token signed with the old key has expired.
I should be honest about my own setup: rotation there is still a manual checklist following that order, not an automated process. What made it tractable was reducing how many long-lived secrets there are to rotate. The one-config-module rule made every consumer of a credential knowable, and deployment credentials are already short-lived and keyless, so each retired static key is one fewer rotation to remember.
At the time of writing, Secret Manager pricing charges per active secret version per location per month, plus a fee per 10,000 access operations, with a small free tier. An enabled old version bills like the current one, so disable or destroy versions once a rotation has settled. Code that calls the access API on every request turns a fixed cost into one that scales with traffic, which is another reason to let the platform resolve secrets at instance start.
Trap: the deploy flag that drops everything#
What you see. After a routine deploy the service crashes at boot with a config validation error for a secret that was there yesterday.
Why. --set-secrets replaces the entire set of secrets on the service with the list you give it. A deploy that sets only the new secret removes the rest.
Fix. Use --update-secrets and --update-env-vars for incremental changes. Better, check a declarative service YAML or Terraform resource into the repository per service and have the deploy step apply it, so the full set is always stated.
Trap: a comma splits your value in two#
What you see. ALLOWED_ORIGINS=https://a.example,https://b.example arrives as one variable with the first origin, and the second origin is rejected as a malformed variable.
Why. --set-env-vars and --update-env-vars split on commas.
Fix. Change the delimiter with gcloud's escaping syntax: a caret, a new delimiter, a caret, then the list.
gcloud run services update web-api \
--update-env-vars="^@^ALLOWED_ORIGINS=https://a.example,https://b.example@LOG_LEVEL=info"Trap: services quietly disagree about their environment#
What you see. One service still reads secret version 2 after the rest moved to 3. One has LOG_LEVEL=debug left from an incident. One lacks a variable that matters only on a rare code path.
Why. Each service is configured separately, and nothing compares them.
Fix. An audit script. Sorting by variable name puts every service's view of JWT_SECRET or LOG_LEVEL on adjacent lines, which makes a stale version or an odd value obvious.
#!/usr/bin/env bash
set -euo pipefail
REGION=europe-west1
for svc in $(gcloud run services list --region="$REGION" --format='value(metadata.name)'); do
gcloud run services describe "$svc" --region="$REGION" --format=json |
jq -r --arg svc "$svc" '
.spec.template.spec.containers[0].env[]?
| [$svc, .name,
(if .valueFrom.secretKeyRef
then "secret:\(.valueFrom.secretKeyRef.name):\(.valueFrom.secretKeyRef.key)"
else "plain:\(.value)" end)]
| @tsv'
done | sort -k2,2 -k1,1 | column -tThe script prints plain values, so it is for people who can already view the services, and by construction none of those values are secrets. Run it per environment, diff staging against production, and consider a scheduled job that fails when two services disagree on a secret version for longer than a rotation window.
A related rule: plain env vars are not secret. Anyone with run.services.get can read them. If you would not paste a value into a chat channel, it belongs in Secret Manager.
Trap: staging and production share a store#
What you see. A staging deploy reads a production credential, or a production service starts with a staging database URL.
Why. Secret IDs are reused across environments inside one project, or one team-wide group holds access to both.
Fix. The per-environment secrets project from the identity trap above. Because the project already encodes the environment, secret IDs do not need to: db-password in secrets-prod and db-password in secrets-staging are the same logical secret, and deploy configuration differs only in the project number. I also give staging real values through the same deployment mechanism as production, because a configuration path you never rehearse is one that fails when it matters; between my environments the variable names and validation are identical and only the values differ. Name secrets after what they are (payments-api-key), not who uses them, because consumers change and the credential does not.
If you want a checklist to run on every new service, it is short: one config module and a lint rule, no credential defaults, secrets created with printf, one service account per service with per-secret grants, pinned versions for env var secrets, and an audit script on a schedule.
Where this stops being worth it#
A single service on a platform with a built-in secret store and one environment needs only the validator; a drift audit for one service is ceremony. Secrets that are genuinely dynamic, such as per-tenant database credentials or tokens minted per request, do not fit platform-bound env vars, and want the Secret Manager API or a broker such as Vault with in-process caching. On Kubernetes with an operator that syncs secrets from a cloud store, keep the validation half of this article and let the operator handle the binding.
Related articles
All articles →Why a scheduled job can fail for weeks without anyone noticing, and the retry, idempotency and alerting setup that makes Cloud Scheduler failures loud.
13 min
Mocks pass while real systems fail. How I run a small Playwright suite against a deployed, no-traffic staging revision and gate promotion on the result.
11 min
JSON logs that Cloud Logging understands, correlated by trace, with the noise excluded before it is billed. A pino setup for Cloud Run and what it costs.
12 min