Structured logging on a budget
JSON logs that Cloud Logging understands, correlated by trace, with the noise excluded before it is billed. A pino setup for Cloud Run and what it costs.
A Node service on Cloud Run that logs "incoming request" and "request completed" for every request, and prints a debug line inside a hot loop, can produce more gigabytes of logs than it stores in its database. And when something does break, the lines that matter are plain strings with no severity, no request ID and no way to find the other lines from the same request. You pay for volume and get little from it. On a large-scale social platform I'm building, the main reader of those logs is me, usually at the worst hour, which is the standard I set the design against.
This walkthrough builds the opposite: a pino logger whose output Cloud Logging parses properly, a per-request child logger that carries the trace ID, redaction of credentials, and exclusion filters that decide what gets stored. The examples use pino and Fastify 5, but the Cloud Logging parts apply to any language that can print JSON to stdout.
Know what Cloud Run does with stdout#
Cloud Run captures everything a container writes to stdout and stderr. Per the Cloud Run logging documentation, a line that is a single serialised JSON object becomes a structured entry with a jsonPayload; anything else lands in textPayload. A few JSON keys are lifted into the LogEntry itself. The special fields used here:
| JSON key in your line | Becomes | Why it matters |
|---|---|---|
severity | LogEntry.severity | Filtering, alerting, colouring in Logs Explorer |
message | The summary line in Logs Explorer | Readable lists instead of raw JSON |
time | LogEntry.timestamp | When you logged it, not when it was received |
httpRequest | LogEntry.httpRequest | Status, latency, method and URL as indexed fields |
logging.googleapis.com/trace | LogEntry.trace | Groups your lines under the request that produced them |
logging.googleapis.com/trace_sampled | LogEntry.traceSampled | Links the entry to a sampled trace in Cloud Trace |
Severity accepts DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT and EMERGENCY. pino's default output uses a numeric level (30 for info, 50 for error) and a msg key, neither of which Cloud Logging recognises. Out of the box a pino line shows up with severity DEFAULT, so every severity filter and error alert misses it.
Cloud Run also writes its own request log for every request, under run.googleapis.com/requests, with httpRequest already filled in: status, latency, sizes, user agent and remote IP. That means the application does not need an access log of its own, which is where most of the volume usually comes from.
I write JSON to stdout rather than calling the Cloud Logging API from the process. API writes add batching, retries and quota to your service, and on Cloud Run work left over after the response is not guaranteed CPU. Stdout hands the problem to the platform. Google also publishes a maintained pino configuration package if you prefer not to own the formatter below, and if you already run an OpenTelemetry collector, sending logs through it is reasonable.
Make pino speak Cloud Logging#
Four changes from the defaults: severity instead of level, message instead of msg, an ISO time, and no pid or hostname (Cloud Run labels entries with the service and revision). Errors also get a stack_trace copy.
import pino from "pino";
import { env } from "./config/env";
// pino level label -> Cloud Logging severity
const SEVERITY: Record<string, string> = {
trace: "DEBUG",
debug: "DEBUG",
info: "INFO",
warn: "WARNING",
error: "ERROR",
fatal: "CRITICAL",
};
export const logger = pino({
level: env.LOG_LEVEL, // "info" in production
messageKey: "message",
base: undefined,
timestamp: pino.stdTimeFunctions.isoTime,
formatters: {
level(label) {
return { severity: SEVERITY[label] ?? "DEFAULT" };
},
log(object) {
// Error Reporting looks for a stack trace in `stack_trace`, then `exception`, then `message`.
const err = object.err as Error | undefined;
return err?.stack ? { ...object, stack_trace: err.stack } : object;
},
},
redact: {
paths: [
"req.headers.authorization",
"req.headers.cookie",
'req.headers["x-api-key"]',
"*.password",
"*.token",
"*.secret",
"*.email",
],
censor: "[redacted]",
},
});Two constraints. The pino API docs say formatters.level cannot replace the numeric level when you use multiple transport targets, because pino needs the number to route lines; writing straight to stdout with no transport, as here, avoids that and is the fast path. And do not add pino-pretty in production: pretty output is not JSON, so Cloud Logging falls back to textPayload and you lose everything in the table above. Use it locally by piping (node dist/server.js | pino-pretty).
Error Reporting's format rules check stack_trace, exception and message in that order, so copying err.stack into stack_trace makes every log.error({ err }, "...") appear as a grouped error without a client library.
Redaction#
A request logger that serialises headers writes bearer tokens into a system with long retention and broad read access. Pino redaction applies at serialisation time, so the value never reaches stdout. The redact block above covers the authorisation header, cookies, an API key header, and credential-shaped fields one level down.
Redaction paths are explicit, and a wildcard only matches one level, so a nested body.user.password needs its own path. Treat the list as a living document and add to it when a review finds a new shape. Redaction is my second line of defence; the first is not logging whole request objects at all, so a credential only reaches the logs if the mistake is made twice, in two different files. Never log the parsed config object, not even at debug level. The same goes for validation errors at startup, which should report paths and rules rather than values (see configuration and secrets in multi-service Node apps).
Bind the trace to every request#
Cloud Run adds a W3C traceparent header to incoming requests, and Google Cloud services still accept the older X-Cloud-Trace-Context header. To make Logs Explorer nest your lines under the request log, set logging.googleapis.com/trace to projects/PROJECT_ID/traces/TRACE_ID. Cloud Run does not give the container the project ID as an environment variable, so pass it at deploy time and validate it with the rest of the config.
import Fastify from "fastify";
import type { IncomingHttpHeaders } from "node:http";
import { logger } from "./logger";
import { env } from "./config/env";
const TRACEPARENT = /^00-([0-9a-f]{32})-[0-9a-f]{16}-([0-9a-f]{2})$/;
function traceContext(headers: IncomingHttpHeaders) {
let traceId: string | undefined;
let sampled = false;
const tp = headers["traceparent"];
const legacy = headers["x-cloud-trace-context"];
const m = typeof tp === "string" ? TRACEPARENT.exec(tp) : null;
if (m) {
traceId = m[1];
sampled = (parseInt(m[2], 16) & 1) === 1;
} else if (typeof legacy === "string") {
traceId = legacy.split("/")[0]; // TRACE_ID/SPAN_ID;o=OPTIONS
sampled = legacy.includes(";o=1");
}
return { traceId, sampled };
}
// Deterministic per trace: every service keeps or drops the same requests.
function keepDebug(traceId: string | undefined, rate: number) {
if (!traceId) return false;
return parseInt(traceId.slice(-8), 16) / 0xffffffff < rate;
}
export const app = Fastify({
loggerInstance: logger,
disableRequestLogging: true, // Cloud Run's request log already records every request
childLoggerFactory(parent, bindings, opts, rawReq) {
const { traceId, sampled } = traceContext(rawReq.headers);
const fields = traceId
? {
"logging.googleapis.com/trace": `projects/${env.GCP_PROJECT_ID}/traces/${traceId}`,
"logging.googleapis.com/trace_sampled": sampled,
}
: {};
const childOpts = keepDebug(traceId, env.DEBUG_SAMPLE_RATE) ? { ...opts, level: "debug" } : opts;
return parent.child({ ...bindings, ...fields }, childOpts);
},
});
// Log only what the request log cannot tell you: slow requests, by route template.
app.addHook("onResponse", async (request, reply) => {
if (reply.elapsedTime < env.SLOW_REQUEST_MS) return;
request.log.warn(
{
route: request.routeOptions.url,
httpRequest: {
requestMethod: request.method,
requestUrl: request.routeOptions.url,
status: reply.statusCode,
latency: `${(reply.elapsedTime / 1000).toFixed(3)}s`,
},
},
"slow request",
);
});Why it is shaped this way:
childLoggerFactoryruns once per request with the raw Node request, so the trace fields are computed once and carried by everyrequest.logcall. It is a server option in the Fastify reference.httpRequest.latencyis a duration string such as"1.234s", not a number.requestUrlcarries the route template (/orders/:id), not the raw URL, which keeps IDs and token-bearing query strings out of the log and makes grouping by route possible.- Deterministic debug sampling. The decision is a function of the trace ID. Because the trace ID propagates to downstream services, a request sampled at the edge is sampled everywhere, so you get a complete debug story for a small share of requests instead of the same share of lines from each service, which is useless for reconstructing anything.
disableRequestLoggingis the Fastify 5 option. It removes two info lines per request, which on most services is most of the application log volume.
What I actually query during an incident is short: a request ID from an error report, expanded into its trace across services; an event identity followed through the async pipeline to answer "did the side effects run?"; instance-age filters to separate cold-start oddities from steady-state bugs; and error groupings by route and version to ask whether the last deploy caused it. Each of those works because the fields were attached automatically, back when nobody was under pressure.
Agree on what each level means#
Levels only save money if everyone uses them the same way. The table I would put in a backend conventions document:
| Level | Use it for | In production |
|---|---|---|
error | A request or job failed and someone may need to act | Always on, alerted |
warn | Degraded but handled: retries, fallbacks, slow requests | Always on, reviewed weekly |
info | Business events for an incident timeline, at most a few lines per request | On |
debug | Internal state for bug-fixing | Sampled per trace only |
trace | Loop bodies, payload dumps | Off |
My alerting follows from this table. I alert on symptoms a user can feel, such as an elevated error rate on a route or an async pipeline processing nothing during active hours, and never on a single line that a retry will absorb thirty seconds later. When an alert fires and needs no action, it gets one of two fixes: the threshold moves or the log level drops. A pager that is wrong twice a week gets ignored on the night it is right, and with a team of one there is no second reader to catch what alarm fatigue missed.
Two rules make it stick. An expected client error (a 404, a validation failure) is not an error; it is already in the request log with its status. And a caught exception is logged once, where it is handled, not at every layer it passes through.
Also decide what is never logged: request and response bodies, credentials of any kind (query strings are the usual leak, hence the route template), and direct identifiers when an internal ID will do. Cloud Run's request log includes the client IP and user agent, and an IP address is usually personal data under GDPR, so retention and access on that bucket matter even if your code never logs an IP. roles/logging.viewer grants access to everything the application writes, so treat log access as data access.
Exclude noise before it is billed#
At the time of writing, Cloud Logging charges per GiB ingested into the _Default bucket and user-defined buckets after a monthly free allotment of 50 GiB per project, at $0.50 per GiB, and the first 30 days of retention are included. Entries dropped by an exclusion filter never reach a bucket and are not charged. The _Required bucket holds audit logs, is free and cannot be changed.
Exclusion filters live on sinks. Two worth adding first for Cloud Run are health checks and a sample of successful request logs:
# Health checks: no value at all.
gcloud logging sinks update _Default \
--add-exclusion='name=drop-health-checks,filter=resource.type="cloud_run_revision" AND httpRequest.requestUrl:"/healthz"'
# Keep 10% of successful request logs; keep every 4xx and 5xx.
gcloud logging sinks update _Default \
--add-exclusion='name=sample-ok-requests,filter=resource.type="cloud_run_revision" AND logName:"run.googleapis.com%2Frequests" AND httpRequest.status<400 AND sample(insertId, 0.9)'The sample() function matches a fraction of entries based on a hash of the field, so in an exclusion filter sample(insertId, 0.9) drops about 90% and keeps about 10%. gcloud parses the exclusion value as comma-separated key-value pairs, so a filter that contains a comma needs the console or a config file instead.
Dropping 90% of request logs does not lose the numbers. Cloud Run's request count and latency metrics come from the platform, not from your logs. And the log-based metrics documentation states that user-defined log-based metrics are calculated from both included and excluded logs, so a counter you define keeps counting every matching entry after the line itself is excluded.
Turn recurring searches into metrics#
A log query you run every morning should be a metric. Counting events is far cheaper to keep than the lines:
gcloud logging metrics create payment_declined \
--description="Payments declined by the provider" \
--log-filter='resource.type="cloud_run_revision" AND jsonPayload.event="payment.declined"'A metric only counts entries received after it is created; it is not back-filled. User-defined log-based metrics are billed as Cloud Monitoring ingestion, so the cost depends on time series count. The expensive mistake is a label with unbounded values such as a user ID, which creates one time series per user. Keep labels to a small fixed set.
Keep long retention for a narrow slice#
The _Default bucket keeps logs for 30 days and that is included in the ingestion price, so shortening retention does not reduce the bill. Lengthening it is billed beyond 30 days, so extend it only on a user-defined bucket that receives a narrow slice, such as security-relevant application events:
gcloud logging buckets create security-events --location=global --retention-days=365
gcloud logging sinks create security-events \
logging.googleapis.com/projects/my-project/locations/global/buckets/security-events \
--log-filter='jsonPayload.category="security"'Routing an entry to two buckets means paying to ingest it twice. If those events also land in _Default, add an exclusion there or accept the duplicate cost knowingly.
What can still go wrong#
- Exclusions are invisible at incident time. The dropped 90% is not in Logs Explorer when you need it. Keep every non-2xx request and every application
warnand above, and review the filters whenever you add one. - Sampling hides rare paths. A bug that affects one request in ten thousand will almost never appear in 1% debug sampling. When chasing something specific, raise the rate for one service with an environment variable and a redeploy, then lower it.
- Large lines. A log entry has a size limit of about 256 KiB that cannot be raised, and an entry near it is almost certainly a payload dump. Cap array lengths and string sizes in serializers.
- Formatter drift. If the logger module is copied into ten services, one will eventually log
msgwith numeric levels. Publish it as a small internal package, or check the shape of a sample log line in each service's tests. console.logstill exists. In my own production code paths it is banned, because it cannot be filtered, priced, redacted or found when it matters; the one exception is a fatal startup error, where the logger may not exist yet. A library that writes plain text to stdout producestextPayloadentries with no severity and no trace. Most accept a logger, so passlogger.child({ component: "name" }).
If your total volume sits well inside the free allotment and is likely to stay there, skip the exclusions and sampling and keep the structured format and trace correlation, which cost nothing.
Related articles
All articles →Why a scheduled job can fail for weeks without anyone noticing, and the retry, idempotency and alerting setup that makes Cloud Scheduler failures loud.
13 min
Mocks pass while real systems fail. How I run a small Playwright suite against a deployed, no-traffic staging revision and gate promotion on the result.
11 min
Nine traps in Node config and Secret Manager on Cloud Run, from silent fallbacks to a trailing newline, each with the symptom, the cause and the fix.
13 min