Structured logging on a budget

The first surprise on a small team's cloud bill is rarely compute. It's logs. A service that serves a few million requests a day, logs "incoming request" and "request completed" for each one, and prints a debug line inside a hot loop can produce more gigabytes of logs than it stores in its database. Meanwhile, when something does break, the logs that matter are plain strings with no severity, no request ID and no way to find the other lines from the same request. You pay for volume and get little value from it. This article is for engineers running Node services on Cloud Run who want logs that Cloud Logging parses properly, lines from one request that can be found together, and a bill that stays inside the free allotment for as long as possible. The examples use pino and Fastify, but the Cloud Logging parts apply to any language that can print JSON.

The constraints I design for#

• One line, one event, one JSON object. Anything a human might filter on is a field, not part of a sentence.
• Severity is real. An error has ERROR severity in Cloud Logging, not DEFAULT with the word "error" somewhere in the text.
• Every line from a request can be found from any other line. Given one error, I want the request log and every application line from that request, across services, with one click.
• Cost scales with problems, not with traffic. A healthy request should cost close to nothing to log. An unhealthy one can cost more.
• No personal data or secrets end up in a system with broad read access and 30-day retention.
• No vendor agent or sidecar. Cloud Run already ships stdout to Cloud Logging. I want to use that path, not add another one.

How Cloud Run turns stdout into log entries#

Cloud Run captures everything a container writes to stdout and stderr. According to the Cloud Run logging documentation, a line that is a single serialised JSON object becomes a structured entry with a jsonPayload; anything else lands in textPayload. A handful of JSON keys are treated specially and lifted into the LogEntry itself rather than left in the payload. The ones I use are listed in the special fields table:

Severity accepts the Cloud Logging names: DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT and EMERGENCY. pino's default output uses a numeric level (30 for info, 50 for error) and a msg key, neither of which Cloud Logging recognises. Out of the box, a pino line on Cloud Run shows up with severity DEFAULT, which means every severity filter and every error alert misses it.

Cloud Run also writes its own request log for every request, under the log name run.googleapis.com/requests, with httpRequest already filled in: status, latency, sizes, user agent and remote IP. That changes the design. You don't need an application access log at all.

Options#

Writing through the API deserves a second look, because it's what many tutorials show. It sends batched network requests from your process, which is extra work for your container, and if your service only gets CPU during requests, work left over after the response isn't guaranteed to finish. Printing to stdout hands the problem to the platform, which already solved it.

The decision#

I use pino writing JSON to stdout, configured once in a shared module, with four changes from the defaults: severity instead of level, message instead of msg, an ISO time, and no pid or hostname. The request's trace ID is bound into a child logger at the start of each request. Fastify's built-in request logging is off, because Cloud Run's request log already covers it. Cost control happens in three places: log levels in code, sampling per request, and exclusion filters on the _Default sink.

Implementation#

Step 1: a logger Cloud Logging understands#

import pino from "pino";
import { env } from "./config/env";
 
// pino level label -> Cloud Logging severity
const SEVERITY: Record<string, string> = {
  trace: "DEBUG",
  debug: "DEBUG",
  info: "INFO",
  warn: "WARNING",
  error: "ERROR",
  fatal: "CRITICAL",
};
 
export const logger = pino({
  level: env.LOG_LEVEL, // "info" in production
  messageKey: "message",
  base: undefined, // no pid/hostname: Cloud Run labels entries with service and revision
  timestamp: pino.stdTimeFunctions.isoTime,
  formatters: {
    level(label) {
      return { severity: SEVERITY[label] ?? "DEFAULT" };
    },
    log(object) {
      // Error Reporting looks for a stack trace in `stack_trace`, then `exception`, then `message`.
      const err = object.err as Error | undefined;
      return err?.stack ? { ...object, stack_trace: err.stack } : object;
    },
  },
  redact: {
    paths: [
      "req.headers.authorization",
      "req.headers.cookie",
      "*.password",
      "*.token",
      "*.email",
    ],
    censor: "[redacted]",
  },
});

The pino API docs are explicit about one constraint: formatters.level can't replace the numeric level when you use multiple transport targets, because pino needs the number to route lines between them. Writing to stdout with no transport, as here, is the fast path anyway. Resist the temptation to add pino-pretty in production. Pretty output isn't JSON, so Cloud Logging falls back to textPayload and you lose everything above. Use it locally by piping (node dist/server.js | pino-pretty).

Error Reporting's format rules check stack_trace, exception and message in that order. Copying err.stack into stack_trace means every log.error({ err }, "...") also shows up as a grouped error in Error Reporting, with no extra client library.

Redaction is covered in more depth in configuration and secrets in multi-service Node apps; the paths above are a starting point, not a complete list.

Step 2: bind the trace to every request#

Cloud Run populates the W3C traceparent header on incoming requests, and Google Cloud services still accept the older X-Cloud-Trace-Context header. To make Logs Explorer nest your lines under the request log, set logging.googleapis.com/trace to projects/PROJECT_ID/traces/TRACE_ID. Cloud Run doesn't set a project ID environment variable for you, so I pass it in at deploy time and validate it with the rest of the config.

import Fastify from "fastify";
import type { IncomingHttpHeaders } from "node:http";
import { logger } from "./logger";
import { env } from "./config/env";
 
const TRACEPARENT = /^00-([0-9a-f]{32})-[0-9a-f]{16}-([0-9a-f]{2})$/;
 
function traceContext(headers: IncomingHttpHeaders) {
  let traceId: string | undefined;
  let sampled = false;
  const tp = headers["traceparent"];
  const legacy = headers["x-cloud-trace-context"];
  const m = typeof tp === "string" ? TRACEPARENT.exec(tp) : null;
  if (m) {
    traceId = m[1];
    sampled = (parseInt(m[2], 16) & 1) === 1;
  } else if (typeof legacy === "string") {
    traceId = legacy.split("/")[0]; // TRACE_ID/SPAN_ID;o=OPTIONS
    sampled = legacy.includes(";o=1");
  }
  return { traceId, sampled };
}
 
// Deterministic per trace: every service keeps or drops the same requests.
function keepDebug(traceId: string | undefined, rate: number) {
  if (!traceId) return false;
  return parseInt(traceId.slice(-8), 16) / 0xffffffff < rate;
}
 
export const app = Fastify({
  loggerInstance: logger,
  disableRequestLogging: true, // Cloud Run's request log already records every request
  childLoggerFactory(parent, bindings, opts, rawReq) {
    const { traceId, sampled } = traceContext(rawReq.headers);
    const fields = traceId
      ? {
          "logging.googleapis.com/trace": `projects/${env.GCP_PROJECT_ID}/traces/${traceId}`,
          "logging.googleapis.com/trace_sampled": sampled,
        }
      : {};
    const childOpts = keepDebug(traceId, env.DEBUG_SAMPLE_RATE) ? { ...opts, level: "debug" } : opts;
    return parent.child({ ...bindings, ...fields }, childOpts);
  },
});
 
// Only log requests that tell you something the request log doesn't: slow ones, with the route template.
app.addHook("onResponse", async (request, reply) => {
  if (reply.elapsedTime < env.SLOW_REQUEST_MS) return;
  request.log.warn(
    {
      route: request.routeOptions.url,
      httpRequest: {
        requestMethod: request.method,
        requestUrl: request.routeOptions.url,
        status: reply.statusCode,
        latency: `${(reply.elapsedTime / 1000).toFixed(3)}s`,
      },
    },
    "slow request",
  );
});

A few details here are worth explaining:

• childLoggerFactory runs once per request with the raw Node request, so the trace fields are computed once and carried by every request.log call. Fastify documents it as a server option in its reference.
• httpRequest.latency is a duration string such as "1.234s", not a number, and requestUrl uses the route template (/orders/:id) rather than the raw URL. That keeps IDs and query strings, which can carry tokens, out of the log, and it makes grouping by route possible.
• Deterministic sampling. The debug decision is a function of the trace ID. Because the trace ID propagates to downstream services, a request sampled at the edge is sampled everywhere, and you get a complete debug story for 1% of requests instead of 1% of lines from each service, which is useless.
• disableRequestLogging is the Fastify 5 option. It removes two info lines per request, which on most services is most of the application log volume.

Step 3: level discipline#

Levels only save money if everyone uses them the same way. This is the table I put in the backend conventions document:

Two rules make the table stick. An expected client error (a 404, a validation failure) is not an error; it's already in the request log with its status code. And a caught exception is logged once, at the boundary where it's handled, not at every layer it passes through on the way up.

Step 4: exclude noise before it's billed#

At the time of writing, Cloud Logging charges $0.50 per GiB for logs stored in the _Default bucket and user-defined buckets, after 50 GiB per project per month, and that includes 30 days of retention. Entries that an exclusion filter drops never reach a bucket and aren't charged. The _Required bucket, which holds audit logs, is free and can't be changed.

Exclusion filters live on sinks. For Cloud Run, the two I add first are health checks and a sample of successful request logs:

# Health checks: no value at all.
gcloud logging sinks update _Default \
  --add-exclusion='name=drop-health-checks,filter=resource.type="cloud_run_revision" AND httpRequest.requestUrl:"/healthz"'
 
# Keep 10% of successful request logs; keep every 4xx and 5xx.
gcloud logging sinks update _Default \
  --add-exclusion='name=sample-ok-requests,filter=resource.type="cloud_run_revision" AND logName:"run.googleapis.com%2Frequests" AND httpRequest.status<400 AND sample(insertId, 0.9)'

The sample() function matches a fraction of entries based on a hash of the field, so sample(insertId, 0.9) in an exclusion filter drops about 90% and keeps about 10%. Note the quoting: gcloud parses the value as comma-separated key-value pairs, so a filter that contains a comma needs a different approach, such as creating the exclusion from the console or from a config file.

Dropping 90% of request logs sounds like losing data, but it isn't losing the numbers. Cloud Run's built-in request count and latency metrics come from the platform, not from your logs. And user-defined log-based metrics are calculated from both included and excluded entries, so a counter you define keeps counting every request even after the log line itself is dropped.

Step 5: log-based metrics instead of log searches#

When I catch myself running the same log query every morning, it should be a metric. Counting events is far cheaper to keep than the lines themselves:

gcloud logging metrics create payment_declined \
  --description="Payments declined by the provider" \
  --log-filter='resource.type="cloud_run_revision" AND jsonPayload.event="payment.declined"'

The metric only counts entries received after it's created; it isn't back-filled. At the time of writing, user-defined log-based metrics are billed as Cloud Monitoring ingestion: 8 bytes per point for a counter and 80 bytes for a distribution, with the first 150 MiB per billing account free. A counter written once a minute is a few hundred kilobytes a month per time series. The expensive mistake is a label with unbounded values, like a user ID, which creates one time series per user.

Step 6: retention and buckets#

The _Default bucket keeps logs for 30 days, and those 30 days are included in the ingestion price. Shortening retention doesn't reduce the bill. Lengthening it costs $0.01 per GiB per month beyond 30 days at the time of writing, so I only extend it on a user-defined bucket that receives a narrow slice of logs, such as security-relevant application events:

gcloud logging buckets create security-events --location=global --retention-days=365
 
gcloud logging sinks create security-events \
  logging.googleapis.com/projects/my-project/locations/global/buckets/security-events \
  --log-filter='jsonPayload.category="security"'

Remember that routing an entry to two buckets means paying for it twice. If the security events also land in _Default, add an exclusion there, or accept the duplicate cost knowingly.

What not to log#

Structured logs make it easy to log whole objects, which makes it easy to log things you shouldn't. My rules:

• No request or response bodies by default. They contain whatever users typed. Log IDs and sizes, not contents.
• No credentials of any kind. Authorization headers, cookies, API keys, session tokens, signed URLs and password reset links. Query strings are the usual leak, which is one more reason to log the route template.
• No direct identifiers when an internal ID will do. Log userId, not an email address or phone number. When support needs to find a user's requests, they can look up the ID.
• Know what the platform logs for you. Cloud Run's request log includes the client IP address and user agent. Under GDPR an IP address is usually personal data, so the retention and access controls on that bucket matter even if your code never logs an IP itself.
• Restrict who can read logs. roles/logging.viewer doesn't grant access to data access audit logs, but it does grant access to everything your application writes. Treat log access as data access.

Trade-offs and failure modes#

Exclusions are invisible at incident time. When you open Logs Explorer during an incident, the dropped 90% isn't there. That's fine for successful requests and a real problem if an exclusion filter is too broad. I keep every non-2xx and every application warn and above, and I review exclusion filters whenever I add one.

Sampling hides rare paths. A bug that affects one request in ten thousand will almost never appear in 1% debug sampling. When I'm chasing something specific, I raise the sample rate for one service with an environment variable and a redeploy, then lower it again.

Large lines. A log entry has a size limit of about 256 KiB that can't be raised, and anything near it is almost certainly a payload dump. Don't count on an oversized line arriving intact. Cap array lengths and string sizes in serializers instead.

Formatter drift. The logger module is copied into ten services and one of them still logs msg with numeric levels. I publish it as a small internal package, or at least check a sample log line's shape in each service's tests.

console.log still exists. A third-party library that writes plain text to stdout produces textPayload entries with no severity and no trace. Most libraries accept a logger; pass them logger.child({ component: "name" }).

Checklist#

When not to do this#

If your total log volume is well inside the free allotment and likely to stay there, exclusion filters and sampling add complexity with no saving; keep the structured format and trace correlation, and skip the rest. And if you already run OpenTelemetry end to end with a collector, send logs through it alongside traces and metrics rather than maintaining a separate path. The formatter and trace binding here are for teams whose logging pipeline is just stdout and Cloud Logging, which, for a small team on Cloud Run, is usually the right pipeline to have.