Dead-letter queues: a small team's safety net

A dead-letter queue turns a stuck message into a ticket instead of an outage. How to set one up on Pub/Sub, alert on it, and replay safely after a fix.

Published

Updated

Reading time

13 min

A dead-lettered message is a symptom. Before building anything, it helps to know what the symptom can mean, because the right response differs: some of these messages should be replayed within minutes, some need a code fix first, and some should be discarded on purpose with a note saying why.

This article is a reference for that triage on Google Cloud Pub/Sub, followed by the setup that makes the symptom visible and the replay procedure that makes recovery a reviewed command instead of an improvised script. It assumes the consumer already guards against duplicates, as described in idempotent Pub/Sub consumers; that article links here for everything dead-letter related. I wrote the setup for a large-scale social platform I'm building, where I am the whole on-call rotation, so the design has to degrade in ways that can wait for me.

Why messages end up dead-lettered#

A message is forwarded to the dead-letter topic when it has been delivered the configured number of times without an ack. That count says nothing about why. The table is the reference I keep next to the alert.

CauseHow to tellReplay as is?Fix
Malformed payload (a producer shipped a schema change first, a null that should not exist, a hand-published test message)Every attempt fails in parsing or validation; the logs show the same error text per messageNoFix the consumer or the producer; discard or repair the messages
Reference to data that does not exist (deleted record, or the producer published before its own commit)Lookup fails with not-found; the record exists now, or never willOften yes, once the record is therePublish after commit (outbox), or retry not-found for a short window
Bug triggered by some inputs (Unicode edge case, overflow, timezone boundary)Most messages succeed; the failing ones share a traitNo, until fixedFix, deploy, then replay
Downstream rejection that will not change (a 400 or 422 from a third party)The provider's error body names the inputNoQuarantine on first attempt; correct the data
Outage longer than the retry budgetFailures cluster in a time window and the dependency is healthy againYes, immediatelySize attempts and backoff so a routine outage is absorbed
Forwarding or ack path broken (missing IAM, wrong topic name)No dead letters at all, while retries and logs keep growingNot applicableFix the bindings; see the setup below
Table 1— Causes of dead letters, how to recognise each, and whether replaying helps.

The first four should usually be caught by the consumer itself, which quarantines the message on the first attempt and returns success. The dead-letter queue is the backstop for what that classification missed, and the only defence against the fifth row, which no classification can catch because the message was healthy. It is also a different tool from app-level quarantine: quarantine stores context you chose, the broker DLQ catches crashes, timeouts and bugs in the quarantine code itself.

Setting up the dead-letter path#

The design: every subscription that triggers a side effect gets its own dead-letter topic, every dead-letter topic gets one pull subscription that exists only to hold messages, and replays go through a dedicated per-consumer replay topic, never the original topic. Per-consumer topics cost nothing while empty, and the alert then names the broken consumer without anyone reading attributes. My rule is that every push subscription gets one, with no exceptions. Not every subscription will need it, but the moment you need one and do not have it, the message is already gone, either acknowledged into the void or looping for ever.

One dead-letter topic, one inspection subscription and one replay topic per consumer. Only the broken consumer sees replayed messages.

The examples use an orders topic with an orders-fulfilment push subscription delivering to a Cloud Run service.

infra/dlq.shbash
PROJECT_ID="my-project"
CONSUMER="orders-fulfilment"
URL="$(gcloud run services describe fulfilment --region=europe-west1 --format='value(status.url)')"
PUSH_SA="pubsub-push@${PROJECT_ID}.iam.gserviceaccount.com"
 
# 1. The dead-letter topic and its inspection subscription.
gcloud pubsub topics create "${CONSUMER}-dlq"
gcloud pubsub subscriptions create "${CONSUMER}-dlq-inspect" \
  --topic="${CONSUMER}-dlq" \
  --message-retention-duration=31d \
  --expiration-period=never
 
# 2. The source subscription, with a retry policy and dead-lettering.
gcloud pubsub subscriptions create "${CONSUMER}" \
  --topic=orders \
  --push-endpoint="${URL}/pubsub/orders" \
  --push-auth-service-account="${PUSH_SA}" \
  --ack-deadline=60 \
  --min-retry-delay=10s \
  --max-retry-delay=600s \
  --dead-letter-topic="${CONSUMER}-dlq" \
  --max-delivery-attempts=10
 
# 3. The replay path: same endpoint, its own topic, same DLQ.
gcloud pubsub topics create "${CONSUMER}-replay"
gcloud pubsub subscriptions create "${CONSUMER}-replay" \
  --topic="${CONSUMER}-replay" \
  --push-endpoint="${URL}/pubsub/orders" \
  --push-auth-service-account="${PUSH_SA}" \
  --ack-deadline=60 \
  --min-retry-delay=10s \
  --max-retry-delay=600s \
  --dead-letter-topic="${CONSUMER}-dlq" \
  --max-delivery-attempts=10 \
  --expiration-period=never

Three flags matter more than they look.

--max-delivery-attempts accepts 5 to 100 and defaults to 5. With backoff between 10 seconds and 10 minutes, ten attempts spread over tens of minutes, though Pub/Sub decides the exact spacing. The count is approximate because forwarding is best-effort, so never build logic that expects exactly N attempts. Too few attempts turns a ten-minute dependency outage into a pile of dead letters. Too many hides poison: at 100 attempts a bad message can write an error log for most of a day before anyone hears about it.

--expiration-period=never on the inspection subscription is the one to remember. Subscriptions expire after 31 days of inactivity by default, and a healthy dead-letter subscription is never pulled. After a month of good behaviour it deletes itself, and the next dead letter is published to a topic with no subscriptions, where the documentation says messages are lost.

--message-retention-duration=31d is the maximum for a subscription and the upper bound on how long you have to notice and act.

Dead-lettering is performed by the Pub/Sub service agent, not your consumer's identity. It must publish to the dead-letter topic and acknowledge on each source subscription:

infra/dlq-iam.shbash
PROJECT_NUMBER="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')"
PUBSUB_AGENT="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"
 
gcloud pubsub topics add-iam-policy-binding "${CONSUMER}-dlq" \
  --member="$PUBSUB_AGENT" --role=roles/pubsub.publisher
 
for SUB in "${CONSUMER}" "${CONSUMER}-replay"; do
  gcloud pubsub subscriptions add-iam-policy-binding "$SUB" \
    --member="$PUBSUB_AGENT" --role=roles/pubsub.subscriber
done

Grant these on the individual topic and subscriptions rather than the project; least privilege in practice explains why. If either binding is missing, Pub/Sub cannot forward, so it keeps redelivering to the failing consumer: retries continue, the DLQ stays empty, and the alert never fires. That is the sixth row of the table, and it looks exactly like a healthy system. Check both bindings in CI with gcloud pubsub topics get-iam-policy instead of trusting memory.

Alerting on the inspection subscription#

Pub/Sub exports the metrics needed in Cloud Monitoring:

  • subscription/num_undelivered_messages: the backlog. On the inspection subscription, anything above zero means a message was dead-lettered and nobody has dealt with it.
  • subscription/oldest_unacked_message_age: seconds since the oldest backlog message was published. This protects against retention expiry.
  • subscription/dead_letter_message_count: messages forwarded from the source subscription, useful on a dashboard of which consumer produces dead letters and how fast.

These gauges are sampled every 60 seconds and can lag a couple of minutes, so do not expect a page within seconds. Two conditions on the inspection subscription:

monitoring/dlq-alert.yamlyaml
displayName: "DLQ: orders-fulfilment"
combiner: OR
severity: WARNING
documentation:
  mimeType: text/markdown
  content: |
    Messages were dead-lettered from `orders-fulfilment`.
    Runbook: inspect `orders-fulfilment-dlq-inspect`, fix, replay via `orders-fulfilment-replay`.
conditions:
  - displayName: "DLQ backlog above zero"
    conditionThreshold:
      filter: >-
        resource.type = "pubsub_subscription" AND
        resource.labels.subscription_id = "orders-fulfilment-dlq-inspect" AND
        metric.type = "pubsub.googleapis.com/subscription/num_undelivered_messages"
      comparison: COMPARISON_GT
      thresholdValue: 0
      duration: 300s
      aggregations:
        - alignmentPeriod: 300s
          perSeriesAligner: ALIGN_MAX
  - displayName: "Oldest dead letter older than 7 days"
    conditionThreshold:
      filter: >-
        resource.type = "pubsub_subscription" AND
        resource.labels.subscription_id = "orders-fulfilment-dlq-inspect" AND
        metric.type = "pubsub.googleapis.com/subscription/oldest_unacked_message_age"
      comparison: COMPARISON_GT
      thresholdValue: 604800
      duration: 300s
      aggregations:
        - alignmentPeriod: 300s
          perSeriesAligner: ALIGN_MAX
alertStrategy:
  autoClose: 86400s
terminalbash
gcloud monitoring policies create \
  --policy-from-file=monitoring/dlq-alert.yaml \
  --notification-channels="projects/${PROJECT_ID}/notificationChannels/CHANNEL_ID"

The first condition says something broke. The second is a reminder that nobody dealt with it, firing with three weeks of retention left. I would route the first to a chat channel and the second to email, because a week-old dead letter is a process failure, not an incident. I learned the need for the second kind of signal the hard way: when I reviewed the infrastructure of my own staging environment, I found dead-letter queues that had been collecting messages with no alerting on any of them. The mechanism worked perfectly and the process around it did not exist, so every parked message was a side effect some user action was still waiting on. A standing alert on the source subscription's oldest_unacked_message_age catches the opposite failure: a consumer that is down while messages pile up before they ever reach the DLQ.

CostWhere the cost of a DLQ actually sits

An empty dead-letter topic and subscription cost nothing, and a few thousand dead letters of a couple of kilobytes each are a rounding error in Pub/Sub pricing. The money goes elsewhere: ten delivery attempts means ten handler invocations per poison message, and every failed attempt writes logs. A poison message that retries for hours costs more in Cloud Run time and log ingestion than the DLQ ever will, which is another argument for quarantining known-bad input on the first attempt.

Reading a dead letter#

When the alert fires, look before touching anything. Pulling without --auto-ack leases the messages for the ack deadline and then returns them to the backlog:

Inspect dead lettersbash
gcloud pubsub subscriptions pull orders-fulfilment-dlq-inspect \
  --limit=10 --format=json \
  | jq '.[] | {
      id: .message.messageId,
      attempts: .message.attributes.CloudPubSubDeadLetterSourceDeliveryCount,
      source: .message.attributes.CloudPubSubDeadLetterSourceSubscription,
      published: .message.attributes.CloudPubSubDeadLetterSourceTopicPublishTime,
      body: (.message.data | @base64d)
    }'

When Pub/Sub forwards a message it wraps the original data and attributes in a new message with a new ID and adds attributes naming where it came from: CloudPubSubDeadLetterSourceDeliveryCount, CloudPubSubDeadLetterSourceSubscription, CloudPubSubDeadLetterSourceSubscriptionProject and CloudPubSubDeadLetterSourceTopicPublishTime. If the consumer logs the event ID from the payload on every attempt, one log query shows every failed attempt with its error, which is usually enough to place the message in a row of Table 1.

gcloud pull is not guaranteed to return the whole backlog even with a higher --limit, so for anything beyond a quick look use the replay tool below in dry-run mode, which pages through the subscription and prints a summary.

The ways replay goes wrong#

Replay is where a dead-letter queue can turn into a second incident, and the failure modes are specific:

  • Republishing to the original topic. The message is delivered to every subscription on that topic, not just the broken one, so healthy consumers process it again.
  • Dropping the original attributes. The consumer may route on an attribute, and a replayed message without it is handled differently.
  • Replaying while the bug is still deployed. The messages return to the DLQ after ten more attempts, and a loop around the tool turns that into an endless cycle.
  • Acking before the publish succeeds. A crash between the two loses the message.

The tool below avoids all four. It reads from the inspection subscription, publishes to the per-consumer replay topic with the producer's attributes intact, caps replays per message, and acknowledges only after the publish succeeded.

tools/replay-dlq.tsts
import { PubSub, v1 } from "@google-cloud/pubsub";
 
const MAX_REPLAYS = 3;
 
function originalAttributes(attrs: Record<string, string>) {
  // Keep the producer's attributes; drop what dead-lettering added.
  return Object.fromEntries(
    Object.entries(attrs).filter(([k]) => !k.startsWith("CloudPubSubDeadLetter")),
  );
}
 
async function main() {
  const [project, consumer, mode = "dry-run", limitArg = "100"] = process.argv.slice(2);
  const limit = Number(limitArg);
 
  const sub = new v1.SubscriberClient();
  const subscription = sub.subscriptionPath(project, `${consumer}-dlq-inspect`);
  const replayTopic = new PubSub({ projectId: project }).topic(`${consumer}-replay`);
 
  let handled = 0;
  let overLimit = 0;
 
  while (handled < limit) {
    const [res] = await sub.pull({ subscription, maxMessages: Math.min(50, limit - handled) });
    const received = res.receivedMessages ?? [];
    if (received.length === 0) break;
 
    for (const { ackId, message } of received) {
      if (!ackId || !message) continue;
      const attrs = message.attributes ?? {};
      const replays = Number(attrs["x-replay-count"] ?? "0");
 
      if (mode !== "apply" || replays >= MAX_REPLAYS) {
        if (replays >= MAX_REPLAYS) overLimit++;
        console.log(JSON.stringify({ id: message.messageId, replays, attrs }));
        // Leave it in the DLQ and make it visible again straight away.
        await sub.modifyAckDeadline({ subscription, ackIds: [ackId], ackDeadlineSeconds: 0 });
      } else {
        await replayTopic.publishMessage({
          data: Buffer.from(message.data as Uint8Array),
          attributes: { ...originalAttributes(attrs), "x-replay-count": String(replays + 1) },
        });
        // Ack only after the publish succeeded: a crash in between means a duplicate, never a loss.
        await sub.acknowledge({ subscription, ackIds: [ackId] });
      }
      handled++;
    }
  }
 
  console.error(`handled=${handled} mode=${mode} over-replay-limit=${overLimit}`);
  await sub.close();
}
 
main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Run npx tsx tools/replay-dlq.ts my-project orders-fulfilment for a dry run, and add apply 500 to replay up to 500 messages. The x-replay-count cap keeps a message that three fixes could not save in the DLQ, where a human has to read it. The duplicate that a crash can create between publish and ack is harmless only because the consumer is idempotent, and that is not optional. A staging incident showed me why: a database was unavailable for a while, a large retry backlog replayed all at once, and because the handlers were idempotent the replay was a non-event rather than a second incident of double-created records and double-counted stats. It matters again here, because some of these messages partly succeeded before failing, and a replay runs the handler from the top.

Triage when the alert fires#

In practice the sequence is short for me: a poison message at 3 a.m. parks itself in the dead-letter topic, I read about it over coffee, fix the handler and replay. It is written into the alert's documentation field, so whoever is paged does not rely on memory:

  1. Look, do not touch. Dry-run the replay tool, count the messages and group them by error.
  2. Place each group in Table 1. Search the consumer's logs for the event IDs and read the error on the last attempt.
  3. Outage: the dependency is back, replay now. Bug: fix and deploy first, because replaying into the same code just refills the DLQ. Bad data: the message is wrong and will never be right.
  4. Replay a small batch first. apply 5, watch the consumer logs, then the rest.
  5. For bad data, record and discard. Save the payloads to a bucket or ticket with the reason, then ack them. Discarding is a decision someone writes down, not a default.
  6. Close the loop. If the consumer should have quarantined these messages on the first attempt, add that classification so the next batch never arrives.

Edge cases that bite later#

Ordering keys. With message ordering, a failing message blocks later messages on its key until it is dead-lettered. After that, ordering continues past the gap, so the consumer sees message 5 before message 4 is replayed. If order matters, carry a version in the payload and reject stale writes. Pub/Sub ordering keys: when ordering is worth the cost covers this.

The DLQ as a work queue. A queue that always holds a few hundred messages trains everyone to ignore the alert. If a class of failure is expected, handle it in code (quarantine, skip, retry on its own schedule). The DLQ should be empty most days.

Personal data. Payloads can contain it, and retention is up to 31 days. Grant roles/pubsub.subscriber on the inspection subscription only to the people who triage, and keep dry-run output out of shared chat channels.

Messages worth nothing late. A presence heartbeat, a cache invalidation the next write supersedes, a metrics sample: dead-lettering these only creates work. Let them expire, or ack on failure and count failures in a metric. The same holds where the source of truth can regenerate every message, so rerunning the job is the recovery.

Proving the path works before you need it#

Run once per consumer, and again after any change to its subscription

Checklist · 0 / 5
  1. The Pub/Sub service agent has publisher on the dead-letter topic and subscriber on the source and replay subscriptions (to do)
  2. The inspection subscription uses --expiration-period=never and 31-day retention (to do)
  3. A deliberately malformed message published to staging reaches the DLQ after the configured attempts (to do)
  4. The backlog alert fires for that message and the runbook text is in the notification (to do)
  5. A dry run of the replay tool lists the message; apply 1 sends it to the consumer only (to do)
Share
All articles →