Idempotent Pub/Sub consumers: exactly-once is your job
Pub/Sub will deliver some messages twice, and that is by design. Here is how to build consumers whose side effects happen once anyway, with working code.
Take a post with 101 likes whose counter reads 103. Nobody wrote a loop. The broker delivered two of the like.added events twice, which it is allowed to do, and the consumer treated every delivery as new. The same shape produces a receipt email sent twice or a partner system that books three orders from one webhook.
This article walks through the traps that produce those bugs on Google Cloud Pub/Sub, in the order a consumer usually meets them, with the fix for each. Most of it carries over to SQS, Kafka consumers or any other at-least-once queue. The running example is a push consumer on Cloud Run that maintains a likeCount on a post document. The habits come from a worker I run for a large-scale social platform I'm building, where a request handler validates, writes the one record that must exist and publishes an event, and everything else happens downstream.
Why a message arrives twice#
Pub/Sub's default guarantee is at-least-once delivery. Duplicates come from independent places, so closing one leaves the others open:
- Ack deadline expiry. If the consumer does not acknowledge before the ack deadline, Pub/Sub assumes the message was lost and sends it again while the first attempt may still be running.
- Explicit retries. A push endpoint that returns a non-success status, or a pull client that nacks, puts the message back in the queue, even if the handler already did half its work.
- Crash after the effect, before the ack. The database write commits, the container is killed during a deploy, and the ack never goes out.
- Publisher retries. The next section.
- Replays. Seeking to a timestamp or snapshot redelivers everything after that point, deliberately.
The working assumption for everything below: the queue delivers messages, and whether the effect happens once is the consumer's job. The rule I hold my own handlers to is that each must leave the same end state whether it runs once or five times.
A publisher retry produces a new messageId#
If a publish call times out on the client although the server received it, the client library retries, and the topic can end up with two messages with identical payloads and different message IDs. Anything that dedupes on Pub/Sub's messageId cannot see this, because for Pub/Sub they are two messages.
The fix is an identifier generated before the publish call, outside any retry loop, and carried in the payload:
import { PubSub } from "@google-cloud/pubsub";
import { randomUUID } from "node:crypto";
const pubsub = new PubSub();
const topic = pubsub.topic("like-events");
export async function publishLikeAdded(postId: string, userId: string) {
const event = {
// Generated once, before any retry, so publisher retries carry the same ID.
eventId: randomUUID(),
type: "like.added",
postId,
userId,
occurredAt: new Date().toISOString(),
};
// The client library retries transient failures internally; eventId stays stable.
await topic.publishMessage({ json: event, attributes: { type: event.type } });
}If publishing has to be atomic with a database write, use a transactional outbox: write the event row in the same transaction as the business change and let a relay publish it. The outbox row ID is then the event ID, stable by construction.
A random event ID only identifies one emission. A user who double-clicks produces two distinct events for one intent. Where the domain has a natural key, use it instead: order:8812:receipt-email, like:{postId}:{userId}, invoice:2026-09:{customerId}. My rule is the natural key when the business says "one per X", the event ID when the event itself is the unit ("apply this +1").
Exactly-once delivery is narrower than its name#
Pub/Sub has an exactly-once delivery option on subscriptions. It is useful and it is limited:
- Pull subscriptions only, StreamingPull included. Push and export subscriptions (BigQuery, Cloud Storage) do not support it.
- Regional. The guarantee holds within one region; subscribers connecting from several regions can still see the same message twice.
- Keyed on message ID. After a successful ack the message is not redelivered. The publisher-retry duplicates from the previous section have different IDs and are delivered normally.
- Acks can fail, and you must check. With the feature on, an ack returns success or failure. In Node.js that is
message.ackWithResponse(). If the ack fails because the deadline expired, the message comes back, and your effect may already have run. - Higher latency. The documentation says publish-to-subscribe latency is significantly higher than without it.
Pub/Sub can promise not to hand you the same message ID after a successful ack. It cannot know whether your database commit happened before the process died. Treat the feature as a way to lower the duplicate rate. Moving a consumer from pull to push to save idle cost also silently removes it, which is one more reason the correctness has to live in your code.
The marker and the effect commit separately#
The usual dedupe design is "record that I handled this event ID, then do the work". If the marker and the effect are two writes, a crash between them either loses the effect (marker first) or repeats it (effect first). When the effect lives in the same database as the marker, put both in one transaction.
For a push subscription the HTTP status is the ack: 102, 200, 201, 202 or 204 acknowledges, anything else is a nack. The handler has three outcomes:
- Success or duplicate: return
204. - Transient failure (contention, a downstream timeout): return
503, so Pub/Sub retries with backoff. - Poison message (unparseable, fails validation): retrying will not help. Quarantine it somewhere a human can read, then return
204.
import Fastify from "fastify";
import { Firestore, FieldValue, Timestamp } from "@google-cloud/firestore";
const db = new Firestore();
const app = Fastify({ logger: true });
type PushBody = {
message: {
data?: string; // base64
attributes?: Record<string, string>;
messageId: string;
publishTime: string;
orderingKey?: string;
};
subscription: string;
deliveryAttempt?: number; // present when a dead-letter policy is set
};
type LikeAdded = { eventId: string; type: "like.added"; postId: string; userId: string };
class PoisonMessage extends Error {}
function parseLikeAdded(message: PushBody["message"]): LikeAdded {
if (!message.data) throw new PoisonMessage("empty data");
let body: unknown;
try {
body = JSON.parse(Buffer.from(message.data, "base64").toString("utf8"));
} catch {
throw new PoisonMessage("invalid JSON");
}
const e = body as Partial<LikeAdded>;
if (e.type !== "like.added" || !e.eventId || !e.postId || !e.userId) {
throw new PoisonMessage("schema mismatch");
}
return e as LikeAdded;
}
const CONSUMER = "like-counter";
// Must outlast everything that can redeliver: the subscription's retention
// (7 days by default) plus the longest replay window you would use.
const DEDUPE_TTL_MS = 14 * 24 * 60 * 60 * 1000;
async function applyOnce(evt: LikeAdded): Promise<"applied" | "duplicate" | "skipped"> {
const markerRef = db.collection("processedEvents").doc(`${CONSUMER}:${evt.eventId}`);
const postRef = db.collection("posts").doc(evt.postId);
return db.runTransaction(async (tx) => {
// Firestore transactions require all reads before any writes.
const [marker, post] = await Promise.all([tx.get(markerRef), tx.get(postRef)]);
if (marker.exists) return "duplicate";
// Record the event even when the post is gone, so a redelivery is still a no-op.
tx.create(markerRef, {
processedAt: FieldValue.serverTimestamp(),
expireAt: Timestamp.fromMillis(Date.now() + DEDUPE_TTL_MS),
});
if (!post.exists) return "skipped";
tx.update(postRef, { likeCount: FieldValue.increment(1) });
return "applied";
});
}
app.post<{ Body: PushBody }>("/pubsub/like-events", async (req, reply) => {
const { message, deliveryAttempt, subscription } = req.body;
const log = req.log.child({ messageId: message.messageId, deliveryAttempt, subscription });
let evt: LikeAdded;
try {
evt = parseLikeAdded(message);
} catch (err) {
// Poison: quarantine for inspection, then ack so it stops consuming retries.
await db.collection("quarantine").doc(`${CONSUMER}:${message.messageId}`).set({
reason: (err as Error).message,
message,
receivedAt: FieldValue.serverTimestamp(),
});
log.error({ err }, "poison message quarantined");
return reply.code(204).send();
}
try {
const outcome = await applyOnce(evt);
log.info({ eventId: evt.eventId, outcome }, "like event handled");
return reply.code(204).send();
} catch (err) {
log.warn({ err, eventId: evt.eventId }, "transient failure; asking Pub/Sub to retry");
return reply.code(503).send();
}
});
app.listen({ host: "0.0.0.0", port: Number(process.env.PORT ?? 8080) });The highlighted function carries the guarantee. Reading the marker, creating it and incrementing the counter commit atomically. If two deliveries of one event run concurrently, Firestore's optimistic concurrency retries one transaction, and on the retry it sees the marker and returns "duplicate". If the process dies before commit, nothing was written and the redelivery starts from scratch.
On Cloud Run you do not need to verify the push request's OIDC token in code. Deploy with --no-allow-unauthenticated, give the subscription a push service account, and grant it roles/run.invoker. Cloud Run rejects unauthenticated requests before the handler runs.
The same pattern in a relational database is a unique constraint:
CREATE TABLE processed_events (
consumer text NOT NULL,
event_id text NOT NULL,
processed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer, event_id)
);import type { PoolClient } from "pg";
export async function applyOncePg(client: PoolClient, consumer: string, eventId: string, effect: () => Promise<void>) {
await client.query("BEGIN");
try {
const res = await client.query(
"INSERT INTO processed_events (consumer, event_id) VALUES ($1, $2) ON CONFLICT DO NOTHING",
[consumer, eventId],
);
if (res.rowCount === 0) {
await client.query("ROLLBACK");
return "duplicate" as const;
}
await effect(); // must use the same client so it joins this transaction
await client.query("COMMIT");
return "applied" as const;
} catch (err) {
await client.query("ROLLBACK");
throw err;
}
}A Redis SET key 1 EX ttl NX guard is faster and cheaper, but it cannot be transactional with the effect. Set the key first and crash, and the retry skips an effect that never ran. Run the effect first and crash, and the retry repeats it. Choose the failure you can live with, and reserve Redis for effects where that is acceptable: cache warming, analytics fan-out, notifications you would rather drop than repeat.
The transactional guard costs one extra document read and one write per message, plus a TTL delete later. Firestore bills per operation, so 10 million messages a month means roughly 10 million extra reads and writes. That is usually small next to a double charge or a corrupted metric, but for very high-volume, low-stakes counters a Redis guard or batched deltas can be cheaper.
A dedupe key shared across consumers#
If two subscriptions on one topic both write processedEvents/{eventId}, whichever consumer runs first makes the other skip the event. Prefix every key with the consumer name, as CONSUMER does above. Each consumer needs its own record of what it has processed.
Effects that cannot join your transaction#
An email or a third-party API call cannot commit with your database. For these, claim the work with a lease, and pass the provider's own idempotency key where it has one (payment APIs such as Stripe accept an Idempotency-Key header; check what your email or webhook provider supports):
import { Firestore, FieldValue } from "@google-cloud/firestore";
const db = new Firestore();
type Claim = "claimed" | "done" | "busy";
export async function claim(key: string, leaseMs: number): Promise<Claim> {
const ref = db.collection("effects").doc(key);
return db.runTransaction(async (tx) => {
const snap = await tx.get(ref);
const now = Date.now();
if (!snap.exists) {
tx.create(ref, { status: "pending", leaseUntil: now + leaseMs, attempts: 1 });
return "claimed";
}
const d = snap.data()!;
if (d.status === "done") return "done";
if (d.leaseUntil > now) return "busy"; // someone else is working on it right now
tx.update(ref, { leaseUntil: now + leaseMs, attempts: FieldValue.increment(1) });
return "claimed"; // previous holder's lease expired; take over
});
}
export async function markDone(key: string) {
await db.collection("effects").doc(key).update({ status: "done", doneAt: FieldValue.serverTimestamp() });
}The handler calls claim("receipt-email:order:8812", 30_000). On "done" it returns 204. On "busy" it returns 503, so Pub/Sub backs off and retries. On "claimed" it sends the email with the same key as the provider's idempotency key, calls markDone, and returns 204.
One window remains. If the provider accepts the send and the process dies before markDone, the next holder sends again. Only the provider-side key closes it. Without one the window is small but not zero, and the design document should say so instead of claiming exactly-once. Refresh-on-read instead of a cron job uses the same claim idea for scheduled refreshes.
Counters have a simpler fix than leases: turn "increment" into "set membership". Make the source of truth a document per (postId, userId) with a deterministic ID, so a duplicate create is a no-op, and derive the count with an aggregation query or keep it with the guarded increment above.
A dedupe TTL shorter than the redelivery horizon#
A marker only protects you while it exists. Set a Firestore TTL policy on processedEvents.expireAt so the collection does not grow forever, but size the TTL against everything that can redeliver: the subscription's message retention duration plus any replay or seek window. The rule is that a marker must outlive the last moment its message can still be delivered. A 24-hour Redis TTL on a subscription with 7-day retention leaves a six-day gap in which an incident replay re-applies every effect.
TTL deletion is also not immediate. Expired documents are typically removed within about a day, so never write logic that depends on a marker vanishing at a precise moment.
Ordering keys redeliver runs#
With message ordering enabled, a nack or an expired deadline redelivers that message and the messages after it on the same key, including ones already acknowledged. One slow handler therefore produces a run of duplicates. A poison message also blocks every later message on its key until it is acknowledged or dead-lettered. If correctness depends on order, carry a version in the payload and reject stale writes (if incoming.version <= stored.version, skip), which tolerates both reordering and duplicates. In my own consumers the version or timestamp comes from the source, and a handler discards any update older than the state it would overwrite. I also treat the payload as a pointer rather than a fact: for anything security- or integrity-relevant the handler re-fetches the current record before acting, because a retry that runs minutes later may find the account restricted or the content deleted. Pub/Sub ordering keys: when ordering is worth the cost works through that trade-off in full.
Getting transient and poison backwards#
Returning 503 for a validation error spends all the delivery attempts, each with backoff, before the message reaches the dead-letter topic, and it pollutes your retry metrics. The opposite error is worse: acknowledging a message after a database timeout "because we logged it" is silent data loss. Classify at the handler: poison is quarantined and acked on the first attempt, anything transient is nacked, and when in doubt, nack and let the dead-letter topic catch what never succeeds.
The subscription that backs this handler needs a retry policy and a dead-letter topic:
gcloud pubsub subscriptions create like-events-counter \
--topic=like-events \
--push-endpoint="${URL}/pubsub/like-events" \
--push-auth-service-account="$PUSH_SA" \
--ack-deadline=60 \
--min-retry-delay=10s \
--max-retry-delay=600s \
--dead-letter-topic=like-events-dlq \
--max-delivery-attempts=10Three flags matter here. The ack deadline should exceed the handler's worst realistic latency, cold start included, and the service's request timeout should sit at or below it, so a slow request fails visibly instead of producing a background duplicate. The retry policy applies exponential backoff, without which Pub/Sub redelivers as fast as it can, which during a downstream outage becomes a retry storm. And max-delivery-attempts accepts 5 to 100 and is approximate.
Creating the dead-letter topic, its inspection subscription, the IAM grants that let Pub/Sub forward messages, the alert and the replay procedure are all covered in dead-letter queues: a small team's safety net. Skipping any of them leaves a dead-letter policy that silently drops or never fires.
The worst failure I have had in this area was not a duplicate at all. During an environment rebuild a couple of my topics came back without their subscriptions, and publishing to a topic with no subscribers is not an error, so every publisher reported success while the events went nowhere. Every component looked healthy, which is why it took me a few days to find. I now recreate environments from declarations rather than from memory, and I alert on the absence of expected work: a pipeline that processes zero events on a day users were active counts as an alarm even though nothing failed.
A hot key under a transactional guard#
Firestore transactions on the same document serialise. A viral post receiving thousands of likes per second causes transaction retries and latency, and the dedupe guard adds a second document to the contended transaction. The standard fixes are sharded counters, or aggregating in the consumer and flushing batches of deltas under one dedupe key per batch.
When the guard is not needed#
If the effect is already naturally idempotent (overwriting a document with the latest profile snapshot, recomputing a cache entry from source, setting a status to shipped), a dedupe store adds cost and contention and buys nothing. Model the write as "make it so" and skip the guard. That is why I prefer a set over an increment wherever the data allows it, and where a counter is genuinely needed I write a processed marker in the same atomic batch as the increment. Best-effort telemetry, where a duplicate disappears into the noise, is the same. Spend the machinery on effects that people or money can see: counters users look at, messages they receive, anything that moves funds or calls a partner.
Related articles
All articles →Ordering keys give per-key order and take away parallelism. Learn what they guarantee, how one failure stalls a key, and when versioned events are a better fix.
9 min
A dead-letter queue turns a stuck message into a ticket instead of an outage. How to set one up on Pub/Sub, alert on it, and replay safely after a fix.
13 min
Payment webhooks arrive twice, late and out of order. A handler that verifies, stores, acknowledges fast and applies effects idempotently, shown with Stripe.
11 min