Pub/Sub ordering keys: when ordering is worth the cost
Ordering keys give per-key order and take away parallelism. Learn what they guarantee, how one failure stalls a key, and when versioned events are a better fix.
Put the ordering in the data, not in the broker. Give every event for an entity a version, apply it only if it is newer than what you have stored, and Pub/Sub can go on doing what it is best at: moving independent messages in parallel. Ordering keys are for the narrower case where the sequence itself carries meaning and a version check cannot express it. Turning them on changes the delivery model underneath you, and the price is paid in stalls, replays and operational work, not in a line on the bill.
The rest of this article makes the case, takes the best counter-argument seriously, and then shows how to run ordering keys properly for the cases where they do earn their place. It assumes consumers already tolerate duplicates, as in idempotent Pub/Sub consumers.
What the flag actually changes#
The behaviour below comes from the Pub/Sub documentation on message ordering and on publishing.
- Order is per key, not per topic. Messages sharing an ordering key are delivered in publish order. Messages without a key are unordered. A key can be up to 1 KB.
- Ordering is a subscription property, set at creation and immutable afterwards.
- Publishes must stay in one region for the guarantee to hold, so publishers outside Google Cloud or spread over regions should use a locational endpoint. Subscribers can connect to any region.
- Publish throughput is limited to 1 MBps per ordering key.
- Delivery is serialised per key. Pull subscriptions have one batch outstanding per key; push has one message outstanding per key, which the documentation flags as a poor fit for frequently published keys.
- Redelivery replays the tail. Redelivering one message redelivers all later messages on that key, including ones you already acknowledged.
- A failed publish blocks the key. Queued messages for that key in the publisher fail, as do later publishes, until you resume publishing.
- Dead-lettering can break order.
Read that list as a description of a different system. Without ordering, Pub/Sub is a pool of independent messages any worker can take in any sequence. With ordering, each key is a single-file queue, and the broker behaves more like a log partition.
Why a version number usually beats a lane#
The question to ask first is what actually breaks when two events swap places. The common answer is a stale write overwriting a newer one: "account closed" is processed, then the older "account updated" arrives and resurrects the record. A version solves exactly that, and it solves duplicates with the same mechanism.
| Approach | Order within a key | Parallelism | Effect of a poison message | Extra work |
|---|---|---|---|---|
| Events are full snapshots, last write wins | Not needed | Full | Isolated to that message | None |
| Version in each event, conditional write | Stale events are discarded | Full | Isolated | A version column |
| Consumer re-reads current state from the source | Not needed | Full | Isolated | A source that is always readable |
| Ordering keys on the subscription | Yes, per key | None within a key | Blocks later messages on the key | Resume handling, stall alerts |
| A log-style broker with partitions | Yes, per partition | Limited by partitions | Blocks the partition | A different system to run |
The version-checked consumer is short. Each event for a customer carries a monotonically increasing version (a counter you control or a database sequence), and the write is conditional:
import { PubSub, type Message } from "@google-cloud/pubsub";
import type { Pool } from "pg";
declare const pool: Pool;
const subscription = new PubSub().subscription("customer-events", {
flowControl: { maxMessages: 20 },
});
subscription.on("message", async (message: Message) => {
const event = JSON.parse(message.data.toString()) as {
customerId: string;
version: number;
status: string;
};
try {
// Apply only if newer. A replayed, duplicate or reordered event changes nothing.
await pool.query(
`INSERT INTO customer_state (customer_id, version, status)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id) DO UPDATE
SET version = EXCLUDED.version, status = EXCLUDED.status
WHERE customer_state.version < EXCLUDED.version`,
[event.customerId, event.version, event.status],
);
message.ack();
} catch (err) {
console.error(JSON.stringify({ msg: "apply failed", error: String(err) }));
message.nack();
}
});Nothing here depends on ordering, so a poison message affects one message, and every consumer instance can work on the same customer at once.
The best argument for keys: deltas and state machines#
The version check has a limit, and it is the honest reason ordering keys exist. It works when each event carries enough to describe the state on its own. It discards a late event, so it is only correct when the discarded event does not matter. Two kinds of data break that:
- Deltas. A ledger of deposits and withdrawals cannot drop the older event and keep the newer one, because the balance needs both. You could buffer out-of-order events until the gap fills, but that is a reimplementation of ordering inside your consumer.
- State machines whose valid next step depends on the previous one, such as a document moving through draft, review and published, where "published" before "review" is a bug rather than a stale write.
For these, a per-key lane is the right tool, and the cost is worth paying. Even then, keep the version in the payload as a second guard: it makes the replayed tail harmless, which the next section shows is a certainty, not a risk. Key by the smallest entity that needs order (one customer, not all customers), and create the ordered subscription separately from any consumers that do not care about order, since ordering is set per subscription.
Running keys properly#
Enable it on the subscription#
gcloud pubsub topics create customer-events
gcloud pubsub subscriptions create customer-events-ordered \
--topic=customer-events \
--enable-message-ordering \
--ack-deadline=60A second subscription on the same topic without the flag receives the same messages unordered, which suits an analytics sink. Because the property is immutable, turning ordering on later means creating a new subscription and migrating.
Publish with a key, and treat a failure as a sequence problem#
In the Node.js client, ordering is enabled on the topic handle with messageOrdering: true and each message carries an orderingKey:
import { PubSub } from "@google-cloud/pubsub";
const pubsub = new PubSub({
apiEndpoint: "europe-west4-pubsub.googleapis.com:443",
});
// Cache the topic object and reuse it: ordering state lives in the publisher.
const topic = pubsub.topic("customer-events", { messageOrdering: true });
export async function publishEvent(customerId: string, event: Record<string, unknown>) {
const message = {
data: Buffer.from(JSON.stringify(event)),
orderingKey: customerId,
};
try {
return await topic.publishMessage(message);
} catch (err) {
// A failed publish with an ordering key blocks that key in this publisher,
// including future publishes, until publishing is resumed.
topic.resumePublishing(customerId);
throw err; // let the caller decide whether to retry
}
}Resuming is not retrying. The failed message and everything queued behind it on that key were rejected. If you resume and retry only the one that failed, you have published out of order. The caller must re-send everything it had in flight for that key, in the original order. The simplest sound design is a per-key outbox: write the event in the same transaction as the change and let a relay publish in sequence order per key. Several publisher instances each keep their own state per key, so the guarantee holds only if there is one logical writer per key at a time.
What a stall looks like#
A nack or a missed ack deadline does not redeliver one message, it redelivers the message and its tail. A handler that fails every time on one message loops on that key, and the lane behind it does not move. Three defences, in the order I would add them:
- Fail only for transient reasons. A structurally invalid message should be recorded and acknowledged, not nacked, or it blocks its lane forever.
- Alert on the age of the oldest unacknowledged message. The documentation suggests watching
subscription/oldest_unacked_message_agefor hot keys. An age that climbs while other keys keep flowing means one lane is stuck. This catches a looping handler that is quiet in the logs. - Dead-letter with the caveat understood. A dead-letter topic lets a poison message leave the lane, but order may not be preserved when messages are written to it. Treat it as a place to repair and replay by hand, with the version check making the replay safe. Dead-letter queues: a small team's safety net covers the setup and the replay procedure.
Replays after a nack are also additional deliveries, and they count towards what you pay for, so a looping poison message generates billable traffic until someone looks at the age metric.
Three limits worth stating in numbers#
Per-key throughput is bounded by handler latency. Because a key is processed serially, take a handler that needs 200 milliseconds per message (a hypothetical figure). One key then tops out near five messages a second, however many subscriber instances you add. More instances help only across keys. The usual responses are a faster handler or a finer key, and both cost engineering time. One entity producing most of the traffic makes that entity the throughput ceiling for the system.
Exactly-once delivery and ordering interact badly. The exactly-once documentation says Pub/Sub expects acknowledgements in order when the two are combined, and that out-of-order acks cause temporary failures and lower client throughput. I do not combine them by default: with a version check in the consumer, duplicates are already harmless.
Order within a key says nothing about order across keys. Two entities that interact still need a data-level rule, such as a version check or a transaction.
A recommendation for each situation#
- Events are snapshots, or a newer one makes the older irrelevant: no ordering. Last write wins, or a version check.
- Events are deltas or steps of a state machine, with moderate traffic per entity: ordering keys, keyed by the smallest entity, with a version in the payload, an age alert and a tested resume path.
- One entity dominates traffic, or the handler is slow: do not use ordering keys. Redesign so the entity's events can be merged or applied commutatively.
- You need a replayable, strictly ordered log with consumers that track their own position: you want a log-style system, not a queue with an ordering mode.
- Your consumers are not idempotent yet: fix that first. Ordering makes replays more frequent, not less.
Related articles
All articles →A dead-letter queue turns a stuck message into a ticket instead of an outage. How to set one up on Pub/Sub, alert on it, and replay safely after a fix.
13 min
Pub/Sub will deliver some messages twice, and that is by design. Here is how to build consumers whose side effects happen once anyway, with working code.
13 min
A practical split for small teams: coarse per-client limits at the edge, per-user and per-cost limits in the service, and the 429 behaviour both must share.
11 min