Designing idempotent webhooks from payment providers

Payment webhooks arrive twice, late and out of order. A handler that verifies, stores, acknowledges fast and applies effects idempotently, shown with Stripe.

Published

Updated

Reading time

11 min

An order is paid, then refunded, and the webhooks arrive in the opposite order: charge.refunded first, payment_intent.succeeded a moment later. A handler that treats each webhook as an ordered command sees the refund against an order that is still pending, finds no valid transition, ignores it and marks the event done. Then the payment event arrives and moves the order to paid. The customer has their money back and the system says they paid.

That is one trap among several, all with the same root: a webhook is an unreliable, unordered, at-least-once notification, and handlers keep treating it as a reliable, ordered, exactly-once command. This article goes through the traps in the order a handler meets them, with Stripe as the worked example and Node.js, Express and Firestore for the code. The design applies to any provider that retries on failure. It builds on idempotent Pub/Sub consumers and dead-letter queues, because once an event is acknowledged the rest of the pipeline is an ordinary queue consumer problem.

What Stripe promises and what it leaves to you#

From Stripe's webhook documentation:

  • Delivery is retried. In live mode for up to three days with exponential back-off; sandbox events are retried three times over a few hours. A non-2xx response, a timeout, a redirect or a TLS failure all count as failures.
  • Duplicates happen. The same event can arrive more than once, and in some cases two separate Event objects are generated for one change. The documentation suggests combining the ID of data.object with the event type to spot those.
  • Order is not guaranteed. Creating a subscription can emit a subscription event, an invoice event and a charge event, in any order. created is recorded in whole seconds, so events can share a timestamp, and the documentation says not to use it to decide order.
  • Answer fast. Return a 2xx before any complex logic that could time out.
  • Signatures need the raw body. Each event carries a Stripe-Signature header; verification needs the exact bytes sent.
  • Manual replay exists. The Dashboard can resend an event up to 15 days after creation, and stripe events resend <event_id> --webhook-endpoint=<endpoint_id> works up to 30 days. Resending does not cancel the automatic retries.

At-least-once delivery, no ordering, a retry window measured in days. Everything else is design. The handler below does four things, in this order, and nothing else: verify the signature against the raw body, record the event durably keyed by its ID, enqueue the work and acknowledge, then let a worker apply the effect guarded by the real state of the business object.

Verify, record, enqueue and acknowledge; apply effects later, guarded by state.

A body parser that changes the bytes#

The most common first failure is a global JSON parser mounted before the webhook route. It re-serialises the body, the signature no longer matches, and every event is rejected with a 400. Mount the raw parser on the webhook route before any express.json():

webhook.tsts
import express from "express";
import Stripe from "stripe";
 
const stripe = new Stripe(process.env.STRIPE_API_KEY ?? "");
const endpointSecret = process.env.STRIPE_WEBHOOK_SECRET ?? "";
 
export const app = express();
 
app.post(
  "/webhooks/stripe",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    const signature = req.header("stripe-signature");
    if (!signature) {
      res.status(400).send("missing signature");
      return;
    }
 
    let event: Stripe.Event;
    try {
      event = stripe.webhooks.constructEvent(req.body, signature, endpointSecret);
    } catch (err) {
      res.status(400).send("invalid signature");
      return;
    }
 
    await acceptEvent(event, res);
  },
);
 
// Every other route can use the JSON parser.
app.use(express.json());

The official libraries default to a five-minute tolerance between the signed timestamp and the current time, which limits replay of a captured request. The documentation warns against a tolerance of zero, which disables the recency check, and recommends NTP so your clock does not drift into rejecting genuine events. Each endpoint has its own signing secret, and test-mode and live-mode secrets differ, so a deployment pointing at the wrong one fails every request with a signature error. Read secrets from your secret store, as in configuration and secrets in Node services.

Doing the work inside the request#

If the handler does the business logic before it responds, a slow downstream call causes a timeout, the timeout causes a retry, and the retry causes a second execution. Stripe's documentation recommends asynchronous queues for exactly this reason: a spike, such as every subscription renewing at the start of the month, can overwhelm endpoints that process synchronously.

So the request path only records the event and enqueues. Firestore's create() fails if the document exists, which turns "have I seen this event" and "remember it" into one atomic operation:

accept-event.tsts
import { Firestore, Timestamp } from "@google-cloud/firestore";
import { CloudTasksClient } from "@google-cloud/tasks";
import type { Response } from "express";
import type Stripe from "stripe";
 
const db = new Firestore();
const tasks = new CloudTasksClient();
 
const ALREADY_EXISTS = 6; // gRPC status code
 
export async function acceptEvent(event: Stripe.Event, res: Response) {
  const ref = db.collection("webhookEvents").doc(event.id);
 
  try {
    await ref.create({
      type: event.type,
      payload: JSON.stringify(event),
      receivedAt: Timestamp.now(),
      status: "received",
    });
  } catch (err) {
    if ((err as { code?: number }).code === ALREADY_EXISTS) {
      // Duplicate delivery. The first one is already queued or done.
      res.status(200).send("duplicate");
      return;
    }
    // Storage failed. Do not acknowledge: let the provider retry.
    res.status(500).send("store failed");
    return;
  }
 
  try {
    await enqueueProcessing(event.id);
  } catch (err) {
    // The event is stored, so a reconciler can pick it up. Still ask for a retry.
    res.status(500).send("enqueue failed");
    return;
  }
 
  res.status(200).send("ok");
}
 
async function enqueueProcessing(eventId: string) {
  const parent = tasks.queuePath(
    process.env.GCP_PROJECT ?? "",
    process.env.TASKS_LOCATION ?? "",
    "webhook-processing",
  );
  await tasks.createTask({
    parent,
    task: {
      name: `${parent}/tasks/${eventId}`,
      httpRequest: {
        httpMethod: "POST",
        url: process.env.WORKER_URL ?? "",
        headers: { "Content-Type": "application/json" },
        body: Buffer.from(JSON.stringify({ eventId })).toString("base64"),
        oidcToken: { serviceAccountEmail: process.env.TASKS_INVOKER_SA ?? "" },
      },
    },
  });
}

Three details carry weight. The duplicate path returns 200, not an error: from the provider's side the event was delivered, and an error only provokes more retries. If storing fails, return 500 so the provider retries; nothing is lost. If enqueueing fails after the event was stored, also return 500, but understand the consequence: on the retry, create() reports a duplicate and skips the enqueue. That is a gap, and it is why the stored status matters. A small reconciler that finds events still in received after a few minutes and enqueues them closes it. Finally, using the event ID as the Cloud Tasks task name adds a second layer of deduplication, because a second task with the same name is rejected; check the Cloud Tasks documentation for how a name behaves after the task has run, since a deleted or executed task's name cannot be reused for a period.

Storing the payload is a deliberate choice. It lets you replay after fixing a bug without asking the provider, and it is your audit trail. It also contains customer data, so give the collection a retention period and put it on your deletion list; GDPR deletion in distributed systems explains why.

The out-of-order event that gets marked done#

This is the trap from the opening, and the obvious fix is the wrong one. A state machine with explicit transitions sounds right:

tsts
const ALLOWED = {
  pending: ["paid", "canceled"],
  paid: ["refunded"],
  refunded: [],
  canceled: [],
};
// if (!ALLOWED[current].includes(target)) return;  // "stale or out of order"

The comment is the bug. An early charge.refunded is not stale, it is premature. Ignoring it and then marking the event done throws away information that the later payment_intent.succeeded cannot reconstruct, and the order ends up paid after a full refund.

There are two sound fixes. One is to defer: if an event cannot apply yet, return an error from the worker so the queue retries it later, by which time the earlier event has usually landed. The other, which I prefer, is to stop trusting the payload for the decision. Stripe's own suggestion for disorder is to retrieve the missing objects through the API. The worker takes the event only as a hint that something changed, fetches the current PaymentIntent and its charge, derives the order status from that, and uses the transition table purely as a guard against moving backwards:

worker.tsts
import { Firestore } from "@google-cloud/firestore";
import Stripe from "stripe";
 
const db = new Firestore();
const stripe = new Stripe(process.env.STRIPE_API_KEY ?? "");
 
type OrderStatus = "pending" | "paid" | "refunded" | "canceled";
 
// Only ever move forward. A stale read can never undo a later state.
const ALLOWED: Record<OrderStatus, OrderStatus[]> = {
  pending: ["paid", "canceled", "refunded"],
  paid: ["refunded"],
  refunded: [],
  canceled: [],
};
 
function paymentIntentId(event: Stripe.Event): string | null {
  const obj = event.data.object as { id: string; object: string; payment_intent?: string | null };
  if (obj.object === "payment_intent") return obj.id;
  if (obj.object === "charge") return obj.payment_intent ?? null;
  return null;
}
 
// Derive the status from current truth, not from the event that woke us up.
function derive(pi: Stripe.PaymentIntent): { status: OrderStatus; refunded: number } {
  const charge = typeof pi.latest_charge === "object" ? pi.latest_charge : null;
  const refunded = charge?.amount_refunded ?? 0;
  if (pi.status === "canceled") return { status: "canceled", refunded };
  if (pi.status !== "succeeded") return { status: "pending", refunded };
  // charge.refunded fires for partial refunds too; only a full refund changes the status.
  if (charge && refunded >= charge.amount) return { status: "refunded", refunded };
  return { status: "paid", refunded };
}
 
export async function processEvent(eventId: string) {
  const eventRef = db.collection("webhookEvents").doc(eventId);
  const snap = await eventRef.get();
  if (!snap.exists || snap.get("status") === "done") return;
 
  const event = JSON.parse(snap.get("payload")) as Stripe.Event;
  const piId = paymentIntentId(event);
 
  if (piId) {
    const pi = await stripe.paymentIntents.retrieve(piId, { expand: ["latest_charge"] });
    const orderId = pi.metadata.orderId;
    if (!orderId) throw new Error(`payment intent ${piId} has no orderId metadata`);
 
    const { status: target, refunded } = derive(pi);
    const orderRef = db.collection("orders").doc(orderId);
 
    await db.runTransaction(async (tx) => {
      const order = await tx.get(orderRef);
      const current = (order.get("status") ?? "pending") as OrderStatus;
      const refundedSoFar = (order.get("refundedAmount") ?? 0) as number;
 
      const update: Record<string, unknown> = {
        refundedAmount: Math.max(refundedSoFar, refunded),
        lastEventId: event.id,
      };
      if (target !== current && ALLOWED[current].includes(target)) {
        update.status = target;
        tx.create(db.collection("outbox").doc(`${orderId}-${target}`), {
          kind: `order.${target}`,
          orderId,
        });
      }
      tx.set(orderRef, update, { merge: true });
    });
  }
 
  // Event types we do not act on are marked done too, on purpose.
  await eventRef.update({ status: "done" });
}

Several things happen in that transaction. Two workers handling duplicates cannot both transition the order. A worker that read the API a moment before another one cannot move the order backwards, because the transition table refuses it. The outbox document has a deterministic ID, so the side effect (the receipt email, the fulfilment call) is created at most once, using the same create-only trick as the event store. A separate process sends outbox entries; that is where the techniques from protecting pay-per-call APIs apply if the downstream call costs money.

The cost is one API call per processed event. In return, a premature refund event now works: the worker fetches the PaymentIntent, finds it succeeded and fully refunded, and moves the order straight from pending to refunded, which is why that transition is allowed. The late payment_intent.succeeded then derives the same target and changes nothing.

The metadata.orderId convention works because you attach it when creating the payment object. Do not infer the order from the customer's email or from an amount.

Partial refunds that look like refunds#

charge.refunded fires whenever a charge is refunded, including partially. A handler that maps the event type to the refunded status will mark an order refunded after a refund of a single line item. The derive function compares amount_refunded with the charge amount and records the running total on the order instead, so a partial refund leaves the order paid with a refundedAmount that downstream code can show. The same reasoning covers any event type that names a category of change rather than the resulting state: decide from the object, not from the name.

Two Event objects for one change, and events you never wanted#

Event-ID deduplication cannot catch the case where Stripe generates two separate Event objects for one change. The state guard does, because the second event derives the same target and finds nothing to do. For effects that are not state transitions, deduplicate on the object ID plus event type, as the documentation suggests.

Subscribing an endpoint to every event type means handling types you have never seen. Configure it for only the types you handle, and make the worker ignore unknown ones without error, since an error just retries them.

A payload that always makes the worker throw will retry forever unless the queue has a retry limit and a dead-letter path. Reuse the dead-letter pattern and alert when anything lands there.

Recovering what the retries missed#

The retry window is days, not forever. If your own outage outlasts it, only reconciliation recovers the missing events. Two small jobs make the pipeline recoverable. The first finds events stuck in received for more than a few minutes and re-enqueues them. The second lists recent events from the Events API, compares them with your store and feeds the difference through the same worker. The API only keeps events for a limited period, so reconciliation has to run well inside it rather than after a long outage is noticed.

To test without a public URL, the Stripe CLI forwards events to your machine:

terminalbash
stripe listen --forward-to localhost:8080/webhooks/stripe
stripe trigger payment_intent.succeeded

listen prints a signing secret for the local session; use that as the endpoint secret while testing. Two automated tests earn their keep: deliver the same event twice and assert the order and the outbox each hold one entry, and deliver charge.refunded before payment_intent.succeeded and assert the order ends refunded.

Key takeaways

  • Verify against the raw body, with the endpoint's own secret and the default five-minute tolerance.
  • Store each event with a create-only write keyed by the event ID, enqueue, and acknowledge; duplicates return 2xx.
  • Never decide an order's state from the type of the event that arrived. Fetch the current object and derive the state, and use the transition table only to stop moves backwards.
  • Do not mark an event done because it did not apply yet. Either derive from fresh state or return an error so it is retried.
  • charge.refunded includes partial refunds; compare amount_refunded with the amount.
  • Reconcile events stuck in received and events missing from your store, inside the provider's retention window.
Share
All articles →