Protecting pay-per-call APIs: validation, idempotency, locks

Every upstream call spends money and quota. A layered defence of validation, rate limits, caching, single-flight and leases keeps both under control.

Published

Updated

Reading time

15 min

Take a geocoding endpoint that costs $5 per 1,000 upstream calls, behind a page that makes one lookup per view. At 200,000 uncached views a day, that is $1,000 a day, and a script hitting the endpoint in a loop can spend it faster than anyone reads a dashboard. The numbers are invented, but the shape is real for any provider that bills per request: a lookup, a document check, an SMS gateway.

Most advice for protecting an endpoint like that is a catalogue of textbook patterns. This article is a design review of one geocoding backend that concentrates on what is different when each call costs money. Four things matter more than they do elsewhere: negative caching, one global token bucket sized under the provider's quota and charged only for real calls, single-flight against stampedes, and a spend cap you cannot argue with. The rest is plumbing that other articles already cover, and I link to them.

What a paid upstream changes#

The service runs as several stateless instances behind a load balancer (Cloud Run, Kubernetes), so any state shared between instances sits in Redis or Firestore. Three facts about the provider shape everything:

  • The quota belongs to your key, not to an instance. A limiter in process memory protects nothing; the bucket has to be shared.
  • Waste has four sources. Invalid input that is forwarded and billed anyway, duplicates (retries, double submits, a page that fires the same query every render), concurrency (many users asking for the same thing the moment a cache entry expires), and abuse such as enumerating a lookup endpoint through your paid key.
  • Errors may cost money too. Some providers bill throttled or failed calls. Read the terms before assuming a 429 is free.

My own version of this problem is a large-scale social platform I'm building that calls several third-party APIs, some billed per call and some metered by a fixed quota that takes the feature down for every user once it is spent. Users decide when those requests happen. That is why I stopped thinking of this as load protection: my infrastructure scales happily under a hammered endpoint, so the attack lands on the bill, and autoscaling makes it more effective.

Cheapest checks first. The paid call comes last, reached only on a cache miss by a single lease holder that has taken a global token.

The order is the design. Everything before the cache is cheap and can reject; everything after it exists to make sure one miss produces at most one paid call.

Make equal questions identical#

" 10 Main St " and "10 main st" should be one cache entry, not two paid calls, and an input that could never succeed should be rejected locally. Normalise first, derive the cache key from the normalised form, and version the key so a change to the rules does not serve entries built under the old ones:

src/geocode/input.tsts
import { createHash } from "node:crypto";
 
const MAX_LEN = 200;
 
export type GeocodeQuery = { address: string; country: string };
 
export function normaliseGeocodeQuery(raw: unknown): GeocodeQuery | null {
  if (typeof raw !== "object" || raw === null) return null;
  const { address, country } = raw as Record<string, unknown>;
  if (typeof address !== "string" || typeof country !== "string") return null;
 
  const a = address.normalize("NFKC").trim().replace(/\s+/g, " ").toLowerCase();
  const c = country.trim().toUpperCase();
 
  if (a.length < 3 || a.length > MAX_LEN) return null;
  if (!/[\p{L}\p{N}]/u.test(a)) return null;      // must contain a letter or digit
  if (!/^[A-Z]{2}$/.test(c)) return null;          // ISO 3166-1 alpha-2
 
  return { address: a, country: c };
}
 
export function cacheKey(q: GeocodeQuery): string {
  const h = createHash("sha256").update(`${q.country}|${q.address}`).digest("hex");
  return `geocode:v1:${h}`;
}

The ordering is the discipline: validation is free and upstream calls are not, so it runs first. In my services the schemas already existed for other reasons, and a surprising share of junk traffic (malformed IDs, fuzzing probes, botched client requests) dies there at no cost. I also reject identifiers that cannot exist upstream, such as a wrong length or an illegal prefix, because there is no point asking a paid API to confirm what a regular expression already knows.

A per-user limit comes right after validation, keyed by authenticated identity rather than IP, since IPs are shared behind carrier NAT and easily rotated. That is the ordinary rate limiter, and any token bucket will do. The next layer is not ordinary.

One global bucket, charged only for real calls#

The provider's limit applies to your whole key. A global token bucket in Redis, sized just under that limit, turns a provider-side 429, which may be billed and may count against you, into a cheap local rejection. The detail that matters is where it sits. Charge it only at the moment a request is about to make a paid call, not at the front door, so cache hits do not consume quota. If it sat at the front, a burst of cached reads would starve the real misses.

The check must be atomic, and using the server's TIME avoids clock skew between instances:

src/limits/redis-bucket.tsts
import Redis from "ioredis";
 
const redis = new Redis(process.env.REDIS_URL!);
 
const SCRIPT = `
local key      = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local cost     = tonumber(ARGV[3])
local t        = redis.call('TIME')
local now      = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local s        = redis.call('HMGET', key, 'tokens', 'ts')
local tokens   = tonumber(s[1]) or capacity
local ts       = tonumber(s[2]) or now
tokens = math.min(capacity, tokens + (now - ts) / 1000 * rate)
local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end
redis.call('HSET', key, 'tokens', tostring(tokens), 'ts', tostring(now))
redis.call('PEXPIRE', key, math.ceil(capacity / rate * 1000) + 1000)
return allowed
`;
 
export async function allow(
  key: string,
  capacity: number,
  refillPerSecond: number,
  cost = 1,
): Promise<boolean> {
  const res = await redis.eval(SCRIPT, 1, key, capacity, refillPerSecond, cost);
  return res === 1;
}

Size the numbers from the provider's documented quota, with headroom for other clients of the same key. If the provider allows about 50 calls per second, a bucket of capacity 40 refilling at 40 per second keeps a margin. The per-user bucket is a smaller one of the same shape, say capacity 10 refilling at one token every five seconds.

Remember "not found"#

The response cache is where most of the money is saved, because it deduplicates across users. Two users geocoding the same address, or a thousand viewing the same public entity, should cost one call per TTL. Choose the TTL from how fast the data changes and from the provider's terms, since some restrict how long you may store responses. For identity-style lookups I ask what staleness the feature can honestly tolerate, and the answer has been far more than my paranoia suggested; an account's identity mapping changes slowly, so caching it turns the second request through the thousandth into a free local read. Providers that use OAuth-style flows need the same treatment for the access token: fetching a fresh one per request doubles the call volume for no information, so I reuse it until it is near expiry. Quick-start client code often skips that.

The part specific to paid APIs is the negative entry. Without one, a lookup for something that does not exist is the most expensive request a user can make, because it can never be cached: every repeat is a paid call that returns nothing. An enumeration script lives on exactly that gap. Cache "not found" with a shorter TTL than a hit:

src/geocode/cache.tsts
import Redis from "ioredis";
 
const redis = new Redis(process.env.REDIS_URL!);
 
type Cached<T> = { found: true; value: T } | { found: false };
 
const HIT_TTL_S = 7 * 24 * 3600;   // positive results
const MISS_TTL_S = 6 * 3600;       // negative results
 
export async function readCache<T>(key: string): Promise<Cached<T> | null> {
  const raw = await redis.get(key);
  return raw ? (JSON.parse(raw) as Cached<T>) : null;
}
 
export async function writeCache<T>(key: string, entry: Cached<T>): Promise<void> {
  const ttl = entry.found ? HIT_TTL_S : MISS_TTL_S;
  await redis.set(key, JSON.stringify(entry), "EX", ttl);
}

One rule keeps this safe: only a definitive answer from the provider ("no such address") becomes a negative entry. A timeout, a 429 or a 5xx must never be cached as "not found", or a short upstream incident becomes six hours of wrong answers. Lookup endpoints that reveal whether something exists deserve a failure lockout on top (a counter of misses per user and a growing lock window), and the same response shape and latency for "not found" and "locked", so the endpoint does not leak which happened.

The failure rate is the more useful signal for that lock. Legitimate use mostly asks questions with a "yes" answer, and enumeration is overwhelmingly "no", so a stream of misses from one actor is the fingerprint of someone using my wallet as a search engine. The lock stops that stream before the upstream call is made. I accepted the trade-off: someone who mistypes an ID several times gets told to wait, which I prefer to a quota outage for everybody.

Collapse the stampede#

When a popular entry expires, every request arriving before it is refilled misses. Two layers deal with it. In process, a single-flight map makes concurrent callers for one key share one promise. I key it by the upstream call a request would trigger rather than by HTTP route, because it earns its keep during a slow upstream response, when retries from clients would otherwise multiply my own spend:

src/concurrency/single-flight.tsts
export class SingleFlight<T> {
  private readonly inflight = new Map<string, Promise<T>>();
 
  do(key: string, fn: () => Promise<T>): Promise<T> {
    const existing = this.inflight.get(key);
    if (existing) return existing;
 
    const p = fn().finally(() => this.inflight.delete(key));
    this.inflight.set(key, p);
    return p;
  }
}

That removes duplicates within an instance and does nothing across instances. For those, a lease: a lock with an expiry, so a crashed holder cannot block the key forever. The holder fetches and fills the cache while everyone else waits for the fill. The lease is an efficiency lock, not a correctness lock: if the holder stalls past its TTL a second caller may fetch too, which for a cache fill costs one extra call. Idempotent Pub/Sub consumers shows a Firestore claim with a lease and when you need fencing instead, and refresh-on-read instead of a cron job uses the same idea to refresh a record once when it is read. In Redis the lease is one command to acquire and a token-checked script to release, so you never delete someone else's lease:

src/concurrency/redis-lease.tsts
import { randomUUID } from "node:crypto";
import Redis from "ioredis";
 
const redis = new Redis(process.env.REDIS_URL!);
 
const RELEASE = `
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0
`;
 
export async function acquireLease(key: string, ttlMs: number): Promise<string | null> {
  const token = randomUUID();
  const ok = await redis.set(`lease:${key}`, token, "PX", ttlMs, "NX");
  return ok === "OK" ? token : null;
}
 
export async function releaseLease(key: string, token: string): Promise<void> {
  await redis.eval(RELEASE, 1, `lease:${key}`, token);
}

Composing the layers with a stated failure policy#

Every layer is also a dependency. The policy for paid calls: fail closed on the limiter and the lease (if the shared state cannot say how much budget is left, do not spend), and fail open on the cache read, treating a read error as a miss, because the limiter and the lease still bound the cost of a miss. A failed cache write after a successful paid call is logged and swallowed, since the caller already has the result they paid for.

The waiting timings must agree with the lease. A waiter that polls for three seconds against a ten-second lease gives up before the lease can expire, so a crashed holder produces 503s for a lease length. Here waiters keep trying to acquire the lease while they poll, and only give up after the lease could have expired, so a dead holder is replaced instead of waited out. The lease is sized from the provider's latency: with a p99 of about two seconds (hypothetical), six seconds leaves margin.

src/geocode/service.tsts
import { normaliseGeocodeQuery, cacheKey } from "./input";
import { readCache, writeCache } from "./cache";
import { allow } from "../limits/redis-bucket";
import { SingleFlight } from "../concurrency/single-flight";
import { acquireLease, releaseLease } from "../concurrency/redis-lease";
import { geocodeUpstream, type GeoResult } from "./provider";
import { isEnabled } from "../flags/flags";
import { logger } from "../logger";
 
const LEASE_MS = 6_000;           // a few times the provider's p99
const WAIT_MS = LEASE_MS + 1_000; // waiters outlast the lease
const POLL_MS = 300;
 
const flights = new SingleFlight<GeoResult | null>();
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
 
export class HttpError extends Error {
  constructor(readonly status: number, message: string, readonly retryAfterS?: number) {
    super(message);
  }
}
 
const unwrap = (c: { found: true; value: GeoResult } | { found: false }) => (c.found ? c.value : null);
 
// Fail open: a cache read error is treated as a miss.
async function tryRead(key: string) {
  try {
    return await readCache<GeoResult>(key);
  } catch (err) {
    logger.warn({ err }, "cache read failed, treating as miss");
    return null;
  }
}
 
export async function geocode(uid: string, raw: unknown): Promise<GeoResult | null> {
  if (!isEnabled("geocode")) throw new HttpError(503, "feature_disabled");
 
  const q = normaliseGeocodeQuery(raw);
  if (!q) throw new HttpError(400, "invalid_query");
 
  // Fail closed: if the limiter cannot answer, do not spend.
  try {
    if (!(await allow(`rl:user:${uid}:geocode`, 10, 0.2))) throw new HttpError(429, "slow_down", 5);
  } catch (err) {
    if (err instanceof HttpError) throw err;
    throw new HttpError(503, "limiter_unavailable");
  }
 
  const key = cacheKey(q);
  const hit = await tryRead(key);
  if (hit) return unwrap(hit);
 
  return flights.do(key, async () => {
    const deadline = Date.now() + WAIT_MS;
    for (;;) {
      let token: string | null;
      try {
        token = await acquireLease(key, LEASE_MS); // fail closed
      } catch {
        throw new HttpError(503, "lease_unavailable");
      }
 
      if (token) {
        try {
          const again = await tryRead(key); // filled while we acquired?
          if (again) return unwrap(again);
 
          // The global bucket is charged only for a real upstream call.
          if (!(await allow("rl:global:geocode", 40, 40))) throw new HttpError(503, "busy", 1);
 
          const result = await geocodeUpstream(q); // throws on timeout/429/5xx: never cached
          try {
            await writeCache(key, result ? { found: true, value: result } : { found: false });
          } catch (err) {
            logger.warn({ err }, "cache write failed after a paid call");
          }
          return result;
        } finally {
          await releaseLease(key, token).catch(() => undefined);
        }
      }
 
      // Someone else holds the lease. Wait for the fill, but keep trying to
      // take over, so a crashed holder is replaced once its lease expires.
      if (Date.now() >= deadline) throw new HttpError(503, "try_again", 1);
      await sleep(POLL_MS);
      const filled = await tryRead(key);
      if (filled) return unwrap(filled);
    }
  });
}

The isEnabled call is the kill switch from feature flags in Firestore, which already handles a slow or unreachable store by falling back to defaults.

Retries deserve one sentence of discipline, because they are where a failing provider multiplies your bill. Retry only 429 and 5xx, cap attempts at two or three, use exponential backoff with full jitter, honour Retry-After, and stop calling altogether behind a circuit breaker after repeated failures, serving a stale cache entry meanwhile if you have one. A 400 never succeeds on retry, so retrying it only spends money. The global bucket applies to every attempt, retries included.

Everything above is about reads. For a paid action that changes state, such as sending a message or submitting a verification check, the client sends an Idempotency-Key header, the convention in the IETF draft-ietf-httpapi-idempotency-key-header. That draft is still an Internet-Draft, not an RFC (revision 08 at the time of writing), though it is widely implemented. The server stores the outcome by user and key and replays it for retries, with an explicit in-progress state so a retry that arrives while the first attempt is running gets a 409 instead of starting a second paid call.

The claim-and-lease mechanics are the ones in the Pub/Sub article above. One judgement call is specific to payments for calls: if you delete the key when your call errors, and the provider had actually performed the action before the timeout, the client's retry double-charges. Pass your key through to the provider if it supports one, or record the ambiguous state and reconcile.

A cap you cannot argue with#

Every layer above has a bug you have not found yet. The last line of defence is financial. Most providers let you set a daily quota or spend cap in their own console; set it just above expected peak. On Google Cloud, Cloud Billing budgets send alerts at thresholds and do not stop spending on their own, but you can route budget notifications to Pub/Sub and act on them programmatically, for example by flipping the kill switch.

I also treat third-party usage as a budget line. It sits in my cost model next to database reads and storage, with an assumed number of calls per user per day and an alert threshold on actual consumption. A spike in upstream calls pages me even when nothing is down, because it means either a bug is spending money in a loop or someone has found a way around the layers above.

CostDo the arithmetic before launch

With the hypothetical $5 per 1,000 calls and 200,000 uncached views a day, a shared cache with a 90% hit rate brings $1,000 down to $100, and single-flight shaves the stampede spikes on top. The negative entries matter most for the traffic you do not want: a script probing nonexistent addresses costs nothing after the first probe of each. Check the provider's pricing page at the time of writing, multiply by real traffic, and choose the TTL and the cap from the result, not after the first invoice.

Where the design bends#

Stale data. Long TTLs save money and serve old answers. Offer a user-triggered refresh that is itself rate limited, rather than shortening the TTL for everyone.

Lease TTL. Too short and a slow upstream call lets a second fetcher in; too long and a crashed holder delays its key. Tie it to the provider's p99, as above.

Waiters trade latency for cost. A caller that loses the lease waits up to seven seconds in the worst case, which is slower than calling upstream directly and is the point.

Lockouts hit legitimate users. Keep the threshold generous, the first lockout short, and give support a way to clear it.

If the call is free, unmetered and fast, most of this is overhead, and a timeout plus a modest cache is enough. And if the data could be fetched on a schedule in one batch and served from your own store, moving the paid call off the request path is the strongest protection there is.

Share
All articles →