Rate limiting at the edge versus in the service: where each limit goes
A practical split for small teams: coarse per-client limits at the edge, per-user and per-cost limits in the service, and the 429 behaviour both must share.
Take an API with three kinds of traffic: a login endpoint that credential-stuffing bots will find, a search endpoint that costs fifty times what a read costs, and ordinary reads from a mobile app that occasionally gets stuck in a retry loop. One rate limit cannot serve all three. A limit at the edge can say "this address has sent far too much", and cannot say "this account has spent its share of search". A limit in the service can say the second, but only after you have paid to receive, parse and authenticate the request you are about to refuse.
So the question is not whether to limit, it is which limit goes where. This guide compares the places a limit can live, shows what tips the choice, then builds the two I recommend for a small team: a Cloud Armor throttle rule at the edge and a cost-weighted token bucket in the service, sharing one 429 contract. It builds on protecting pay-per-call third-party APIs, which covers limits that exist to protect money, and on JWT at the edge: verify once, trust a header, which explains how the service learns who a request belongs to.
Four places a limit can live#
| Layer | What it sees | Strength | Weakness |
|---|---|---|---|
| Load balancer or WAF, such as Cloud Armor | IP, path, headers, region and other request attributes | Rejects before your code runs; one rule covers every backend | Cannot see the verified user or the cost of a request |
| API gateway quotas | API key or consumer identity, depending on product | Per-consumer quotas without code | Another component to configure and learn |
| In-process limiter in the service | Anything your code knows | Per-user, per-plan, per-operation cost | State is per instance on a platform that scales out; you pay to receive the request |
| Shared-store limiter (Redis, a database counter) | Same as in-process, but global | Accurate across instances | A network hop and a dependency on the hot path |
What tips the decision#
Four questions settle most cases.
What is the limit protecting? A limit with no named resource is a guess. A connection pool, a paid third-party call and a login endpoint under attack each have a different right answer. Connection pools and flood protection point at the edge; paid calls and fairness between accounts point at the service.
Who is the caller at that layer? At the edge you mostly have an IP address, which is a weak identity: many users share one behind a corporate or mobile network, and an attacker can rotate addresses. That is why edge limits must be generous. In the service you may have a verified user and a plan, which is what makes fair limits possible.
Is rejecting cheaper than serving? Only at the edge. A request refused by the load balancer never starts a container or touches your code. The cost is that the edge layer is a separate product: an external Application Load Balancer and Cloud Armor are billed apart from Cloud Run, and a load balancer has a fixed component that a scale-to-zero service does not. For a low-traffic site, a service limiter plus a maximum instance count is a reasonable first step. For a public API that attracts abuse, the edge layer pays for itself the first time it refuses a flood before your instances scale up to serve it. Check Cloud Armor pricing and the load balancing pricing page when you decide.
Does the limit need to be exact? Most limits only need to be roughly right, and a per-instance counter is roughly right when traffic spreads across instances. A paid quota or a login lockout needs a shared store, and only those endpoints should pay the extra hop.
My recommendation follows from those answers: a coarse edge limit per client and per sensitive path, a service limit per authenticated user weighted by cost, one response contract, and shared state only where a limit must be exact.
The edge: a Cloud Armor throttle rule#
Cloud Armor rate limiting is enforced on load balancers at the network edge, so using it with Cloud Run means putting an external Application Load Balancer with a serverless network endpoint group in front of the service. Google's guide for that setup deploys the service with --ingress=internal-and-cloud-load-balancing, which blocks external requests sent directly to the service's default URLs. That setting matters as much as the rule: without it, a client skips the limit by calling the default URL. See setting up a load balancer with Cloud Run.
gcloud run deploy my-api \
--image=IMAGE_URL \
--region=europe-west4 \
--ingress=internal-and-cloud-load-balancingA throttle rule takes an action of throttle, a threshold count, a threshold interval in seconds, a conform action of allow and an exceed action that is one of the deny codes or a redirect. The documentation recommends 429 for the deny status. The policy name, priority and path here are placeholders:
gcloud compute security-policies rules create 1000 \
--security-policy=api-edge-policy \
--expression="request.path.matches('^/auth/')" \
--action="throttle" \
--rate-limit-threshold-count=20 \
--rate-limit-threshold-interval-sec=60 \
--conform-action=allow \
--exceed-action=deny-429 \
--enforce-on-key=IPTwenty requests per minute per IP on authentication paths is an arbitrary figure for the example; derive yours from what a legitimate user does. The rate limiting overview covers the details worth knowing:
- Throttle allows a threshold count of 1 to 1,000,000 per client per interval; rate-based ban allows 1 to 10,000. The interval must be one of a fixed list of values between 10 and 3,600 seconds, so 60 is valid and 45 is not.
- A rate-based ban applies the exceed action to all of a client's requests for 60 to 3,600 seconds. You can change a throttle rule to a rate-based ban later but not the other way round, so start with throttle.
- The key can be the IP, the first address in
X-Forwarded-For, a header, a cookie, the path, a region code and others, and you can combine up to three. - Rules can be tested in preview mode, with the effects visible in request logs.
The command reference notes that --enforce-on-key-configs replaces --enforce-on-key and --enforce-on-key-name, so check the current reference for the form your gcloud version prefers. Attach the policy to the backend service of the load balancer, and run it in preview mode before enforcing.
The service: a token bucket that knows the user#
In the service you know who is calling, because the token was verified once and the identity travels as a trusted header or request property. A token bucket is the simplest model that tolerates bursts: each key has a bucket that refills at a steady rate, and each request takes tokens. Here is a small in-process version:
interface Bucket {
tokens: number;
at: number;
}
export class TokenBucket {
private buckets = new Map<string, Bucket>();
constructor(
private capacity: number,
private refillPerSec: number,
private now: () => number = () => Date.now(),
) {}
take(key: string, cost = 1): { allowed: boolean; retryAfterSec: number } {
const t = this.now();
const b = this.buckets.get(key) ?? { tokens: this.capacity, at: t };
b.tokens = Math.min(
this.capacity,
b.tokens + ((t - b.at) / 1000) * this.refillPerSec,
);
b.at = t;
const allowed = b.tokens >= cost;
if (allowed) b.tokens -= cost;
this.buckets.set(key, b);
const retryAfterSec = allowed
? 0
: Math.ceil((cost - b.tokens) / this.refillPerSec);
return { allowed, retryAfterSec };
}
}The clock is injectable so a test can advance time without sleeping. As Express middleware, with a cost per route:
import type { Request, Response, NextFunction } from "express";
import { TokenBucket } from "./limiter";
const perUser = new TokenBucket(60, 1); // 60 burst, 1 token per second
const perAddress = new TokenBucket(20, 0.5); // tighter, for callers with no identity
export function limit(cost: number) {
return (req: Request, res: Response, next: NextFunction) => {
// Set by the component that verified the token; absent on anonymous routes.
const userId = req.header("x-user-id");
const { allowed, retryAfterSec } = userId
? perUser.take(`user:${userId}`, cost)
: perAddress.take(`ip:${req.ip}`, cost);
if (allowed) return next();
res
.status(429)
.set("Retry-After", String(retryAfterSec))
.json({ error: "rate_limited", retryAfterSec });
};
}
// cheap read: limit(1), expensive search: limit(10), report export: limit(30)Weighting by cost is the main reason to limit in the service at all. One export can cost as much as fifty reads, and an edge rule that counts requests cannot see that. Two further details. The map grows with the number of distinct keys, so evict idle entries on a timer or cap the size, or the limiter is a slow memory leak. And the identity header is only trustworthy if the service rejects requests that did not come through the component that sets it.
Behind a proxy, req.ip is not the client#
The version of this middleware that is easiest to write keys on x-user-id ?? req.ip ?? "anonymous", and it has a bug that only shows up in production. Behind Cloud Run, and behind a load balancer in front of Cloud Run, the TCP peer your container sees is a Google front end, not the caller. Unless Express is told to trust the proxy, req.ip is that peer's address, so every anonymous caller lands in one bucket: one noisy client throttles all the others, or the limit is so loose it is useless. The client address is in the X-Forwarded-For header, and Express only reads it when trust proxy is configured.
Set it to the exact number of proxy hops in front of the app, not to true:
import express from "express";
const app = express();
// Hops between the client and this process. Verify by sending a request
// and logging the X-Forwarded-For header, then counting from the right.
app.set("trust proxy", 2);With a number, Express takes the address that many hops from the right end of X-Forwarded-For, which are the entries your own infrastructure appended. true trusts the leftmost entry, which the client can write. This is the same hazard that applies to the edge rule: keying on X-Forwarded-For without knowing which proxies set it lets a client choose its own key. At the edge, use the key types Cloud Armor documents for your load balancer type. In the service, count hops from the right.
The code above also keeps anonymous traffic in its own, tighter bucket and prefixes keys with user: and ip: so that a header value can never collide with an address.
Instances multiply in-process budgets#
Cloud Run scales by adding instances, and each has its own memory. A limiter budget of 60 tokens gives a user up to 60 per instance, so a user whose requests land on four instances can in principle use four times the budget. For abuse and runaway loops that is acceptable, because the aim is to stop them, not to bill precisely. Two things reduce the error: a maximum instance count caps the multiplication, and higher per-instance concurrency concentrates one user's requests on fewer instances. The default concurrency depends on how the service was created: 80 in the console, and 80 times the number of vCPUs for services deployed with the gcloud CLI or Terraform.
When a limit must be exact, such as a paid quota or a login lockout, move the counter into a shared store. A database row with an atomic increment and a window key works at small scale; a managed in-memory store works at larger scale. Accept the extra hop for those endpoints only.
One 429 contract for both layers#
Status 429 means too many requests, and RFC 6585 says the response may include a Retry-After header saying how long to wait. The edge returns a plain 429 without your JSON body, so client code should key off the status first and treat the body as optional. A correct client stops sending, waits at least the Retry-After value when present, and adds random jitter so a thousand clients do not retry in the same second:
export async function fetchWithBackoff(url: string, init?: RequestInit, tries = 4) {
for (let attempt = 0; attempt < tries; attempt++) {
const res = await fetch(url, init);
if (res.status !== 429) return res;
const header = Number(res.headers.get("retry-after"));
const base = Number.isFinite(header) && header > 0 ? header * 1000 : 500 * 2 ** attempt;
await new Promise((r) => setTimeout(r, base + Math.random() * 250));
}
throw new Error("rate limited after retries");
}Retrying only on 429 is deliberate in this snippet, and the count is capped: clients that retry without limit turn a limit into an amplifier.
A limit nobody sees is a silent outage, so log every rejection with the key, route and cost, and chart the rejection rate. Cloud Armor writes rule evaluation into the load balancer request logs, which you can query in Cloud Logging, and preview results should be read before enforcing a new rule. In the service, count 429s as their own metric: a spike has a different cause and fix than a generic error.
Which setup for which situation#
- Public API with login, signup or search paths that attract abuse. Both layers. Edge throttle on sensitive paths with generous numbers, service bucket per user weighted by cost, shared store only for lockouts and paid quotas.
- Low-traffic site or early product. Skip the load balancer for now. Use the service limiter with a maximum instance count, and move to the edge when abuse shows up in the rejection chart or the bill.
- Traffic that already comes through a managed API gateway with per-consumer quotas. Use the gateway quotas, and keep service code only for the cost-weighted limits the gateway cannot express.
- Internal tool with a few dozen known users. A rate limiter is complexity without a threat. A sensible maximum instance count and a database connection cap are enough.
- One slow endpoint exhausting a connection pool. A limit only hides it. Fix the query, cap concurrency on that route, or move the work to a queue.
Whichever you pick, review the numbers when you ship a feature that adds calls. A limit chosen in a quiet month becomes an outage in a busy one, and a burst size smaller than what one page view needs punishes normal use.
Related articles
All articles →A feature-flag vendor is more than a small team needs. One Firestore document, a short in-memory cache and safe defaults give you kill switches and rollouts.
11 min
Every upstream call spends money and quota. A layered defence of validation, rate limits, caching, single-flight and leases keeps both under control.
15 min
Ordering keys give per-key order and take away parallelism. Learn what they guarantee, how one failure stalls a key, and when versioned events are a better fix.
9 min