Refresh-on-read instead of a cron job
A scheduled sync pays to refresh every record, including the ones nobody reads. Refreshing stale data when someone looks makes cost follow attention.
If you cache third-party data (exchange rates, repository stats, shipment status, profile data from an external service), refresh a record when someone reads it and it is stale, not on a schedule. Serve the stored copy immediately, and let the read start one background refresh. The upstream bill then follows how many records people look at, not how many records you hold. A nightly sweep is the obvious design and it is easy to defend, which is exactly why it deserves an argument: its cost grows with inventory, and inventory only ever grows. I chose it for stats synced from third-party accounts that users link on a large-scale social platform I'm building, where a nightly sweep's bill would grow with linked accounts while its value grew only with the profiles people actually open.
That position has limits, and the second half of this article is about where it stops holding. First, the mechanism, because the argument only works if the mechanism is safe: no duplicate refreshes, no stampede on the upstream, no read that fails because a refresh did.
What a nightly sweep pays for#
Read counts for almost any user-facing dataset have a long tail. A small set of records is read many times a day, a larger set a few times a month, and most are read rarely or never: accounts that signed up and left, entities linked once and forgotten. A sweep ignores that shape and pays full price for every record, every night:
- Upstream calls. One per record per run, often against an API that charges per call or enforces a daily quota.
- Database writes. One per record per run, even when nothing changed, unless you diff first, which costs a read.
- Blast radius. When the upstream has a bad night, the sweep fails for everyone at once, and the retry doubles the load on a service that is already struggling.
The freshness is uneven too: a record refreshed at 03:00 is twenty hours old when its owner opens it at 23:00.
In my case the deciding detail was the dormant user. Someone who linked an account once and never came back would still cost a nightly upstream call, a database write and a slot in the rate-limit budget. The sweep would also have created a daily spike to provision for, which sits badly with a scale-to-zero budget that assumes load follows real users.
The strongest case for the cron job#
The sweep has real advantages, and a fair comparison starts with them.
Load is predictable. You know when the upstream will be hit and how hard, and you can schedule it for the quiet hours. Refresh-on-read produces load that follows your traffic, including a burst when a newsletter goes out.
Staleness is bounded for everyone. After a sweep, every record is at most a day old. Refresh-on-read gives an unread record unbounded staleness, and gives a rarely read record an old value on first read.
Batching is cheaper per record, and it is simpler. One job can use a bulk endpoint that a single-record refresh cannot, and it is one scheduler and one place to look. Refresh-on-read needs a lease, a queue, a worker and a backoff rule.
Those are all true. The counter-argument is not that they are wrong, but that for the common case, data that is displayed rather than acted on, they are bought at the price of refreshing records no one will ever see. The table compares the options on the dimension that matters most here, what the cost scales with.
| Approach | Cost scales with | Freshness when read | Read latency | Moving parts |
|---|---|---|---|---|
| Nightly full sweep | Total records | Up to 24 h old | Fast (cached) | Scheduler, batch job |
| Sweep of recently active records | Active records | Up to one run interval old | Fast | Scheduler, batch job, activity index |
| Synchronous refresh on read | Reads of stale records | Always fresh | Slow on stale reads, fails with the upstream | None extra |
| Stale-while-revalidate on read | Distinct stale records read | One TTL, plus one refresh | Fast (cached) | Lease, queue, worker |
| Upstream push (webhooks) | Changes upstream | Near real time | Fast | Webhook endpoint, signature checks |
Webhooks beat everything when the upstream offers them. Synchronous refresh ties your page latency to someone else's API. The recently-active sweep is a decent middle ground, but it still refreshes records whose readers will not come back before the next run.
Serve the stored copy, refresh behind it#
HTTP has a name for the behaviour: RFC 5861 defines the stale-while-revalidate Cache-Control extension, where a cache keeps serving a response after it goes stale, for a bounded time, while it revalidates in the background. MDN's description puts the benefit well: revalidation hides its latency from the client. The pattern here applies the same idea one layer down, in the service that owns the data, because the origin is a third-party API and the cache is a Firestore document.
The rules:
- Every read returns the stored snapshot immediately, with its
fetchedAt, so the UI can say "updated 3 hours ago". - A snapshot older than its TTL makes the read trigger one background refresh. The reader does not wait.
- TTL depends on the reader. An owner sees a shorter TTL than a visitor, because owners notice staleness in their own data and visitors mostly do not. The owner has usually just done something and wants to see the result, whereas a stranger has no idea what the number was an hour ago. Visitor views vastly outnumber owner views, so in my system this one rule cut upstream volume noticeably.
- At most one refresh per record is in flight. A record read a thousand times in the minute after it goes stale causes one upstream call. The case I designed against was a popular stale profile getting fifty views in the same second: without the guard that is fifty upstream calls for identical data.
- A failing upstream backs off per record, and readers keep getting the last good snapshot.
- The refresh runs on a rate-limited queue sized to the upstream's quota, not on the request thread.
- Manual refresh exists, with a cooldown.
With these rules, upstream calls in any period are bounded by the number of distinct records read in that period, divided by how many TTL windows fit in it. Records nobody reads cost nothing.
One read, one lease, one named task#
Each cached record is one Firestore document keyed by the upstream identity, for example snapshots/{source}:{externalId}. It holds the payload plus bookkeeping: fetchedAt, leaseUntil, failures, nextAttemptAt. Keeping the bookkeeping on the same document means the hot path, a fresh read, costs one document read.
The read does a plain read first. Only if the snapshot is stale, not leased and not in backoff does it pay for a transaction to claim the lease, so most reads of a hot record are decided from the read already done.
import { createHash } from "node:crypto";
import { DocumentReference, Firestore, Timestamp } from "@google-cloud/firestore";
import { CloudTasksClient } from "@google-cloud/tasks";
import { env } from "../config/env";
import { logger } from "../logger";
const db = new Firestore();
const tasks = new CloudTasksClient();
export type Audience = "owner" | "visitor";
const HOUR = 60 * 60 * 1000;
const TTL_MS: Record<Audience, number> = { owner: 24 * HOUR, visitor: 72 * HOUR };
const LEASE_MS = 5 * 60 * 1000;
const GRPC_ALREADY_EXISTS = 6;
export interface Snapshot<T> {
data: T | null;
fetchedAt: Timestamp | null;
leaseUntil?: Timestamp;
nextAttemptAt?: Timestamp;
failures?: number;
lastManualAt?: Timestamp;
}
const isStale = (s: Snapshot<unknown> | undefined, ttlMs: number, now: number) =>
!s?.fetchedAt || now - s.fetchedAt.toMillis() > ttlMs;
export const isBlocked = (s: Snapshot<unknown> | undefined, now: number) =>
(s?.leaseUntil?.toMillis() ?? 0) > now || (s?.nextAttemptAt?.toMillis() ?? 0) > now;
export async function readWithRefresh<T>(key: string, audience: Audience) {
const ref = db.collection("snapshots").doc(key);
const s = (await ref.get()).data() as Snapshot<T> | undefined;
const result = { data: s?.data ?? null, fetchedAt: s?.fetchedAt?.toDate() ?? null, refreshing: false };
const ttlMs = TTL_MS[audience];
if (!isStale(s, ttlMs, Date.now()) || isBlocked(s, Date.now())) return result;
try {
const generation = await claimLease(ref, ttlMs);
if (generation !== null) {
await enqueueRefresh(key, generation);
result.refreshing = true;
}
} catch (err) {
// A failed refresh must never fail the read. The lease expires on its own.
logger.warn({ err, key }, "refresh enqueue failed");
}
return result;
}
export async function claimLease(ref: DocumentReference, ttlMs: number): Promise<number | null> {
return db.runTransaction(async (tx) => {
const now = Date.now();
const s = (await tx.get(ref)).data() as Snapshot<unknown> | undefined;
if (!isStale(s, ttlMs, now) || isBlocked(s, now)) return null; // someone else won
tx.set(ref, { leaseUntil: Timestamp.fromMillis(now + LEASE_MS) }, { merge: true });
return s?.fetchedAt?.toMillis() ?? 0; // the generation this refresh replaces
});
}
export async function enqueueRefresh(key: string, generation: number) {
const parent = tasks.queuePath(env.GCP_PROJECT, env.TASKS_LOCATION, env.REFRESH_QUEUE);
const taskId = createHash("sha256").update(`${key}:${generation}`).digest("hex");
try {
await tasks.createTask({
parent,
task: {
name: `${parent}/tasks/${taskId}`,
httpRequest: {
httpMethod: "POST",
url: env.REFRESH_WORKER_URL,
headers: { "Content-Type": "application/json" },
body: Buffer.from(JSON.stringify({ key, generation })).toString("base64"),
oidcToken: { serviceAccountEmail: env.TASKS_INVOKER_SA, audience: env.REFRESH_WORKER_URL },
},
},
});
} catch (err) {
if ((err as { code?: number }).code !== GRPC_ALREADY_EXISTS) throw err;
}
}Three details carry most of the weight.
The transaction re-checks everything. Two instances can both see a stale, unleased snapshot on their plain read. Both enter runTransaction, and Firestore reruns a transaction when a concurrent write touches documents it read, so only one commits the lease. The loser re-reads, finds the lease, and returns null. Reads come before writes inside a Firestore transaction, which this function respects.
The task name is a second dedupe layer. Cloud Tasks rejects a create call with ALREADY_EXISTS when the ID matches an existing task or one deleted or executed recently. The IDs of deleted tasks are not immediately reusable: the reference says it can take up to 24 hours (9 days for a queue created from a queue.yaml file). Hashing the key together with the generation (the fetchedAt being replaced) means "refresh this record from this version" can be enqueued once, even if a lease expires early or a retry replays the enqueue, and since each generation gets a fresh ID, the reuse delay never blocks a legitimate refresh. A SHA-256 hex string also avoids the sequential IDs, such as timestamps, that the documentation warns raise latency and error rates.
The read awaits the enqueue. It is tempting to fire the enqueue and return. On Cloud Run with request-based billing, CPU is only allocated during request processing, so work left running after the response may stall until the next request reaches that instance. Paying a few tens of milliseconds on the rare read that starts a refresh is cheaper than switching billing mode.
A worker that records failures instead of retrying them#
The worker is an HTTP handler on Cloud Run that Cloud Tasks calls with an OIDC token. It checks whether someone has already refreshed past the generation it was asked to replace, calls the upstream once, and writes the outcome.
import { FieldValue, Firestore, Timestamp } from "@google-cloud/firestore";
import type { Snapshot } from "./read-with-refresh";
import { fetchUpstream } from "./upstream";
const db = new Firestore();
const HOUR = 60 * 60 * 1000;
const BASE_BACKOFF_MS = 0.25 * HOUR;
const MAX_BACKOFF_MS = 24 * HOUR;
export async function handleRefresh(key: string, generation: number): Promise<void> {
const ref = db.collection("snapshots").doc(key);
const s = (await ref.get()).data() as Snapshot<unknown> | undefined;
if ((s?.fetchedAt?.toMillis() ?? 0) > generation) return; // already refreshed
try {
const data = await fetchUpstream(key, { timeoutMs: 10_000 });
await ref.set(
{
data,
fetchedAt: FieldValue.serverTimestamp(),
failures: 0,
leaseUntil: FieldValue.delete(),
nextAttemptAt: FieldValue.delete(),
},
{ merge: true },
);
} catch (err) {
const failures = (s?.failures ?? 0) + 1;
const cap = Math.min(MAX_BACKOFF_MS, BASE_BACKOFF_MS * 2 ** (failures - 1));
const delay = cap / 2 + Math.random() * (cap / 2); // jitter spreads retries
await ref.set(
{ failures, nextAttemptAt: Timestamp.fromMillis(Date.now() + delay), leaseUntil: FieldValue.delete() },
{ merge: true },
);
// Returning normally acks the task. The next read after nextAttemptAt retries.
}
}The choice that keeps this cheap is in the catch: an upstream failure is recorded and the task acknowledged. A retry is only worth paying for if someone will read the result. If the record is read again after nextAttemptAt, that read starts a new attempt; if not, the failure costs nothing more. The queue's own retry policy still covers infrastructure failures: a worker that crashes, times out or fails to write to Firestore throws before acknowledging, and Cloud Tasks redelivers.
A 404 from the upstream means the record is gone and deserves a terminal state, not endless backoff; a 429 or 5xx is transient and the backoff covers it. If many records fail at once, that is an outage, and a circuit breaker that pauses enqueues for the whole source is cheaper than letting every record discover it separately.
Manual refresh reuses the same lease and the same enqueue. It skips the TTL check but not the lease, the backoff, or its own cooldown (a lastManualAt field checked in the same transaction). Return the remaining cooldown to the client so the button can say "available again in 6 hours". With a paid upstream, that cooldown is also a per-user spending limit; see protecting pay-per-call APIs.
Pace the queue by the upstream's quota#
gcloud tasks queues create upstream-refresh \
--location=europe-west1 \
--max-dispatches-per-second=5 \
--max-concurrent-dispatches=10 \
--max-attempts=3 \
--min-backoff=30s \
--max-backoff=300s--max-dispatches-per-second and --max-concurrent-dispatches are the queue's rate controls. Set them from the upstream's quota, not from what your worker can handle, because the upstream is the constraint you pay for. A burst of reads after a newsletter then becomes a queue that drains at a steady rate instead of a spike of 429s. The retry flags apply only to the infrastructure failures above.
Cloud Tasks or Pub/Sub for the refresh#
Either works. The comparison comes down to a few properties, all from Google's own Cloud Tasks versus Pub/Sub guide.
| Property | Cloud Tasks | Pub/Sub |
|---|---|---|
| Deduplication at creation | Yes, by task name | No |
| Dispatch rate control | Per queue: rate and concurrency | Not built in; flow control only on pull subscribers |
| Scheduled delivery | Yes, scheduleTime up to 30 days ahead | No |
| Fan-out to several consumers | No, one explicit target | Yes, one topic, many subscriptions |
For refresh-on-read the first three rows decide it. Creation-time dedupe backs up the lease, rate control protects the upstream, and scheduled delivery lets you defer a refresh to the moment a backoff expires without a polling loop.
Pub/Sub is the better choice when the refresh result has to reach several consumers (a search indexer, an analytics sink, a notification service), or when a "record viewed" event is already being published for other reasons. Then the dedupe has to live in the consumer. Ordering keys serialise delivery per record but do not dedupe, and a redelivery replays every later message for that key, including acknowledged ones. The generation check at the top of the worker is what makes the handler safe to run twice, the same approach as in idempotent Pub/Sub consumers. Per-record backoff measured in hours belongs on the document either way.
What the saving looks like#
The shape of the comparison matters more than any particular number, so take one hypothetical: 1,000,000 cached records, where on a typical day 20,000 owners open their own records and 40,000 distinct records are opened by visitors, with an owner TTL of one day and a visitor TTL of three.
| Per month (hypothetical) | Nightly sweep | Refresh-on-read |
|---|---|---|
| Upstream calls | 30,000,000 | at most 0.6M (owners) plus 1.2M (visitors) |
Writes of refreshed data follow the same ratio. Even if every day brought a completely new set of 40,000 visitor records, the total stays below 2 million refreshes against 30 million, and real visitor sets overlap from day to day. Firestore bills per document read and write and Cloud Tasks per billable operation; the upstream's per-call price is usually the largest line.
Refresh-on-read is not free per refresh. Each one adds a lease transaction, a task and a worker invocation on top of the upstream call and write the sweep would also have paid. That overhead is a roughly constant factor per refresh, and the saving comes from doing far fewer of them. If most of your records are read every day, the two designs converge, and the sweep, which batches better, may win.
Where the argument stops holding#
A record that has never been fetched has nothing stale to serve. Choose between fetching synchronously (the reader waits, and a timeout needs a fallback) and returning an empty state with refreshing: true so the client can poll or subscribe to the document. Fetching synchronously when a user links a new record is reasonable, because they are waiting anyway; use the async path everywhere else.
Readers see old data, sometimes for more than one TTL. The reader who triggers the refresh gets the stale snapshot, and only the next reader gets the new one. With a three-day visitor TTL a visitor can see data three days old plus the queue's latency. Show fetchedAt in the UI: stale data shown with its age is honest, stale data presented as current is a bug.
Bots and crawlers trigger refreshes. If every read can start a paid upstream call, a crawler walking your public pages is a cost amplifier. Let only authenticated reads, or reads from verified human sessions, start a refresh; anonymous reads get the snapshot and never a refresh.
Anything computed across records sees data of mixed age. Search indexes, rankings and aggregate statistics need every record reasonably fresh, which on-read refresh deliberately does not give. The same goes for scheduled reports and digests that must be built from current data at a fixed time, for retention deletions and expiry checks that an auditor expects on a schedule, and for alerts on a price threshold or status change, which need polling because nobody is reading the record when it changes.
A small dataset, or one read mostly every day, gains nothing. A nightly batch is simpler and costs about the same.
My own preference is to run both: refresh-on-read for what people look at, and a small, slow sweep for the handful of fields that must be fresh for everyone.
Related articles
All articles →A static bundle in Cloud Storage is cheap to run and easy to get wrong: which Cache-Control values to set, how to upload them, and how Cloud CDN behaves.
8 min
A design review of a Google Cloud image pipeline: signed uploads, fixed variants, immutable names and a CDN, with failure and deletion paths worked through.
15 min
Start with a budget that emails a monitored channel, then add a Pub/Sub relay, usage alerts and a heartbeat that notices silence when that is worth building.
10 min