GDPR deletion is a distributed-systems problem
An erasure request touches your database, search index, buckets, logs, analytics, backups and vendors. Here is how I make that deletion retryable and provable.
An erasure request has a deadline, and the deadline is the first thing to design around. Article 12(3) of the GDPR requires the controller to provide information on the action taken on a request "without undue delay and in any event within one month of receipt of the request", and allows that period to be extended by two further months where necessary, considering the complexity and number of requests, provided the person is told of the extension and the reasons within the first month. The text is on EUR-Lex. A deadline that long is not a reason to relax; it means the deletion must be a process that can fail, retry and be inspected for weeks, not a single statement that either ran or did not. This review takes a hypothetical product on Google Cloud, with data in Firestore, buckets, a search index, BigQuery and a few vendors, and builds that process. On a large-scale social platform I'm building, I budgeted this as a button, a confirmation dialog and one DELETE statement, then spent a multiple of that budget building the workflow described here. It is engineering practice, not legal advice: which data you must erase, keep or anonymise depends on your legal basis and jurisdiction, and a qualified person has to make those calls.
It leans on idempotent Pub/Sub consumers and dead-letter queues; a half-finished deletion is worse than none, because you have told someone it happened.
What the regulation turns into requirements#
Beyond the clock in Article 12(3), three articles shape the machinery (read them yourself on EUR-Lex):
- Article 17(1) gives the right to erasure "without undue delay" when a listed ground applies, for example the data is no longer necessary, or consent is withdrawn and no other legal basis remains.
- Article 17(3) lists exceptions. The one that matters for ordinary products is a legal obligation to keep records, such as invoices and tax documents, so erasure is rarely all or nothing.
- Article 19 asks you to communicate the erasure to each recipient the data was disclosed to, unless that is impossible or disproportionate. In practice that means your processors and third parties you passed data to.
As properties a system can have:
- Complete. Every store holding the person's data is covered, including the ones nobody remembers.
- Idempotent. Running the deletion twice neither fails nor deletes someone else's data.
- Retryable. Search clusters, vendor APIs and storage fail occasionally, and a transient error cannot leave a request stuck silently.
- Observable. At any moment you can say which requests are open, which targets are pending and which are near the deadline.
- Provable. You can show what was deleted, where and when, without the proof becoming a new copy of the personal data.
- Honest about what remains. Backups and legally retained records are not pretended away; each has a documented expiry or reason.
Start with the inventory#
No code fixes a missing inventory. For each store, write down the owner and the delete mechanism:
- Primary database (Firestore or Cloud SQL), including subcollections and denormalised copies in other users' documents.
- Authentication provider and search index, which is often fed by a sync and can hold data the primary store has lost.
- Object storage, caches and CDN, and analytics tables such as BigQuery events keyed by user.
- Logs: request logs with IP addresses, and application logs with whatever someone once printed.
- Backups and exports, including recovery windows and any CSV someone downloaded.
- Processors: email delivery, support desk, payment provider, error tracker.
In my own system one user's data lives in at least four places: the document store; a graph database where the user is a node wired into other people's connections, visible from their side; object storage for uploaded media; and denormalised copies, with the display name and avatar stamped onto things the user touched. Those copies are the right call at read time and a scavenger hunt at deletion time.
Keep it as a typed list in the codebase, not a wiki page, so adding a store means adding an entry in the same pull request.
Why a ledger and a worker per store#
| Approach | Complete | Retryable | Observable | Operational cost |
|---|---|---|---|---|
| Manual runbook with a ticket | Depends on the person | By hand | Ticket status only | Low until volume grows |
| One synchronous API call that deletes everywhere | Only if every call succeeds | No: partial failure leaves a mess | Logs | Low to build, high on first failure |
| Nightly batch over a "pending deletions" table | Yes, if the list is right | Yes, next run | Job logs and table | One job, up to a day of latency |
| Pub/Sub fan-out with a ledger and per-target workers | Yes, enforced by the step list | Yes, per target | Ledger plus alerts | One topic, a subscription per target |
| Commercial privacy platform | Connector coverage | Vendor dependent | Vendor dashboard | Subscription plus integration |
The synchronous call couples the user's latency to the slowest vendor and leaves no record after a partial failure. A nightly batch is reasonable at low volume, but one slow target holds up the rest.
So: an intake endpoint verifies the requester's identity, writes a ledger document and publishes one message to an erasure-requests topic. Each data store has its own subscription and its own small worker. Every worker deletes its target idempotently, then records a step. When the last step lands, the request is marked complete and a proof record exists. The queue only delivers work; the ledger is the source of truth, so if the queue were lost the work could be rebuilt from it.
Backups are deliberately not a worker. I do not try to rewrite backups to remove one person; they expire on a schedule, and a restore procedure re-applies deletions afterwards.
I also put a 30-day grace period in front of the destructive work. A deletion request deactivates the account at once but destroys nothing, and cancelling within 30 days brings it back intact. That window is a separate clock from the dueAt target described below, which measures the purge itself. I added it because it turns an irreversible distributed operation into a reversible one: a bug in the scheduling path destroys nothing, and the real purge runs later from a scheduled job through one audited path. Deletion became a state machine (active, pending deletion, purged, with a route back from pending to active), and every part of the system had to learn what "pending deletion" means.
Topic, subscriptions and dead-lettering#
PROJECT_ID="my-project"
TOPIC="erasure-requests"
URL_BASE="https://erasure-workers-abc123-ew.a.run.app"
PUSH_SA="pubsub-push@${PROJECT_ID}.iam.gserviceaccount.com"
gcloud pubsub topics create "$TOPIC"
gcloud pubsub topics create "${TOPIC}-dlq"
gcloud pubsub subscriptions create "${TOPIC}-dlq-inspect" \
--topic="${TOPIC}-dlq" \
--message-retention-duration=31d \
--expiration-period=never
for TARGET in primary search storage analytics processors; do
gcloud pubsub subscriptions create "${TOPIC}-${TARGET}" \
--topic="$TOPIC" \
--push-endpoint="${URL_BASE}/steps/${TARGET}" \
--push-auth-service-account="$PUSH_SA" \
--ack-deadline=120 \
--min-retry-delay=10s \
--max-retry-delay=600s \
--dead-letter-topic="${TOPIC}-dlq" \
--max-delivery-attempts=10
doneEach subscription receives its own copy of every message, and retries and dead-letters on its own schedule, so a failing vendor API cannot block the storage worker. The IAM grants dead-lettering needs are in the dead-letter queue article; the URLs here are placeholders.
The ledger and the open-request guard#
The ledger must not become a second copy of the personal data it exists to delete. It stores a keyed hash of the user identifier, never a name or email, plus statuses and timestamps. Two kinds of document keep two different invariants:
- A ledger document per request, with a random request ID as its key. The same person can file a second request next year without colliding with the first.
- An open-request guard keyed by the subject hash, which exists only while a request is open. Creating it fails if one is already open, so repeated clicks do not start parallel deletions, and completion deletes it.
While a request is open, the ledger also holds the internal user ID, because the workers and the sweeper need it to act. It is removed in the same transaction that marks the request complete, leaving only the hash.
import { createHmac, randomUUID } from "node:crypto";
import { Firestore, Timestamp } from "@google-cloud/firestore";
import { PubSub } from "@google-cloud/pubsub";
export const TARGETS = ["primary", "search", "storage", "analytics", "processors"] as const;
export type Target = (typeof TARGETS)[number];
const db = new Firestore();
const topic = new PubSub().topic("erasure-requests");
export function subjectKey(userId: string, secret: string): string {
return createHmac("sha256", secret).update(userId).digest("hex");
}
export async function publishRequest(requestId: string): Promise<void> {
await topic.publishMessage({ json: { requestId } }); // the message carries no personal data
}
export async function requestErasure(userId: string, secret: string) {
const key = subjectKey(userId, secret);
const requestId = randomUUID();
const guard = db.collection("erasureOpen").doc(key);
const ledger = db.collection("erasures").doc(requestId);
try {
// One atomic batch: both documents are created, or neither is. create()
// fails with ALREADY_EXISTS if a request for this subject is still open.
await db
.batch()
.create(guard, { requestId, openedAt: Timestamp.now() })
.create(ledger, {
subjectKey: key,
userId, // removed when the request completes
status: "pending",
requestedAt: Timestamp.now(),
dueAt: Timestamp.fromMillis(Date.now() + 30 * 24 * 3600 * 1000),
targets: [...TARGETS],
})
.commit();
} catch (err) {
if ((err as { code?: number }).code === 6) {
const open = await guard.get(); // ALREADY_EXISTS: return the open request
return { requestId: open.get("requestId") as string, duplicate: true };
}
throw err;
}
// Ledger first, publish second. If the publish fails, the request is still
// on record and the sweeper below republishes it.
await publishRequest(requestId);
return { requestId, duplicate: false };
}Writing the ledger before publishing is the important ordering. The reverse can leave a message with no record, or no message at all. This way the worst case is a ledger document with no steps started, which the sweeper finds. The dueAt value is a 30-day internal target, shorter than the legal limit so alerts fire while there is time to fix things.
Workers that can run twice#
Pub/Sub push delivers at least once, so every worker follows one pattern: skip if the step is done, run the deletion, record the step, then check whether the whole request is finished.
import { Firestore, FieldValue } from "@google-cloud/firestore";
import type { Target } from "./intake";
const db = new Firestore();
export async function runStep(
target: Target,
requestId: string,
deleteFn: (userId: string) => Promise<void>,
): Promise<void> {
const erasure = db.collection("erasures").doc(requestId);
const stepRef = erasure.collection("steps").doc(target);
const [request, step] = await Promise.all([erasure.get(), stepRef.get()]);
if (!request.exists || request.get("status") === "complete") return; // late redelivery
if (step.exists && step.get("status") === "done") return;
// Must be idempotent: "already gone" counts as success.
await deleteFn(request.get("userId") as string);
await stepRef.set({ status: "done", completedAt: FieldValue.serverTimestamp() });
await completeIfFinished(requestId);
}
async function completeIfFinished(requestId: string): Promise<void> {
const erasure = db.collection("erasures").doc(requestId);
await db.runTransaction(async (tx) => {
const [request, steps] = await Promise.all([
tx.get(erasure),
tx.get(erasure.collection("steps").where("status", "==", "done")),
]);
const expected: string[] = request.get("targets") ?? [];
if (request.get("status") === "complete") return;
if (steps.size < expected.length) return;
tx.update(erasure, {
status: "complete",
completedAt: FieldValue.serverTimestamp(),
userId: FieldValue.delete(), // only the keyed hash remains
});
tx.delete(db.collection("erasureOpen").doc(request.get("subjectKey")));
});
}The transaction makes completion safe when two workers finish together: exactly one sees all steps done and writes the transition, and the other sees the status already set. The HTTP handler around this is the usual push-consumer shape, with the message decoded from the envelope and a 2xx response as the acknowledgement.
Deleting from each store#
Each function treats "nothing to delete" as success, because retries and replays depend on it.
import { Storage } from "@google-cloud/storage";
import { Firestore } from "@google-cloud/firestore";
import { getAuth } from "firebase-admin/auth";
const db = new Firestore();
const bucket = new Storage().bucket("my-user-uploads");
export async function deletePrimary(userId: string): Promise<void> {
// Deletes the document and every subcollection under it, using a BulkWriter.
await db.recursiveDelete(db.doc(`users/${userId}`));
try {
await getAuth().deleteUser(userId);
} catch (err) {
if ((err as { code?: string }).code !== "auth/user-not-found") throw err;
}
}
export async function deleteStorage(userId: string): Promise<void> {
// Deleting nothing is fine; deleting twice is fine.
await bucket.deleteFiles({ prefix: `users/${userId}/` });
}Two cautions. recursiveDelete does not know about copies of the user's data inside other documents, such as a comment under someone else's post, so the worker needs a delete-by-query or a decision to anonymise. And validate the identifier before building a storage prefix: an empty one would produce users//.
The graph database needed its own answer, because other users' worlds reference the deleted node. I strip the node's personal data and leave an anonymous tombstone, so others see a generic deleted-user placeholder rather than broken references or, worse, someone else's data attributed to the wrong person. Restore has a cost too: deactivation zeroes the denormalised copies, so cancelling a pending deletion has to rehydrate all of them. I shipped a bug in that restore path and found it in beta.
For analytics, a keyed delete in BigQuery is a short job:
bq query --use_legacy_sql=false \
--parameter='user_key:STRING:USER_KEY_VALUE' \
'DELETE FROM `my-project.analytics.events` WHERE user_key = @user_key'This works when the table has a pseudonymous user key and the mapping to the real identity lives in the primary store. Whether pseudonymised data is still personal data is a question for your legal reviewer.
For processors, the worker calls each vendor's deletion API and treats "not found" as success. Many vendors delete asynchronously, so record the vendor's job reference and confirm before marking done. A vendor with no API is an inventory finding: the step becomes a manual task with an owner, tracked in the ledger like any other.
The sweeper: republish first, alert second#
A queue protects against crashes, not against forgetting. An hourly job scans open requests. Anything older than an hour is republished, which covers a failed publish at intake and any message that dead-lettered, and is safe because workers skip finished steps. Anything older than 20 days is logged as an error that a log-based alert turns into a page.
import { Firestore, Timestamp } from "@google-cloud/firestore";
import { publishRequest } from "./intake";
const db = new Firestore();
const HOUR = 3_600_000;
const DAY = 24 * HOUR;
export async function sweep(now = Date.now()): Promise<void> {
const open = await db.collection("erasures").where("status", "==", "pending").get();
for (const doc of open.docs) {
const age = now - (doc.get("requestedAt") as Timestamp).toMillis();
if (age > 20 * DAY) {
// Fixed message so a log-based alert can match it.
console.error(
JSON.stringify({ severity: "ERROR", message: "erasure_overdue", requestId: doc.id, daysOpen: Math.floor(age / DAY) }),
);
}
if (age > HOUR) await publishRequest(doc.id);
}
}The query uses a single equality filter, so it needs no composite index. The alert pattern is in structured logging on a budget. The invariant I care about is that an account still pending deletion well past its window is a violation, and I want to hear about it from my own monitoring rather than from a regulator's letter. Day 20 is my choice: it leaves ten days before the internal target and more before the one-month limit. If a request truly needs the extension, the person has to be told within the first month, so that decision belongs on day 20, not day 29.
Backups: expire, do not rewrite#
Rewriting a backup to remove one person is, in most architectures, impossible or riskier than the data it protects, because backups exist to restore a consistent snapshot. The pragmatic approach is a bounded retention period you can state to the person, combined with a rule that deleted data does not come back.
Firestore scheduled backups have a retention you choose, up to 14 weeks per the documentation:
gcloud firestore backups schedules create \
--database='(default)' \
--recurrence=daily \
--retention=8wCloud Storage soft delete is enabled by default with a seven-day retention, so a deleted object stays recoverable for that period. That helps with accidents and complicates erasure, so choose the duration deliberately instead of inheriting it. The flag takes a duration such as 7d, and --clear-soft-delete removes the setting:
gcloud storage buckets update gs://my-user-uploads --soft-delete-duration=7dPick the shortest window your recovery needs allow and say so in the privacy policy. A person can reasonably be told their data leaves live systems immediately and leaves recoverable copies within a fixed number of weeks, but only if the settings enforce it.
The other half of the rule is the restore procedure. A database restored from a backup taken before an erasure contains someone who asked to be forgotten. After any restore, the runbook walks the restored users, computes the keyed hash of each identifier, looks it up against the subjectKey field of ledger documents requested after the backup date, and files a fresh erasure request through the normal workflow for every match. This is why the ledger keys on a keyed hash: it lets you recognise a person in restored data without storing their identity. The ledger entry is a tombstone and is itself minimal data kept to protect the person, so how long to keep it is a decision for your legal reviewer; I would not keep it longer than the longest backup retention.
Logs and caches#
Logs are the store most easily forgotten and the hardest to clean afterwards. Cloud Logging's _Default bucket keeps entries for 30 days unless changed, and _Required, which holds admin activity audit logs, keeps them for 400 days and cannot be changed. I do not try to delete individual log entries per request. Keep personal data out of logs in the first place, log identifiers rather than names and emails, and set retention to the shortest period that supports debugging:
gcloud logging buckets update _Default --location=global --retention-days=30That makes the answer about logs a fixed, documented retention period, the same expire-rather-than-rewrite logic as for backups. Restrict who can read request logs, which include client IP addresses, and give cached personal data a time-to-live no longer than your response window.
Proof of completion#
When the last step lands, the ledger already contains the proof: targets, a completion time for each and an overall completion time. The reply to the person comes from that record: when the request was received and completed, which categories of system were covered, which retained records remain and why (invoices held under a legal duty, for example), and the backup and log expiry windows. The record holds only the keyed hash, dates and statuses, so it is evidence without recreating what it proves was deleted.
What breaks#
Late-arriving writes. A background sync, a retried message or an in-flight upload can write the person's data after the worker deleted it, and everything looks complete. Block it at the source: while a guard document exists, write paths check for it and reject or drop new data for that subject; after completion, a periodic re-run of the workers for requests completed in the last week catches stragglers.
Identity verification. An erasure endpoint that trusts a bare email address lets anyone delete anyone. Verify the requester through an authenticated session or a confirmed email link before anything is written.
Wrong deletes. A bug in a prefix or a query can erase the wrong person's files, and retrying cannot undo it. Validate identifiers strictly, run workers in a dry-run mode against staging data, and treat the soft-delete window as a safety net while you decide its length.
Too much machinery. If you hold data about a few hundred people in two systems, a documented runbook with a ticket per request and a calendar reminder is cheaper and easier to get right. If you mostly act as a processor for customers, the request normally reaches the controller first, and your obligation is a way to delete on their instruction, which can be a simple authenticated API. Build the workflow when volume, the number of stores or the cost of a missed deadline make the manual process fragile.
Related articles
All articles →Start with a budget that emails a monitored channel, then add a Pub/Sub relay, usage alerts and a heartbeat that notices silence when that is worth building.
10 min
A scheduled sync pays to refresh every record, including the ones nobody reads. Refreshing stale data when someone looks makes cost follow attention.
15 min
Batch work behaves badly behind an HTTP endpoint. Compare Cloud Run jobs and services on retries, parallelism and exit codes, and shard work across tasks.
12 min