Firestore data modelling for a million users

The Firestore limits that bite as a database grows, and the schema choices that avoid them: hot documents, index fan-out, ramp-up, pagination, rules and TTL.

Published

Updated

Reading time

13 min

Firestore rarely hurts at a million users because it is slow. It hurts because a schema that felt natural at a thousand users quietly depends on a limit: a document that every user writes to, a query nobody indexed, a field that only ever increases. By the time the limit is visible there is production data and a migration to plan.

What follows is a set of those limits, each with what you see when you hit it, why it happens and how to avoid it. The examples come from a hypothetical shop: orders, products, reviews and likes. For feed-shaped data, where the main problem is fan-out, see social feed fan-out, write versus read; this piece deals with the general mechanics.

The limits are from the official quotas and best practices pages. Limits change, so check the numbers before you design around the edge of one.

Start from the query list#

Firestore serves queries from indexes only, and bills reads per document returned. So the schema follows from the list of queries, which you write first. For the shop:

  1. A user's orders, filtered by status, newest first, paged.
  2. The reviews of one product, newest first, paged.
  3. Everything one user has reviewed, across products.
  4. The total number of likes on a product, shown on every card.
  5. Orders older than a retention window, deleted automatically.

Each line is a filter, an order and a page size, and those three pick the index, which picks the structure. A screen that is not on the list gets added to the list before it gets built.

Figure 1— Each query gets a structure; each derived value gets a named maintainer.

The structure choice itself is a small decision table:

StructureGood atWeak at
Top-level collection (orders/{id})Queries across all owners; documents with several parentsPer-owner isolation needs a uid field and a rule that checks it
Subcollection (products/{id}/reviews/{id})Unbounded lists always read by parent; ownership visible in the pathCross-parent queries; moving a document to another parent
Collection group (every collection named reviews)The same list scoped by parent in one place and unscoped in anotherNeeds its own index scope and its own rules match
Array or map inside the parentSmall bounded data always read with the parentGrowth, partial updates, index entry limits

Embed what is bounded and read together (a shipping address, the line items of one order). Use a subcollection for lists that grow. Use a top-level collection when an entity is queried from several directions, as orders are by user, status and date. Add a collection group query for the other side of a subcollection, such as every review by one user. It reads across collections; it does not change how anything is stored.

Trap: the document that grows#

What you see. Reads get slower and more expensive over time, writes to one document take longer, and eventually a write fails.

Why. A document is capped at 1 MiB including field names, and every update rewrites and re-indexes the whole document and every read ships all of it. A product that embeds all its reviews in an array is a bug waiting to arrive, and it costs money long before it fails.

Fix. Move unbounded lists to subcollections. If the UI needs a preview, keep a bounded copy in the parent, for example the three latest reviews, maintained by a trigger.

Trap: one document, many writers#

What you see. Writes to a popular product become slow, transactions on it abort and retry, and the problem shows up first on your best-selling item.

Why. The quotas page gives the maximum sustained write rate to a single document as about one per second. Short bursts above that often succeed, but contention shows up as latency and aborted transactions. A likes field incremented on every like, or a reviewCount updated on every review, makes the product document the bottleneck.

Fix. Two tools, for different jobs.

An aggregation query computes a total on demand without downloading documents:

reviews-count.tsts
import { getFirestore } from "firebase-admin/firestore";
 
const db = getFirestore();
 
export async function reviewCount(productId: string): Promise<number> {
  const snap = await db
    .collection(`products/${productId}/reviews`)
    .where("status", "==", "published")
    .count()
    .get();
  return snap.data().count;
}

It is billed by index entries, one read per batch of up to 1,000 entries matched (see aggregation queries), so a count over a large result costs a small number of reads. It cannot drive a realtime listener, so it suits "the total on the detail page", not "a live counter on every card".

A distributed counter suits totals that are shown constantly and updated often. The documented pattern splits one counter into shard documents: a write picks a random shard and a read sums them all.

likes-counter.tsts
import { getFirestore, FieldValue } from "firebase-admin/firestore";
 
const db = getFirestore();
const SHARDS = 10; // size to the peak write rate of your hottest item
 
const shards = (productId: string) =>
  db.collection(`products/${productId}/likeShards`);
 
export async function addLike(productId: string, delta: 1 | -1 = 1) {
  const shardId = String(Math.floor(Math.random() * SHARDS));
  await shards(productId)
    .doc(shardId)
    .set({ count: FieldValue.increment(delta) }, { merge: true });
}
 
export async function likeTotal(productId: string): Promise<number> {
  const snap = await shards(productId).get();
  return snap.docs.reduce((sum, d) => sum + (d.get("count") as number), 0);
}

increment is atomic on the server, so no transaction is needed. The catch is the read side: ten shards means each total costs ten document reads, and a list of fifty cards just became five hundred. The fix is a trigger that periodically folds the shards into one likesApprox field on the product, so cards read one value and the exact sum is used only when exactness matters. Most items need a single shard; since reads sum whatever shards exist, you can raise the count later.

A stored counter costs a write per event plus a read per display. A sharded one multiplies the display read by the shard count. For a total displayed constantly and changed rarely, the stored or folded counter wins by a wide margin. For a total shown once a day on an admin page, count on demand and skip the bookkeeping.

Trap: keys and timestamps that only go up#

What you see. Write latency climbs during an import or a launch, even though the data is spread over many documents.

Why. Firestore splits data by key range across servers. Monotonically increasing keys, such as Customer1, Customer2, send every new write to the same range. The same happens in indexes: a field that always increases, like a createdAt timestamp, is a hotspot in its index. The best practices page gives a limit of 500 writes per second to a collection for indexed fields that increase or decrease sequentially.

Fix.

  • Use auto-generated document ids (collection.doc() in the Admin SDK) or random ones.
  • If you never query on a timestamp field, exempt it from indexing.
  • If you do query on it, plan around the per-collection ceiling, or spread writes over collections that have their own index ranges, such as per-user subcollections.
  • Follow the documented 500/50/5 ramp-up for imports and launches: start at 500 operations per second and raise traffic by at most 50 percent every five minutes. The Admin SDK's BulkWriter applies this ramp for you; steady-state traffic is not affected.

Trap: indexes you did not ask for#

What you see. Writes get more expensive for no visible reason, or a write fails because a document has too many index entries.

Why. Every field gets single-field indexes automatically, and each composite index adds entries on top. A document may have at most 40,000 index entries across all of them, and arrays and maps multiply entries quickly because each element or sub-field is indexed. A large free-text field, a blob of serialised JSON or an array of tags you never filter on costs write time and storage for nothing.

Fix. Declare composite indexes in a file under version control, and exempt fields you never query. The first read path in the shop needs a composite index on uid, status and createdAt; the review lookup by author needs a collection-group-scoped one:

firestore.indexes.jsonjson
{
  "indexes": [
    {
      "collectionGroup": "orders",
      "queryScope": "COLLECTION",
      "fields": [
        { "fieldPath": "uid", "order": "ASCENDING" },
        { "fieldPath": "status", "order": "ASCENDING" },
        { "fieldPath": "createdAt", "order": "DESCENDING" }
      ]
    },
    {
      "collectionGroup": "reviews",
      "queryScope": "COLLECTION_GROUP",
      "fields": [
        { "fieldPath": "authorUid", "order": "ASCENDING" },
        { "fieldPath": "createdAt", "order": "DESCENDING" }
      ]
    }
  ],
  "fieldOverrides": [
    {
      "collectionGroup": "orders",
      "fieldPath": "internalNotes",
      "indexes": []
    }
  ]
}

Equality fields come first, then the field you order or range on, and a different filter combination is a different index. A collection group query needs COLLECTION_GROUP scope, because a plain collection index does not serve it. The override with an empty indexes array exempts internalNotes entirely, which also cuts index storage.

Indexes build asynchronously, and a query against one that is still building fails. In a pipeline, deploy indexes first, wait until they report ready, then ship the code that needs them:

Deploy and compare indexesbash
firebase deploy --only firestore:indexes
firebase firestore:indexes > live-indexes.json

Keep the JSON file as the source of truth and avoid creating indexes by hand in the console, or the repository and the database drift apart. A new filter combination in code fails at runtime until its index exists, so review index changes in the same pull request as the query, and review the list now and then to drop unused composite indexes, since each costs on every write.

Trap: offsets and ties in pagination#

What you see. Pages cost more the deeper the user scrolls, or a row appears twice or vanishes at a page boundary.

Why. An offset makes the database read and skip the documents before the page, and you pay for the skipped reads. And if the order field is not unique, two orders created in the same millisecond make the boundary ambiguous.

Fix. Use a cursor, and always end the order with the document id. In an API you cannot hand a snapshot to a client, so the cursor encodes the ordering values:

orders-query.tsts
import { getFirestore, FieldPath, Timestamp } from "firebase-admin/firestore";
 
const db = getFirestore();
const PAGE = 20;
 
type Order = { id: string; createdAt: Timestamp };
 
export function encodeCursor(createdAt: Timestamp, id: string): string {
  return Buffer.from(JSON.stringify([createdAt.toMillis(), id])).toString("base64url");
}
 
function decodeCursor(token: string): [Timestamp, string] {
  const parsed = JSON.parse(Buffer.from(token, "base64url").toString());
  if (!Array.isArray(parsed) || typeof parsed[0] !== "number" || typeof parsed[1] !== "string") {
    throw new Error("bad cursor"); // map to a 400, not a 500
  }
  return [Timestamp.fromMillis(parsed[0]), parsed[1]];
}
 
export async function ordersPage(uid: string, status: string, cursor?: string) {
  let q = db
    .collection("orders")
    .where("uid", "==", uid)
    .where("status", "==", status)
    .orderBy("createdAt", "desc")
    .orderBy(FieldPath.documentId(), "desc") // tiebreaker
    .limit(PAGE);
  if (cursor) q = q.startAfter(...decodeCursor(cursor));
 
  const snap = await q.get();
  const orders = snap.docs.map((d) => ({ id: d.id, ...d.data() }) as Order);
  const last = orders.at(-1);
  return {
    orders,
    cursor: orders.length === PAGE && last ? encodeCursor(last.createdAt, last.id) : undefined,
  };
}

The tiebreaker needs no extra index. A composite index includes the document id as an implicit last field, in the direction of the last ordered field, so the index above serves this query. (If you pass a document snapshot to startAfter on the server, Firestore applies the id tiebreak itself; the explicit form is for cursors that cross an API boundary.) How cursors behave when a list is merged from several sources is covered in the feed article.

Trap: copies with no maintainer#

What you see. A user changes their display name and the old one shows up in some places for weeks.

Why. Firestore has no joins, so you copy fields: names, avatars, product titles, counts. That is the point of a document store, and it is also a debt paid on every write that changes the source. When a copy has no named owner, a failed trigger or a skipped code path leaves it stale and nothing notices.

Fix. For every denormalised field, write down the source document, where the copies live, who updates them and how stale a copy may be. Three kinds of maintainer cover almost everything:

  1. The writer, in the same transaction or batch, when the copies live in a bounded number of documents and must never disagree, such as an order and its owner's orderCount.
  2. A trigger on document write that fans changes out. It runs after commit, at least once and unordered, so it must be idempotent, which is the subject of idempotent Pub/Sub consumers.
  3. Refresh on read, for copies that may be stale, as in refresh on read instead of cron.

If you cannot name the maintainer, do not copy the field; read it from the source. Keep a repair job that recomputes a copy from its source, because a copy you cannot rebuild is a second source of truth. When to denormalise goes further into the trade.

Trap: rules are not filters#

What you see. A client query that looks harmless is rejected with a permission error, although every document it would return is readable.

Why. Firestore evaluates the query against the rule before running it and rejects any query that could return a forbidden document. A rule saying "read if resource.data.uid equals the caller" lets the client list orders only if the query itself includes where("uid", "==", uid). That forces the query list to match the access model, which is a feature, but it surprises people who treat rules as a post-filter.

firestore.rulestext
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
 
    match /orders/{orderId} {
      allow read: if request.auth != null && resource.data.uid == request.auth.uid;
      allow create: if request.auth != null
                    && request.resource.data.uid == request.auth.uid
                    && request.resource.data.status == 'pending';
      allow update, delete: if false; // changed by the server only
    }
 
    match /products/{productId}/reviews/{reviewId} {
      allow read: if true;
    }
 
    // Needed for collection group queries over every "reviews" collection.
    match /{path=**}/reviews/{reviewId} {
      allow read: if request.auth != null && resource.data.authorUid == request.auth.uid;
    }
  }
}

Fix. Design rules and structure together. Collection group queries need their own match: the /{path=**}/reviews/{reviewId} line, available from rules version 2, is what permits a query across every collection called reviews, and without it the query is denied even when the per-product rule allows reads. Be careful with wildcards: a recursive match that is too broad opens every collection with that name, including ones added later. Rules that call get() or exists() on other documents are billed as reads, so keep the owner on the document itself, as uid is here. State the client must not set, such as status transitions, goes through the Admin SDK, which bypasses rules, which is why the example denies client updates.

Trap: TTL is not an immediate delete#

What you see. Expired documents still come back in queries, or orphaned child documents remain after a parent expires.

Why. A TTL policy deletes documents some time after the value of a timestamp field passes. It is a retention tool, not a correctness mechanism.

Enable a TTL policybash
gcloud firestore fields ttls update expireAt \
  --collection-group=orders \
  --enable-ttl

The behaviour to plan around:

  • The field must hold a timestamp. A missing or non-timestamp value means the document is never deleted.
  • Deletion typically happens within about 24 hours of expiry, and expired documents can still be returned by queries until then. If expiry is a correctness rule, such as a token that must stop working, also filter on expireAt in the query.
  • TTL deletes are billed as deletes, which on a large collection is a real cost line. Choose the retention period deliberately.
  • TTL does not delete the subcollections of a document it removes. Give child documents their own expireAt.

Before you lock the schema#

Run the query list against the structure one more time and check each line against these traps: no document grows without bound or takes sustained writes above about one per second, every document id is auto-generated, every denormalised field has a maintainer, every index is in the repository, and every correctness rule is enforced by rules or queries, not by TTL timing. If the product is small and likely to stay small, a top-level layout with auto ids and a handful of indexes avoids most of this. If the data is deeply relational, with many-to-many joins and ad hoc reporting, a relational database is the better home.

Share
All articles →