End-to-end tests against a live staging environment

Mocks pass while real systems fail. How I run a small Playwright suite against a deployed, no-traffic staging revision and gate promotion on the result.

Published

Updated

Reading time

11 min

Some failures cannot occur on a laptop because the thing that fails does not exist there: a service account missing one permission, a CORS header added only by the production load balancer, a database index created in one environment and not another. Unit tests with mocks and integration tests against emulators encode your assumptions about those things, so they pass while the deployment is broken. On a large-scale social platform I'm building, every service had passing unit tests for a stretch while real requests through staging kept failing, and the failures were all in the seams between services.

This walkthrough builds a thin layer that tests the deployment itself. Each merge to main deploys a candidate Cloud Run revision that receives no user traffic, a small Playwright suite runs against that revision's own URL, and traffic moves only if the suite is green. The pieces are Cloud Run, GitHub Actions and Playwright, but each has an equivalent elsewhere.

Decide what the suite is for#

The suite answers one question: does what we just deployed work end to end, with real identity, real network path and real data stores? Business rules belong in faster tests. That keeps the suite to ten or fifteen tests that finish in a few minutes, and a suite that finishes in forty minutes gets bypassed.

Three bugs of mine made the case, and none was findable by the unit tests as written. A service built graph database queries and its tests mocked the driver, so the query text looked right while the real database rejected it with a syntax error and every request down that path returned a 500. An update endpoint worked on a fresh fixture but failed with a field collision once an earlier write had shaped the document, which is the normal order for a real user. And a frontend test fed a component data shaped the way the frontend believed the API responded, after the API had moved on, so both suites passed against their own copy of the contract.

ApproachCatches environment bugsMain weakness
Unit tests with mocksNoMocks encode your assumptions
Local emulatorsPartlyEmulators differ from the real service
Ephemeral environment per pull requestYesProvisioning time, and a second copy of every data store
Shared staging, tests after deployYesShared state and flaky data
Synthetic checks in production onlyYesFinds the bug after users do
Table 1— Where an environment bug can be caught

For a handful of services, shared staging with a gated promotion gives the best return. The tests I would put in it are the ones only a deployed environment can break: the front door through the real ingress, a login against the real identity provider, one write path for the flow that matters most (create, read back, delete), one asynchronous path asserted by its visible outcome with a bounded wait, a browser call from the real front-end origin to the API to surface CORS mistakes, and a refused request from an unauthenticated or wrong-role caller. Everything else is somebody else's test.

I chose shared staging over a fresh stack per run because a hermetic copy of my platform would need a graph database, a document database, a pub/sub emulator, a dozen services and a gateway every time, and emulators bring back the fidelity gap I am trying to close. The price is that tests must tolerate leftover data and test accounts with history, and occasionally a sweep fails because staging is mid-deploy. Two habits make that workable: each test asserts only against state it created itself, and I investigate a flaky test as a probable race in the product instead of retrying it into silence. I also go through the public gateway rather than calling services directly, because direct calls skip auth enforcement and route definitions, and a route missing its security stanza at the gateway is broken for users however well the service behind it behaves. For the asynchronous path I assert on the downstream state change, since a 200 only says the request was accepted; that is how a queue with no subscribers shows up, with every call reporting success and nothing happening.

The suite sits between deploy and promote. A failure leaves the previous revision serving traffic.

Make the base URL impossible to forget#

The most damaging suite is one that quietly tests the wrong environment. The config therefore refuses to run without an explicit URL, and sets the options that stop a CI run from lying.

playwright.config.tsts
import { defineConfig, devices } from "@playwright/test";
 
const baseURL = process.env.E2E_BASE_URL;
if (!baseURL) throw new Error("E2E_BASE_URL is required");
 
export default defineConfig({
  testDir: "tests",
  globalTimeout: 15 * 60 * 1000,
  timeout: 45_000,
  expect: { timeout: 10_000 },
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 2 : undefined,
  reporter: process.env.CI ? [["list"], ["html", { open: "never" }]] : "list",
  use: {
    baseURL,
    trace: "on-first-retry",
    extraHTTPHeaders: { "x-e2e-run": process.env.GITHUB_RUN_ID ?? "local" },
  },
  projects: [
    { name: "setup", testMatch: /.*\.setup\.ts/ },
    {
      name: "chromium",
      use: { ...devices["Desktop Chrome"], storageState: "playwright/.auth/user.json" },
      dependencies: ["setup"],
    },
  ],
});
  • E2E_BASE_URL is mandatory. The config throws instead of defaulting to localhost.
  • globalTimeout caps the whole run, as Playwright's CI guide recommends, so a hung run ends and still produces a report.
  • forbidOnly fails the run if a stray test.only is committed, which would otherwise make a one-test suite look green.
  • retries: 1 with trace: "on-first-retry" records a trace only for tests that failed once. A test that fails and then passes is listed as flaky in the summary, which is the number to watch (see the retries documentation).
  • extraHTTPHeaders stamps requests with the workflow run ID, so test traffic is easy to find if your service copies the header into a log field.
  • workers: 2 keeps load on a scaled-to-zero staging service gentle. More parallelism mostly measures contention on shared data.

Log in once, through the API#

Logging in through the UI in every test is slow and makes a login-page change fail the whole suite. Playwright's authentication guide describes a setup project that authenticates once and saves the browser state for the other tests:

tests/auth.setup.tsts
import { test as setup, expect } from "@playwright/test";
 
const authFile = "playwright/.auth/user.json";
 
setup("authenticate", async ({ request }) => {
  const res = await request.post("/api/session", {
    data: {
      email: process.env.E2E_USER_EMAIL,
      password: process.env.E2E_USER_PASSWORD,
    },
  });
  expect(res.ok()).toBeTruthy();
  await request.storageState({ path: authFile });
});

The endpoint is an example; use whatever your service exposes, and keep one separate test that logs in through the real UI. Add playwright/.auth to .gitignore, because the file can hold cookies that impersonate the user, and use a dedicated test account whose role can reach only test data.

Staging scales to zero, so the first request after idle is slow. Make the setup project's first call a health request with a generous timeout so later tests hit a warm instance (the causes are covered in Cloud Run cold starts).

Write tests that clean up after themselves#

tests/orders.spec.tsts
import { test, expect } from "@playwright/test";
 
test("a created order shows up in the list", async ({ request }) => {
  const sku = `e2e-${Date.now()}`;
  const created = await request.post("/api/orders", {
    data: { sku, quantity: 1, currency: "EUR" },
  });
  expect(created.status()).toBe(201);
  const { id } = await created.json();
 
  try {
    const list = await request.get("/api/orders");
    const orders: { id: string }[] = await list.json();
    expect(orders.some((o) => o.id === id)).toBe(true);
  } finally {
    await request.delete(`/api/orders/${id}`);
  }
});
 
test("the dashboard renders for a signed-in user", async ({ page }) => {
  const errors: string[] = [];
  page.on("pageerror", (e) => errors.push(e.message));
 
  await page.goto("/");
  await expect(page.getByRole("heading", { level: 1 })).toBeVisible();
  expect(errors).toEqual([]);
});

The first test uses the request fixture, which sends HTTP without a browser and is much faster than driving the UI for a write path. The second proves the page loads in a real browser with no uncaught script errors, which is where mismatched API origins and missing headers surface. Anything a test creates carries an e2e- prefix and a unique value, and the finally block deletes it even when the assertion fails. A nightly job that deletes e2e- records older than a day catches the runs that were killed mid-test. For asynchronous outcomes use expect.poll or a web-first assertion with a bounded timeout, never waitForTimeout.

Deploy the candidate with no traffic and a tag#

Cloud Run's rollout documentation describes the mechanism: --no-traffic with --tag deploys a revision that serves nobody, and the tag gives it a URL that starts with the tag name followed by three hyphens, for example https://green---myservice-abcdef.a.run.app. Tags must be lowercase. Promotion is then gcloud run services update-traffic SERVICE --to-tags TAG=100.

.github/workflows/staging.ymlyaml
name: staging
on:
  push:
    branches: [main]
 
concurrency:
  group: staging
  cancel-in-progress: false
 
permissions:
  contents: read
  id-token: write
 
env:
  SERVICE: web
  REGION: europe-west1
 
jobs:
  deploy-and-test:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
          cache: npm
      - uses: google-github-actions/auth@v3
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.DEPLOY_SA }}
      - uses: google-github-actions/setup-gcloud@v3
 
      - name: Deploy candidate with no traffic
        id: deploy
        run: |
          TAG="c-${GITHUB_SHA::7}"
          gcloud run deploy "$SERVICE" \
            --image "$IMAGE" \
            --region "$REGION" \
            --no-traffic --tag "$TAG"
          URL="$(gcloud run services describe "$SERVICE" --region "$REGION" \
            --format='value(status.traffic.filter(tag="'"$TAG"'").url)')"
          echo "tag=$TAG" >> "$GITHUB_OUTPUT"
          echo "url=$URL" >> "$GITHUB_OUTPUT"
        env:
          IMAGE: europe-west1-docker.pkg.dev/${{ vars.PROJECT_ID }}/app/web:${{ github.sha }}
 
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: End-to-end tests
        run: npx playwright test
        env:
          E2E_BASE_URL: ${{ steps.deploy.outputs.url }}
          E2E_USER_EMAIL: ${{ secrets.E2E_USER_EMAIL }}
          E2E_USER_PASSWORD: ${{ secrets.E2E_USER_PASSWORD }}
 
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14
 
      - name: Promote
        run: |
          gcloud run services update-traffic "$SERVICE" --region "$REGION" \
            --to-tags "${{ steps.deploy.outputs.tag }}=100"

The action majors (checkout 7, setup-node 7, upload-artifact 7, google-github-actions/auth 3, setup-gcloud 3) are the current releases of each at the time of writing. Details that matter:

  • Keyless authentication. The workflow uses Workload Identity Federation, as described in keyless CI/CD for GitHub Actions. The deploy service account needs to deploy and to act as the runtime service account, nothing more.
  • Resolving the tagged URL. The describe call filters status.traffic for the entry with the tag and reads its url. If it returns nothing, inspect the real field layout with gcloud run services describe web --region europe-west1 --format=json.
  • Serialised deploys. With cancel-in-progress: false, two quick merges queue instead of racing to move traffic. GitHub keeps one running and one pending run per group and a newer pending run replaces an older one, so the latest commit wins. See GitHub's concurrency documentation.
  • The report uploads even on failure, using the !cancelled() pattern from Playwright's CI guide.
  • Promotion is last. Steps stop at the first failure, so a red run never reaches it. The tagged revision stays deployed with no traffic for debugging until you remove the tag with gcloud run services update-traffic ... --remove-tags.
  • Access. The runner must be allowed to call the tagged URL. If staging requires Cloud Run IAM authentication, the tests have to send an identity token.

The very first deploy#

A brand-new service has no earlier revision to keep serving, so there is nothing for a no-traffic candidate to be held back from. I bootstrap the service once by hand, with a normal deploy, and only then switch the pipeline on. If the pipeline must handle both cases, check whether the service exists with gcloud run services describe and omit --no-traffic and --tag when it does not; every later run takes the gated path.

Carry the same image to production#

Staging passing says nothing about production configuration, so production repeats the gate with a smaller check. The production job takes the image digest that passed in staging, deploys it with no traffic, runs a short smoke test against its tagged URL, and then promotes:

.github/workflows/production.yml (job steps, abridged)yaml
      - name: Deploy the staged image with no traffic
        id: deploy
        run: |
          gcloud run deploy "$SERVICE" --image "$IMAGE_DIGEST" --region "$REGION" \
            --no-traffic --tag "p-${GITHUB_SHA::7}"
      - name: Smoke test (health route and one read-only call)
        run: npx playwright test --project=smoke
        env:
          E2E_BASE_URL: ${{ steps.deploy.outputs.url }}
      - name: Promote
        run: |
          gcloud run services update-traffic "$SERVICE" --region "$REGION" \
            --to-tags "p-${GITHUB_SHA::7}=100"

Production smoke tests are read-only. They do not create records, because there is no staging sweeper to clean up after them.

Where it will bite you#

  • Staging drift. A pass means nothing if staging differs from production. Keep configuration in the same mechanism for both and compare effective settings. IAM is the usual difference, so a periodic policy comparison pays off (see GCP IAM least privilege in practice).
  • Third-party providers. Payment and email providers must run in test mode. A test that sends real email or charges a real card turns a typo into an incident.
  • Migrations. A new revision may run against a database that already has, or does not yet have, a column. The gate catches the second case only if migrations run before the suite, so make that order explicit in the pipeline.
  • Flakiness. If the suite fails one run in ten for no reason, people rerun it, then bypass it. Count flaky tests from the report summary and fix each one. Most flaky end-to-end tests wait for the wrong thing, so prefer assertions that retry until a condition holds.
  • Shared fixtures. Two tests editing one record make the suite order-dependent. Use serial mode only for a short flow that truly must run in sequence, because it reruns the whole group on retry.

If the product is a static site with no backend, a link checker and build-time assertions give most of the value. If a platform cannot start a revision without exposing it, you can still run the suite after rollout, but then it is a smoke test with a rollback plan, not a gate, and the runbook should say so.

Share
All articles →

Why a scheduled job can fail for weeks without anyone noticing, and the retry, idempotency and alerting setup that makes Cloud Scheduler failures loud.

13 min

JSON logs that Cloud Logging understands, correlated by trace, with the noise excluded before it is billed. A pino setup for Cloud Run and what it costs.

12 min