Budget alerts that page you before the bill does

Start with a budget that emails a monitored channel, then add a Pub/Sub relay, usage alerts and a heartbeat that notices silence when that is worth building.

Published

Updated

Reading time

10 min

Take a small Google Cloud account: one production project, a monthly budget, and the default setup. An email goes to billing administrators at 50, 90 and 100 percent of spend, into an inbox nobody reads on a Saturday. By the time the 100 percent message arrives the money has been spent, and the billing data behind it was already delayed when it was computed. This design review starts from that account and works up: what a budget does and does not do, the simplest arrangement that reaches a person, the relay worth building when that is not enough, the usage alerts that fire earlier than any cost alert can, and the heartbeat that catches the whole pipeline going quiet. It is the budget end of the monthly review in reading a GCP bill like an architect.

A budget is a notification, not a limit#

These behaviours come from the Cloud Billing budgets documentation and the programmatic notifications page, and they shape everything below.

  • An alerts-only budget caps nothing. It does not automatically cap usage or spending. Spend cap budgets that can pause services exist, but only for eligible services.
  • Data is delayed. Usage costs reach Cloud Billing after the resources are used, the first notification after creating a budget can take several hours, and a monthly budget keeps tracking for the first two days of the next month to absorb late-reported usage and taxes.
  • Default thresholds are 50, 90 and 100 percent of actual spend. You can add forecast thresholds, which fire when projected spend by the end of the period exceeds the threshold. Budgets with a custom time range cannot trigger forecast alerts.
  • Day boundaries are Pacific time. Calendar and custom ranges start at midnight US and Canadian Pacific Time, so a monthly rollover is not at midnight where you live.
  • Pub/Sub messages are not emails. They arrive multiple times per day with the budget's current status, whether or not a threshold has been crossed and even with no usage, and delivery is at least once, so duplicates and out-of-order messages happen.
  • A broken topic is silent. If the topic is deleted, Cloud Billing loses publish permission, the notification quota is exceeded or a domain-restriction organisation policy blocks publishing, nothing tells you.

Two consequences drive the design. Pub/Sub messages repeat regardless of thresholds, so something must decide which ones deserve a human. And failure is silent, so something must notice when messages stop.

The simplest version that works#

Before any code, three settings fix most of the weakness of the default:

  1. A forecast threshold at 100 percent alongside the actual ones. It fires when the month is on track to exceed the budget even though the money has not been spent, which is the earliest cost-based signal a budget can give.
  2. A Cloud Monitoring email notification channel attached to the budget, pointing at an address someone reads. The gcloud reference allows up to five channels per budget. This replaces the shared inbox with a route you chose.
  3. --max-instances on every Cloud Run service. An alert tells you a loop is running; an instance cap bounds what the loop can spend while you wake up.
terminalbash
gcloud billing budgets create \
  --billing-account=BILLING_ACCOUNT_ID \
  --display-name="monthly-all-projects" \
  --budget-amount=150EUR \
  --calendar-period=month \
  --threshold-rule=percent=0.5 \
  --threshold-rule=percent=0.9 \
  --threshold-rule=percent=1.0 \
  --threshold-rule=percent=1.0,basis=forecasted-spend \
  --notifications-rule-monitoring-notification-channels=projects/finops-admin/notificationChannels/CHANNEL_ID

The flags are from the gcloud reference: --threshold-rule is repeatable, percent is a fraction from 0.0 to 1.0, and basis is current-spend by default or forecasted-spend. The amount is an example; use your own net baseline plus a margin. For a second budget scoped to one risky project, add --filter-projects=projects/PROJECT_ID: per-project budgets turn "the bill is high" into "this project is high", which is most of the diagnosis.

If the account's bill is a few euros and a promotional credit covers it, stop here. Everything below is for the case where the email is not enough: you want chat or phone delivery, one message per threshold instead of the budget's own cadence, or proof that the pipeline is alive.

Four layers when it is worth building#

Cost alerts through a relay, usage alerts directly, and a heartbeat that watches the relay.

The layers, each doing one job: a budget connected to a Pub/Sub topic; a relay that turns repeating status messages into one chat message per threshold per month; usage alerts in Cloud Monitoring on the things that spike before the bill does; and a heartbeat that complains when budget messages stop arriving. Usage alerts are additive, since they catch a different class of problem earlier.

I would not automate anything destructive from these alerts. A function that disables billing on its own can take production down at the worst moment, and I would rather be paged.

Connect the topic#

Put billing topics in an administration project rather than a workload project, as the documentation recommends, so deleting a workload project cannot delete the alerting path:

terminalbash
gcloud pubsub topics create billing-alerts --project=finops-admin

Then connect it to the budget from the console: open the budget, and under Manage notifications choose Connect a Pub/Sub topic, then pick the project and topic. The documented requirement is that whoever does this must have permission to grant the Pub/Sub Publisher role on the topic, so that Cloud Billing can publish to it. Allow several hours before the first message, and treat its arrival as the test: a topic that never receives anything is exactly what the heartbeat below is for.

The relay#

Create a push subscription that calls a small Cloud Run service with an OIDC token (subscriptions reference):

terminalbash
gcloud run deploy budget-relay \
  --project=finops-admin \
  --region=europe-west4 \
  --source=. \
  --no-allow-unauthenticated \
  --max-instances=2 \
  --set-secrets=CHAT_WEBHOOK_URL=chat-webhook-url:latest
 
gcloud run services add-iam-policy-binding budget-relay \
  --project=finops-admin \
  --region=europe-west4 \
  --member=serviceAccount:budget-relay-invoker@finops-admin.iam.gserviceaccount.com \
  --role=roles/run.invoker
 
gcloud pubsub subscriptions create billing-alerts-push \
  --project=finops-admin \
  --topic=billing-alerts \
  --push-endpoint=https://budget-relay-HASH-ew.a.run.app/pubsub \
  --push-auth-service-account=budget-relay-invoker@finops-admin.iam.gserviceaccount.com \
  --ack-deadline=30

The chat webhook URL is a secret, so it comes from Secret Manager rather than the repository; configuration and secrets in Node services covers the pattern. Replace HASH with the URL the deploy prints. If you add a dead-letter topic, see dead-letter queues as a safety net.

The decision rule is the heart of the relay. A message's attributes carry billingAccountId, budgetId and schemaVersion. Its JSON data carries budgetDisplayName, costAmount, costIntervalStart, budgetAmount, budgetAmountType, currencyCode and, when a threshold has been crossed, alertThresholdExceeded for actual spend or forecastThresholdExceeded for the forecast. A message with neither threshold field is a plain status update: record it, do not announce it. For the others, the key budgetId + costIntervalStart + kind + threshold identifies the news, and a create-only Firestore write makes announcing it at most once.

relay.tsts
import express from "express";
import { Firestore, Timestamp } from "@google-cloud/firestore";
 
const db = new Firestore();
const app = express();
app.use(express.json());
 
const ALREADY_EXISTS = 6; // gRPC status code
 
interface BudgetStatus {
  budgetDisplayName: string;
  costAmount: number;
  budgetAmount: number;
  costIntervalStart: string;
  alertThresholdExceeded?: number;
  forecastThresholdExceeded?: number;
  currencyCode: string;
}
 
app.post("/pubsub", async (req, res) => {
  const message = req.body?.message;
  if (!message?.data) {
    res.status(400).send("no message");
    return;
  }
 
  const attrs = message.attributes ?? {};
  const status = JSON.parse(
    Buffer.from(message.data, "base64").toString("utf8"),
  ) as BudgetStatus;
 
  // Heartbeat: any message, even a plain status update, proves the path works.
  await db.collection("budgetRelay").doc("lastSeen").set({ at: Timestamp.now() });
 
  const news: { kind: string; threshold: number }[] = [];
  if (status.alertThresholdExceeded !== undefined) {
    news.push({ kind: "actual", threshold: status.alertThresholdExceeded });
  }
  if (status.forecastThresholdExceeded !== undefined) {
    news.push({ kind: "forecast", threshold: status.forecastThresholdExceeded });
  }
 
  for (const item of news) {
    const key = [attrs.budgetId, status.costIntervalStart, item.kind, item.threshold]
      .join("_")
      .replace(/[^A-Za-z0-9_.-]/g, "-");
    try {
      await db.collection("budgetAnnounced").doc(key).create({ at: Timestamp.now() });
    } catch (err) {
      if ((err as { code?: number }).code === ALREADY_EXISTS) continue;
      res.status(500).send("store failed"); // let Pub/Sub retry
      return;
    }
    await postToChat(
      `${status.budgetDisplayName}: ${item.kind} spend crossed ${item.threshold * 100} percent. ` +
        `Cost so far ${status.costAmount} ${status.currencyCode} of ${status.budgetAmount}.`,
    );
  }
 
  res.status(204).send();
});
 
async function postToChat(text: string) {
  const url = process.env.CHAT_WEBHOOK_URL ?? "";
  const response = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
  });
  if (response.ok === false) throw new Error(`chat webhook returned ${response.status}`);
}
 
app.listen(Number(process.env.PORT ?? 8080));

The announcement is written before it is posted, so a crash between the two loses one message instead of repeating it. For a budget alert that is the better failure, since the 100 percent message follows the 90 percent one anyway. The text body is what Slack-style incoming webhooks accept; check the format of your tool, and route the 100 percent and forecast cases to a phone push where it can. Returning an error makes Pub/Sub redeliver with the back-off you set via --min-retry-delay and --max-retry-delay, and because the relay is idempotent on the announcement key, redelivery is safe, the same property that idempotent Pub/Sub consumers rely on.

Usage alerts fire before the budget can#

Cost is the slowest signal. Two usage signals move within minutes: log ingestion and request volume. Cloud Logging publishes month-to-date ingestion as logging.googleapis.com/billing/monthly_bytes_ingested, which per the Logging metrics documentation is a gauge sampled every 1,800 seconds with data visible after up to 6,000 seconds. logging.googleapis.com/billing/bytes_ingested is a delta metric sampled every 60 seconds and suits a rate alert. A month-to-date threshold policy as a JSON file:

log-ingestion-policy.jsonjson
{
  "displayName": "Log ingestion month-to-date above threshold",
  "combiner": "OR",
  "conditions": [
    {
      "displayName": "monthly_bytes_ingested above 40 GiB",
      "conditionThreshold": {
        "filter": "metric.type=\"logging.googleapis.com/billing/monthly_bytes_ingested\"",
        "comparison": "COMPARISON_GT",
        "thresholdValue": 42949672960,
        "duration": "0s",
        "aggregations": [
          {
            "alignmentPeriod": "1800s",
            "perSeriesAligner": "ALIGN_MAX",
            "crossSeriesReducer": "REDUCE_SUM"
          }
        ]
      }
    }
  ]
}
terminalbash
gcloud monitoring policies create \
  --project=finops-admin \
  --policy-from-file=log-ingestion-policy.json \
  --notification-channels=projects/finops-admin/notificationChannels/CHANNEL_ID

The 40 GiB figure is a placeholder. Derive yours from the Metrics Explorer: read the month-to-date value at the end of each of the last three months and set the threshold a comfortable margin above the highest. The flags are on the policies create reference. Build the condition in Metrics Explorer first, confirm the chart shows what you expect, then export it: a hand-written filter that never matches looks identical to a policy that never fires.

For compute, the Cloud Run metrics include request count, billable container instance time and instance count per service. Alert on a sustained jump in request count for public services, and on billable instance time for services that should scale to zero.

A heartbeat for the pipeline#

Budget status messages arrive several times a day even when no threshold is crossed, so a day of silence means the path is broken: a deleted topic, a lost permission, a failed subscription. The relay already writes a lastSeen document on every message. A daily Cloud Scheduler job calls a second route that complains if the record is stale:

heartbeat.tsts
app.post("/heartbeat", async (_req, res) => {
  const snap = await db.collection("budgetRelay").doc("lastSeen").get();
  const at = snap.exists ? (snap.get("at") as Timestamp).toMillis() : 0;
  const hours = (Date.now() - at) / 3_600_000;
 
  if (hours > 36) {
    await postToChat(`Budget relay has seen no budget message for ${Math.round(hours)} hours. Check the topic, its IAM and the subscription.`);
  }
  res.status(204).send();
});

Thirty-six hours leaves room for the documented delays without waiting for the next invoice. The scheduler job should authenticate with an OIDC token and needs its own alerting so it does not fail silently itself; see Cloud Scheduler jobs that don't silently fail.

What breaks anyway#

  • Alerts do not stop spending. The things that bound damage while nobody is looking are limits: instance caps, API quotas and, for eligible services, spend cap budgets that pause the service. Decide deliberately which services you would accept pausing, and write it down.
  • Latency stacks. Billing data delay, budget message cadence and your relay add up. The budget is a backstop for the month; usage alerts are the smoke detector.
  • Figures are estimates. Budget figures use estimated billing data that can change until the invoice is finalised, so a threshold crossed on Tuesday can look different on the invoice.
  • Credits delay alerts. By default budgets count spend net of credits, so a promotional credit hides growth. If you want to see growth before credits, leave the credit types out of the budget's savings options so they are not subtracted.
  • Pacific time. Announcement keys use costIntervalStart for exactly this reason.
  • Dedupe state grows. One document per budget, month and threshold is negligible, but give the collection a cleanup policy.
  • Policy surprises. An organisation policy restricting identities by domain can block publishing from Cloud Billing without any notice. Only the heartbeat finds it.
  • Alert fatigue. Every extra policy dilutes the real one. Two or three well-chosen usage alerts beat dozens, and each should have an action someone can take.
  • Running cost. The pieces are small: a Pub/Sub topic receiving a few messages a day, a Cloud Run relay that scales to zero and a few Firestore writes. Check the Cloud Run, Pub/Sub and Firestore pricing pages for current free tiers.

If a runaway cost would be existential, an alert is the wrong tool. Look at spend cap budgets for eligible services, at quotas and at instance limits, accept that something may pause, and write down which things are allowed to.

Share
All articles →

Turn the Cloud Billing export into five queries that explain a Google Cloud invoice: what is fixed, what scales, what is a credit, and who owns each line.

9 min

An erasure request touches your database, search index, buckets, logs, analytics, backups and vendors. Here is how I make that deletion retryable and provable.

16 min