Cloud Scheduler jobs that don't silently fail

Why a scheduled job can fail for weeks without anyone noticing, and the retry, idempotency and alerting setup that makes Cloud Scheduler failures loud.

Published

Updated

Reading time

13 min

A scheduled job has the worst failure profile in a system. When it works, nothing visible happens, and when it stops, nothing visible happens either. A nightly rollup that has not run for nine days produces no error page and no angry user. It produces a dashboard that is slightly wrong, then very wrong, and eventually a customer who notices before you do. Cloud Scheduler is reliable as a clock, but its defaults are tuned for "try once and move on", and several of its failure modes leave no failed request behind to alert on.

This is a catalogue of those traps, for engineers who use Cloud Scheduler to call Cloud Run services. Each one gets what you would see, why it happens and the fix. The fixes add up to one small setup: a job definition, an idempotent handler with a lease that heals, and alerts that do not depend on anything Google does not document.

TrapWhat you seeWhat catches it
No retries configuredOne 503 at 02:00, no run until tomorrowHandler failure alert, absence alert
Token rejected before the handler401 or 403 on every runRequest-log alert, absence alert
Deadline shorter than the workFailed attempts, but the work completesHandler logs, idempotent claim
Retry meets a dead attempt's leaseScheduler stops retrying, run lostAbsence alert, watchdog
Success reported for unfinished workScheduler green, data staleHeartbeat absence alert
Nothing runs at allNo failure anywhereHeartbeat absence alert, watchdog
Two independent alert paths: one for attempts that fail, one for runs that never succeed or never happen.

One failed attempt, and then nothing until tomorrow#

The retry documentation gives the default for maximum retry attempts as 0. With no retry configured, the scheduler waits for the next scheduled time, so a transient 503 at 02:00 means no run until 02:00 tomorrow. The fix is an explicit job definition. Names, regions and URLs here are examples:

terminalbash
gcloud scheduler jobs create http rollup-hourly \
  --location=europe-west1 \
  --schedule="0 * * * *" \
  --time-zone="Etc/UTC" \
  --uri="https://rollup-api-abc123-ew.a.run.app/internal/rollup" \
  --http-method=POST \
  --oidc-service-account-email="scheduler-invoker@my-project.iam.gserviceaccount.com" \
  --oidc-token-audience="https://rollup-api-abc123-ew.a.run.app" \
  --attempt-deadline=300s \
  --max-retry-attempts=3 \
  --min-backoff=30s \
  --max-backoff=300s \
  --max-doublings=2

--max-retry-attempts=3 replaces the default of zero; the documented upper limit is 5. With a minimum backoff of 30 seconds, two doublings and a maximum of 300 seconds, retries land about 30, 60 and 120 seconds after each failure, following the sequence the retry documentation describes. That arithmetic matters again below. The window also has to fit inside the schedule: when retries continue through the next scheduled time, that next execution may be skipped, which is no concern for an hourly job with these settings and a real one for a job that runs every minute.

--time-zone="Etc/UTC" is already the CLI default, but writing it down documents the intent. The troubleshooting guide notes that jobs in time zones with daylight saving can run at irregular intervals when the clocks change, and recommends UTC. The cost is that a "09:00 local" job shifts by an hour twice a year; if the business needs local time, accept the drift deliberately.

The token is rejected before your code runs#

A service account loses the invoker role, an audience stops matching, or the Cloud Scheduler service agent role is removed; the troubleshooting guide lists the resulting 401 and 403 errors. Every run fails, every day, and your handler's logs show nothing because the request never reached it. That last part is why a handler-side alert alone is not enough.

Create a service account that exists only to call this service, and grant it invoker on that one service. The authentication guide for HTTP targets also warns against using the Cloud Scheduler service agent as the job's identity and against revoking its role.

terminalbash
gcloud iam service-accounts create scheduler-invoker \
  --display-name="Cloud Scheduler invoker for rollup-api"
 
gcloud run services add-iam-policy-binding rollup-api \
  --region=europe-west1 \
  --member="serviceAccount:scheduler-invoker@my-project.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

--oidc-token-audience is set to the service URL because Cloud Run's service-to-service guide says the audience must be the URL of the receiving service or a configured custom audience, and that custom domains are not supported for it. If you omit the audience, the scheduler uses the entire target URL including parameters, which will not match. For the wider IAM model see least privilege in practice. The Cloud Run request log still records the rejected request with its status, which is what the failure alert below reads.

The deadline and the handler disagree#

The scheduler cancels an attempt at its deadline and records a failure even if the handler keeps running and succeeds. --attempt-deadline=300s replaces the default of 3 minutes for HTTP targets; the job reference allows 15 seconds to 30 minutes. Set it a little above the handler's own timeout, so the handler gives up first and returns a proper error rather than being cut off. Past a few minutes of real work, do not hold an HTTP connection at all: run the work as a Cloud Run job or behind a queue, and keep the heartbeat described below.

The handler must also tolerate the cancelled attempt's work still being in flight when the retry arrives. That is where the next trap lives.

A retry that finds a dead attempt's lease#

Cloud Scheduler sends an X-CloudScheduler-ScheduleTime header with the original scheduled time, and the documentation states that it stays constant across retries. With the job name it is the idempotency key. The documentation also says that in rare circumstances multiple instances of the same job may be requested, so a handler must expect duplicates as well as retries.

The obvious design is "insert a row, skip if it exists", and it is wrong: the first attempt inserts the row and fails, and the retry sees the row and skips the work. The next design is a status column plus a reclaim timeout: a run stuck in running for longer than 15 minutes can be taken over. That one is wrong in a subtler way. Suppose an attempt dies halfway, because the instance is killed or runs out of memory. The scheduler retries about 30 seconds later and finds a row in running that is only seconds old. If the handler answers "already claimed, nothing to do" with a 200, the scheduler counts the retry as a success and stops retrying. The dead attempt's lease then sits there for 15 minutes with nobody left to reclaim it, and the run is lost.

Two changes fix it. First, a retry that finds a live lease must return a non-2xx status, so the scheduler keeps retrying until the lease expires or the work is done. Second, the lease must be short enough to expire inside the retry window. With retries at roughly 30, 60 and 120 seconds, cumulative about 30, 90 and 210 seconds after the first failure, a 15-minute lease can never expire in time. A short lease needs a heartbeat while the handler runs, so a healthy long run keeps extending it and a dead one stops.

migrations/001_job_runs.sqlsql
CREATE TABLE job_runs (
  job          text        NOT NULL,
  scheduled_at timestamptz NOT NULL,
  status       text        NOT NULL DEFAULT 'running',
  heartbeat_at timestamptz NOT NULL DEFAULT now(),
  finished_at  timestamptz,
  PRIMARY KEY (job, scheduled_at)
);
rollup.tsts
import type { FastifyInstance } from "fastify";
import type { Pool } from "pg";
 
const JOB = "rollup-hourly";
const LEASE_SECONDS = 60; // must expire before the last scheduler retry
const HEARTBEAT_MS = 15_000;
 
export function registerRollup(app: FastifyInstance, pool: Pool) {
  app.post("/internal/rollup", async (req, reply) => {
    const scheduleTime = req.headers["x-cloudscheduler-scheduletime"];
    if (typeof scheduleTime !== "string") {
      return reply.code(400).send({ error: "missing schedule time" });
    }
 
    // Claim: first attempt, a failed run, or a run whose heartbeat has stopped.
    const claim = await pool.query(
      `INSERT INTO job_runs (job, scheduled_at, status, heartbeat_at)
       VALUES ($1, $2, 'running', now())
       ON CONFLICT (job, scheduled_at) DO UPDATE
         SET status = 'running', heartbeat_at = now()
         WHERE job_runs.status = 'failed'
            OR (job_runs.status = 'running'
                AND job_runs.heartbeat_at < now() - make_interval(secs => $3::double precision))
       RETURNING status`,
      [JOB, scheduleTime, LEASE_SECONDS],
    );
 
    if (claim.rowCount === 0) {
      const { rows } = await pool.query(
        "SELECT status FROM job_runs WHERE job = $1 AND scheduled_at = $2",
        [JOB, scheduleTime],
      );
      if (rows[0]?.status === "done") {
        return reply.code(200).send({ skipped: true }); // finished: nothing to retry
      }
      // Another attempt holds a live lease. Non-2xx makes the scheduler retry,
      // so if that attempt is dead, a later retry reclaims the run.
      return reply.code(409).header("Retry-After", "30").send({ error: "run in progress" });
    }
 
    const beat = setInterval(() => {
      pool
        .query("UPDATE job_runs SET heartbeat_at = now() WHERE job = $1 AND scheduled_at = $2", [JOB, scheduleTime])
        .catch((err) => req.log.warn({ err }, "heartbeat failed"));
    }, HEARTBEAT_MS);
 
    try {
      await runRollup(scheduleTime);
      await pool.query(
        "UPDATE job_runs SET status = 'done', finished_at = now() WHERE job = $1 AND scheduled_at = $2",
        [JOB, scheduleTime],
      );
      // The heartbeat the absence alert depends on.
      req.log.info({ job: JOB, scheduleTime, message: "job_succeeded" }, "job_succeeded");
      return reply.code(200).send({ ok: true });
    } catch (err) {
      await pool.query(
        "UPDATE job_runs SET status = 'failed', finished_at = now() WHERE job = $1 AND scheduled_at = $2",
        [JOB, scheduleTime],
      );
      req.log.error({ err, job: JOB, scheduleTime, message: "job_failed" }, "job_failed");
      return reply.code(500).send({ error: "rollup failed" });
    } finally {
      clearInterval(beat);
    }
  });
}
 
declare function runRollup(scheduleTime: string): Promise<void>;

Walk through the cases. First attempt: the insert succeeds, work runs, status becomes done. Attempt dies mid-run: the heartbeat stops, the next retry at about 30 seconds probably still sees a live lease and gets a 409, and the retry after that, at about 90 seconds, finds the heartbeat older than 60 seconds and takes the run over. Scheduler deadline expires while the first attempt keeps working: retries get 409 while the heartbeat continues, and once the first attempt finishes, a later retry sees done and answers 200. A handler failure returns 500 and marks the run failed, so the next retry reclaims it immediately. The one remaining way to lose a run is for every retry to land while the lease is alive, so keep the lease comfortably below the sum of your backoffs, and never let it exceed the last retry's offset.

The status code is how the scheduler decides: the HTTP target reference describes acknowledgement as a response code in the 200 to 299 range. The 409s and the 500s all show up as failed attempts in the scheduler's own logs, which is noise you accept in exchange for a run that heals.

A success that is not a success#

A handler that answers 200 and then works in the background reports success for work that may fail later, and on Cloud Run with request-based billing that background work can also be starved of CPU. A scheduler call that only starts a Cloud Run job execution (a POST to the job's :run endpoint) has the same limit: its success means "an execution was requested". In both shapes the proof of success has to be written by the application at the moment the work completes. That is the job_succeeded line above, and the metric built from it below. For fan-out work, the Pub/Sub shape moves failure to the consumer, and the dead-letter queue article covers its safety net.

Alerts that do not rely on undocumented behaviour#

The scheduler writes AttemptStarted and AttemptFinished log entries under the monitored resource type cloud_scheduler_job, with a job_id label (monitored resources list), and they are the place to look when debugging a single run. Alerts are better built on log lines you control. Two kinds of failure need covering: the handler ran and failed, or the request was refused before it ran. One log-match policy with two conditions handles both. Your handler must emit structured JSON with a top-level message field, which Cloud Logging reads as described in structured logging on a budget:

failed-attempt-policy.jsonjson
{
  "displayName": "rollup-hourly: attempt failed",
  "documentation": {
    "content": "rollup-hourly failed in the handler or was refused before it ran. Check handler logs, the invoker role and the audience.",
    "mimeType": "text/markdown"
  },
  "combiner": "OR",
  "conditions": [
    {
      "displayName": "handler reported job_failed",
      "conditionMatchedLog": {
        "filter": "resource.type=\"cloud_run_revision\" AND jsonPayload.message=\"job_failed\" AND jsonPayload.job=\"rollup-hourly\""
      }
    },
    {
      "displayName": "request refused with 401 or 403",
      "conditionMatchedLog": {
        "filter": "resource.type=\"cloud_run_revision\" AND httpRequest.requestUrl:\"/internal/rollup\" AND httpRequest.status=(401 OR 403)"
      }
    }
  ],
  "alertStrategy": {
    "notificationRateLimit": { "period": "3600s" },
    "autoClose": "604800s"
  },
  "notificationChannels": ["projects/my-project/notificationChannels/1234567890"]
}
terminalbash
gcloud monitoring policies create --policy-from-file=failed-attempt-policy.json

The structure follows the example in the log-based alerting documentation, with a rate limit so a burst of retries produces one notification. With retries configured, a transient problem produces error entries and then a success. If that is too noisy, alert only on the final retry's failure, though noisy alerts for a few weeks show how often "transient" really happens.

Nothing runs, so nothing fails#

The job is paused, the API is disabled, the schedule moved. There is no failed attempt, because there is no attempt, and a hung run silently causes later runs to be skipped. Errors cannot catch an absence, so alert on the absence of the heartbeat. Create a counter metric from the job_succeeded line. Log-based metrics only count entries received after the metric exists (overview), so create it before the job's next run:

terminalbash
gcloud logging metrics create job_succeeded_rollup \
  --description="Successful rollup-hourly runs" \
  --log-filter='resource.type="cloud_run_revision" AND jsonPayload.message="job_succeeded" AND jsonPayload.job="rollup-hourly"'

Then use a metric-absence condition (conditionAbsent). The documentation's sample uses a duration slightly longer than the job's interval: 3900 seconds for an hourly pipeline.

absence-policy.jsonjson
{
  "displayName": "rollup-hourly: no success in over an hour",
  "combiner": "OR",
  "conditions": [
    {
      "displayName": "job_succeeded absent",
      "conditionAbsent": {
        "duration": "3900s",
        "filter": "metric.type=\"logging.googleapis.com/user/job_succeeded_rollup\" AND resource.type=\"cloud_run_revision\""
      }
    }
  ],
  "notificationChannels": ["projects/my-project/notificationChannels/1234567890"]
}
terminalbash
gcloud monitoring policies create --policy-from-file=absence-policy.json

This single policy covers the disguised failures: a paused job, a disabled API, a revoked invoker role, a handler that returns 200 without finishing, and a scheduler call that started a job which then crashed. It does not care why, which makes it the better alert, with the failure alert as the faster one.

The absence window has a ceiling#

The metric-absence duration has a minimum of 120 seconds, and the alerting documentation gives the maximum trigger absence time as 23.5 hours. That has two consequences. A daily job cannot use this alert directly, because its gap between successes is about 24 hours, longer than the largest window; the alert would fire every day. And a weekly job is out of reach by a wide margin. For anything that runs less often than roughly every 20 hours, check from the other side: run a cheap watchdog hourly, which reads your own job_runs table and logs an error when a job is overdue.

watchdog.tsts
import type { FastifyInstance } from "fastify";
import type { Pool } from "pg";
 
// Maximum acceptable age of the last success, per job.
const MAX_AGE: Record<string, string> = {
  "rollup-daily": "26 hours",
  "digest-weekly": "8 days",
};
 
export function registerWatchdog(app: FastifyInstance, pool: Pool) {
  app.post("/internal/watchdog", async (req, reply) => {
    for (const [job, maxAge] of Object.entries(MAX_AGE)) {
      const { rows } = await pool.query(
        `SELECT max(finished_at) < now() - $2::interval AS overdue, max(finished_at) IS NULL AS never
           FROM job_runs WHERE job = $1 AND status = 'done'`,
        [job, maxAge],
      );
      if (rows[0].never || rows[0].overdue) {
        req.log.error({ job, message: "job_overdue" }, "job_overdue");
      }
    }
    req.log.info({ job: "watchdog", message: "job_succeeded" }, "job_succeeded");
    return reply.code(200).send({ ok: true });
  });
}

A log-match alert on jsonPayload.message="job_overdue" pages you for the slow jobs, and the watchdog itself, scheduled hourly with job set to watchdog, gets the same absence alert as any hourly job (use its own metric filter on jsonPayload.job="watchdog"). The watchdog has to be a different job from the ones it watches, so that one failure cannot hide itself.

Drill the alerts once#

An alert you have never seen fire is a hypothesis. After setting this up, pause the job with gcloud scheduler jobs pause rollup-hourly --location=europe-west1, wait for the window to pass, confirm the absence alert arrives, and resume with gcloud scheduler jobs resume. Then point the target at a path that returns 500 and confirm the failure alert. Write down how long each took to arrive, because that is your real detection time. Remember that you cannot reuse a job name in a project even after deleting the job, so choose names you intend to keep.

A job that is only a convenience, such as warming a cache or refreshing a report nobody depends on, needs only the retry settings and the failure alert. And if you can remove the job entirely, do that first: many cron tasks refresh data that could be refreshed when it is read, as in refresh-on-read instead of a cron job. A job that does not exist cannot fail silently.

Share
All articles →

JSON logs that Cloud Logging understands, correlated by trace, with the noise excluded before it is billed. A pino setup for Cloud Run and what it costs.

12 min