Cloud Run jobs or services for batch work
Batch work behaves badly behind an HTTP endpoint. Compare Cloud Run jobs and services on retries, parallelism and exit codes, and shard work across tasks.
Sooner or later a service gets a chore that does not fit a request: recompute a derived table, re-send a month of failed emails, migrate a column, rebuild a search index. The first implementation is usually an HTTP endpoint on the service you already run, called by a scheduler. It works with ten thousand rows. At a few million it times out halfway, retries from the beginning, and fights live traffic for the same database connections. This guide is about when that chore should move into a Cloud Run job, how to shard it across tasks, and what the platform does with exit codes and signals. It follows Cloud Run vs GKE vs Cloud Functions, which put jobs in one row of a table; this is the longer version of that row.
Five ways to run a chore#
The comparison below uses a hypothetical piece of work: five million records to reprocess. It is finite, has a known start and end, and running it twice must give the same result as running it once.
| Approach | Longest single run | Fan-out | Retry unit | Pacing control | Idle cost |
|---|---|---|---|---|---|
| Endpoint on the existing service, called by Scheduler | Request timeout, 60 minutes at most | None, unless the endpoint spawns its own | The whole request | Shares CPU and connections with live traffic | None beyond the service |
| Dedicated batch service plus Pub/Sub, one message per chunk | Per message processing time | Pub/Sub delivers chunks to many instances | A message | Max instances and subscription settings | None if scaled to zero |
| Cloud Run job with sharded tasks | Days per task | Up to 10,000 tasks, with a parallelism cap | A task | --parallelism | None |
| Cloud Run job where tasks pull from a queue | Days per task | Workers pull until empty | A task, plus the queue's own redelivery | Task count and pull rate | None |
| A long-lived worker on GKE or a VM | Unbounded | Whatever you build | Whatever you build | Whatever you build | A node or VM is always there |
The Pub/Sub row has real merit when chunks arrive continuously or other consumers need the same events; the trade-offs are in dead-letter queues: a small team's safety net and idempotent Pub/Sub consumers. For work with a start and an end, a job is simpler: no topic to create, no subscription to tune, and the execution record says whether it finished.
What a job gives you that an endpoint cannot#
These facts come from the Cloud Run documentation on creating jobs, the container contract and job retries. Defaults and limits change, so recheck any number you rely on.
- An execution is made of tasks.
--tasksdefaults to 1 and accepts up to 10,000. Each task is one container instance running the same image. - Each task knows who it is.
CLOUD_RUN_TASK_INDEX(from 0),CLOUD_RUN_TASK_COUNTandCLOUD_RUN_TASK_ATTEMPT(retries so far, from 0) are set in the environment. - Retries and timeouts are per task.
--max-retriesdefaults to 3 and accepts 0 to 10.--task-timeoutdefaults to 10 minutes and can go to 168 hours. There is no timeout for the execution as a whole: it completes when all tasks finish. - Parallelism is a dial.
--parallelismcaps how many tasks run at once. The documentation points at databases that cannot absorb many simultaneous requests as the reason to lower it. - Exit codes carry two meanings. The container contract says to exit 0 when the job has completed and non-zero when it has failed. There is no third value for "failed, do not retry": a failed task is retried up to
--max-retries, and the only way to avoid retries is to configure zero of them or to not fail. - Timeouts and cancellations send SIGTERM, followed by SIGKILL after a short period that the jobs section of the contract puts at 10 seconds.
- Overrides are per execution.
gcloud run jobs executeaccepts--args,--update-env-vars,--tasksand--task-timeout, and they affect only that execution.
For comparison, a service request has a timeout that defaults to 5 minutes and can be raised to 60. A job task can run for days. That alone decides many cases.
What tips the decision#
Duration and retry cost. If the work fits comfortably in one request and a retry from the start is cheap, an endpoint is fine. If a retry would repeat hours of work, you want a retry unit smaller than the whole job, and tasks give you that.
Who shares the database. An endpoint on the live service competes with real traffic for CPU and connections, and you cannot slow it down without slowing the service. A job has its own capacity and a --parallelism cap that exists for exactly this.
Whether the work ever ends. A job has no notion of "keep running and wait for more". If chunks arrive continuously from users, a Pub/Sub consumer matches better. If you need several coordinated stages with branching and approval, look at Workflows before growing a state machine inside one container.
What it costs to have a second deployable. A job means another service account and another place to look when something goes wrong. For a few seconds of work, once a day, well inside the request timeout, the endpoint wins. The real bill for a big batch is usually the database, not Cloud Run: eight parallel tasks holding connections and scanning large ranges can push a database to a larger tier. Let the parallelism cap, not a budget alert, be the control, and rehearse against a copy first. Reading a GCP bill like an architect covers how to see that afterwards.
My default is a job with sharded tasks for finite batch work and services for anything that answers a request. The idempotency that makes retries safe (a processed marker per item, or a claim step as in Postgres SKIP LOCKED job queues and idempotent Pub/Sub consumers) is assumed below, not re-derived.
A sharded worker that stops when told to#
The worker reads its identity from the environment, selects its slice, skips work already done, and exits with a code that tells Cloud Run what happened. Table and column names are illustrative, and the example uses the pg package.
The slice needs thought. The tempting shard is id % taskCount = taskIndex, and it works, but a modulus on a column cannot use a plain index on that column: every task scans the whole table and discards most of it. Shard by contiguous id ranges instead, so each task's query is a range scan. The execution is given fixed bounds, computed once before it starts, so a retried task derives exactly the same range:
-- Partial index: only unprocessed rows, ordered by id. Range scans use it,
-- and it shrinks as the batch progresses.
CREATE INDEX records_unprocessed_id ON records (id) WHERE processed_at IS NULL;
-- Run once before the execution, then pass the result as ID_LOW and ID_HIGH.
SELECT min(id) AS id_low, max(id) + 1 AS id_high FROM records WHERE processed_at IS NULL;import { Pool } from "pg";
const taskIndex = Number(process.env.CLOUD_RUN_TASK_INDEX ?? 0);
const taskCount = Number(process.env.CLOUD_RUN_TASK_COUNT ?? 1);
const attempt = Number(process.env.CLOUD_RUN_TASK_ATTEMPT ?? 0);
const idLow = Number(process.env.ID_LOW);
const idHigh = Number(process.env.ID_HIGH); // exclusive
const BATCH = 500;
// This task's slice: a contiguous id range derived only from the task index.
const span = Math.ceil((idHigh - idLow) / taskCount);
const sliceFrom = idLow + taskIndex * span;
const sliceTo = Math.min(idHigh, sliceFrom + span);
const pool = new Pool({ max: 2 }); // small, because many tasks run at once
let stopping = false;
process.on("SIGTERM", () => {
stopping = true; // checked before every item, so we leave within one item
});
async function main(): Promise<number> {
console.log(JSON.stringify({ msg: "task start", taskIndex, taskCount, attempt, sliceFrom, sliceTo }));
let lastId = sliceFrom - 1;
let done = 0;
let skipped = 0;
while (!stopping) {
const { rows } = await pool.query(
`SELECT id, payload
FROM records
WHERE id > $1
AND id < $2
AND processed_at IS NULL
ORDER BY id
LIMIT $3`,
[lastId, sliceTo, BATCH],
);
if (rows.length === 0) break;
for (const row of rows) {
if (stopping) break; // SIGTERM arrived: do not start another item
lastId = row.id;
try {
await reprocess(row);
await pool.query("UPDATE records SET processed_at = now() WHERE id = $1", [row.id]);
done++;
} catch (err) {
if (isPermanent(err)) {
// A bad record is data, not an outage. Record it and carry on.
await pool.query(
"INSERT INTO reprocess_failures (record_id, reason) VALUES ($1, $2) ON CONFLICT DO NOTHING",
[row.id, String(err)],
);
skipped++;
} else {
throw err; // transient: fail the task so Cloud Run retries it
}
}
}
}
console.log(JSON.stringify({ msg: "task end", taskIndex, done, skipped, interrupted: stopping }));
return stopping ? 1 : 0; // interrupted means unfinished: report failure, not success
}
main()
.then((code) => process.exit(code))
.catch((err) => {
console.error(JSON.stringify({ msg: "task failed", taskIndex, error: String(err) }));
process.exit(1);
});
declare function reprocess(row: { id: number; payload: unknown }): Promise<void>;
declare function isPermanent(err: unknown): boolean;Several decisions are packed into that file.
The slice is a pure function of the task index and the execution's bounds. A retried task re-selects the same range, and the processed_at IS NULL filter makes it skip what the earlier attempt finished. The checkpoint lives in the database, not the container. Do not use OFFSET on a table that changes while you process it, because rows shift under it. Range slices are uneven if ids are sparse or clustered, which makes one task the long pole of the execution; compare row counts per range on a copy before choosing the task count.
SIGTERM is checked per item, not per batch. Cloud Run sends SIGTERM when a task times out or is cancelled, and the container has about ten seconds before SIGKILL. A flag that is only read in the outer loop would let a 500-row batch keep going until the kill. Checking it at the top of each iteration bounds the delay to one item, and that item must still finish inside the window, so keep per-item work small. If it cannot be small, the idempotency of each item has to carry the weight of a mid-item kill.
Item failures and task failures are handled separately. A malformed record that fails every time would otherwise fail the task and its retries, hiding the rest of the slice behind it. Permanent errors go to a failures table. Transient errors, such as a dropped connection, throw and exit non-zero. Because exit codes only distinguish success from failure, the way to avoid pointless retries is to not crash on permanent errors.
The pool is tiny. Every task is its own container with its own pool. Fifty parallel tasks with ten connections each is five hundred connections, which is the failure in connection pooling when every service is tiny. Size pools from the parallelism cap.
Create it, run it, size it#
The flags come from the create jobs and execute jobs pages. Image path, region and service account are placeholders:
gcloud run jobs deploy reprocess-records \
--image=europe-west4-docker.pkg.dev/PROJECT_ID/apps/reprocess:latest \
--region=europe-west4 \
--service-account=reprocess-job@PROJECT_ID.iam.gserviceaccount.com \
--tasks=40 \
--parallelism=8 \
--max-retries=3 \
--task-timeout=2h
gcloud run jobs execute reprocess-records \
--region=europe-west4 \
--update-env-vars=ID_LOW=1,ID_HIGH=5000001 \
--waitgcloud run jobs deploy creates the job if it does not exist and updates it if it does; --wait blocks until the execution completes, which suits a CI step or a manual run. The overrides change only that execution. Changing --tasks per execution is safe here because the slice follows CLOUD_RUN_TASK_COUNT, but keep ID_LOW and ID_HIGH fixed for the life of one execution.
Give the job its own service account, separate from your online services, with only the database and secret access the batch needs (GCP IAM least privilege in practice).
Three numbers need a reason, and none should be copied from this page.
- Tasks. More tasks mean smaller retry units but more container starts. Aim for a task that takes minutes to tens of minutes, so a retry is cheap.
- Parallelism. Start from the database. Raise it only while online latency stays flat in your monitoring.
- Task timeout. Set it from the measured duration of the slowest task plus a margin, not to the 7-day maximum. It is also the backstop against a hung task that would otherwise be billed until someone notices.
Watching it finish#
A scheduler or workflow starts the job, and an alert watches the outcome. Noticing when the start itself fails is covered in Cloud Scheduler jobs that don't silently fail; the row there about a scheduler calling the jobs API applies directly. For the result, the documented commands are:
gcloud run jobs executions list --job=reprocess-records --region=europe-west4
gcloud run jobs executions describe-latest --job=reprocess-records --region=europe-west4An execution that succeeded means the containers exited 0. It does not mean the data is right, so end the batch with a count query and write a summary row with the done and skipped counts: "finished with 312 skipped" should not need log archaeology. If isPermanent is wrong in the permissive direction, a real outage is recorded as thousands of bad records and every task exits 0, so alert on the skipped count as well as on failed executions. The structured log lines above are queryable in the style of structured logging on a budget.
Two more things to keep in mind. Tasks may not start in index order when parallelism is above one, so never write a task that assumes task 0 has finished. And when the batch and online traffic update the same rows, keep per-item writes narrow and conditional, or a stale recomputation will overwrite a fresher value.
Which one for which situation#
- A few seconds of idempotent work, once a day. An endpoint on the existing service, called by Scheduler. A job adds a second deployable for nothing.
- Finite work with millions of items, or anything a retry from zero would make expensive. A job with sharded tasks, a parallelism cap set from the database, and per-item progress in the database.
- Chunks that arrive continuously from users, or events other consumers need too. A Pub/Sub consumer. A job has no way to wait for more.
- Work with stages, branching or human approval. An orchestrator such as Workflows, which can start jobs as steps.
- A process that must stay up for hours holding state in memory. A long-lived worker, and accept the always-on cost.
Related articles
All articles →Turn the Cloud Billing export into five queries that explain a Google Cloud invoice: what is fixed, what scales, what is a credit, and who owns each line.
9 min
Pick the platform by workload shape, not habit: when Cloud Run services, Cloud Run functions, jobs or GKE Autopilot fit, and how to move between them later.
11 min
Measure which phase of a Cloud Run cold start is slow, match it to the lever that shortens it, and pay for warm instances only where the latency earns it.
15 min