Eventarc to Cloud Run: the identities behind a 401 or 403

An Eventarc trigger delivering to a private Cloud Run service passes four identity checks. A 401/403 reference table, and the setup that keeps each one correct.

Published

Updated

Reading time

9 min

The trigger exists, the event source is firing, and the handler never logs a request. Or the only trace is a 401 or 403 in a log you did not know to open. Both are the same kind of problem: Cloud Run answered before your code ran, and it answered because one of several identities in the delivery path did not match what the service expected. This page is a reference for reading that failure. The table comes first, because that is what you need at the moment something is broken; the chain, the setup commands and the audience rules follow.

Start from the receiving service's log#

The trigger console shows little. The evidence is on the receiving side. Eventarc's troubleshooting page says that if the Cloud Run logs show a non-OK status code, the service is rejecting events, and for Pub/Sub transport it suggests watching the push request count by response code. Filter the service's request log for errors:

terminalbash
gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="ingest-worker" AND httpRequest.status>=400' \
  --limit=20 \
  --freshness=1h \
  --format='table(timestamp, httpRequest.status, httpRequest.requestUrl)'

If this returns nothing while the source is clearly emitting events, either the request never reached Cloud Run as an authenticated call, or the rejection happened before a request log was written. The next section covers the second case.

Reading the table#

Cloud Run's own troubleshooting page documents the split that this table builds on: a 401 is about the token, a 403 is about permission or a missing token. For an Eventarc trigger the token is always attached by the platform, so a 403 almost always means the permission is missing, and a 401 means the token is not acceptable to this service.

What you seeCauseFix
401 "The request was not authorized to invoke this service"The token's aud does not match the service URL or a configured custom audienceMake the audience the run.app URL, or register the audience callers use as a custom audience on the service
401, with a Pub/Sub push subscription you created--push-auth-token-audience set to a custom domain, which Cloud Run does not accept as audLeave the override empty, or set it to the service URL
401, token presentAn access token or refresh token instead of an ID token, an invalid signature, several JWTs or several Authorization headersSend exactly one ID token as Authorization: Bearer ...
403 "Your client does not have permission to get URL from this server"The identity that generated the token lacks run.routes.invokeGrant roles/run.invoker to the trigger's service account on this service
403 "The request was not authenticated. Either allow public access or set the proper Authorization header"No token reached the serviceCheck that the trigger or subscription actually has a service account; for your own subscription, check push authentication is enabled
403, and no cloud_run_revision entry in the logA VPC Service Controls perimeter can block the request before Cloud Run logs itRead the policy audit logs of the perimeter
404 instead of 403Ingress is internal and the request does not come from an allowed sourceIngress settings, not IAM
INVALID_ARGUMENT, service account does not existThe trigger's service account was deleted; events are not deliveredCreate a new account, re-grant, and update the trigger
2xx but nothing happensThe handlerApplication logs, not IAM
5xxThe handler failed; the platform retriesFix the code, and make it idempotent
Table 1— What the receiving service reports, the cause, and the fix

The two rows that cost the most time are the audience ones, because the roles all look correct in the console and the failure carries no hint about the token's contents. When the cause is not obvious, decode the token a test call sends and compare aud and email with what the service and its IAM policy expect.

A delivery succeeds only if four things hold at once. They fail independently and look identical from the event source, which is why the table above starts from the status code instead of from the trigger.

The four conditions behind one delivery. A failure at any of them looks the same from the event source.
  1. The trigger has an identity. The roles reference says a trigger can use a user-managed service account as its identity, and uses the Compute Engine default service account when none is given. That account is the principal that calls your service.
  2. That identity may receive and invoke. roles/eventarc.eventReceiver lets the trigger receive events from providers; roles/run.invoker lets it call the target Cloud Run service. The first is project-wide, the second belongs on the one service.
  3. The transport may mint a token for that identity. Eventarc Standard delivers through Pub/Sub. For authenticated push, the Pub/Sub service agent needs iam.serviceAccounts.getOpenIdToken, or the Service Account Token Creator role, on the service account, per authentication for push subscriptions. The Cloud Storage trigger quickstart says the explicit grant is only needed if the Pub/Sub service agent was enabled on or before April 8, 2021; otherwise the role is granted by default.
  4. The token is addressed to the receiver. Per the service-to-service guide, the aud claim must be the receiving service's URL or a configured custom audience.

Links 1 to 3 are about who is allowed. Link 4 is about what the token says it is for.

One delivery identity per target#

The identity choice is the only real design decision in the setup.

ChoiceConsequence
Compute Engine default service accountBroad permissions in many projects, shared with everything else
One account for all triggersOne set of grants, but a leak or mistake reaches every target
One account per target serviceOne more account per service, and the invoker binding on each service names exactly one caller

I use the third. The cost is one gcloud iam service-accounts create per service, and the benefit is that the service's IAM policy shows its single caller at a glance. The commands below follow the official Storage trigger quickstart, with a dedicated account in place of the default one. Names and regions are examples.

terminalbash
PROJECT_ID="my-project"
PROJECT_NUMBER="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')"
 
gcloud services enable eventarc.googleapis.com pubsub.googleapis.com \
  run.googleapis.com storage.googleapis.com
gcloud config set eventarc/location europe-west1
 
# Link 1: the identity
gcloud iam service-accounts create ingest-trigger \
  --display-name="Eventarc delivery for ingest-worker"
 
# Link 2: receive on the project, invoke on one service only
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member="serviceAccount:ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/eventarc.eventReceiver"
 
gcloud run services add-iam-policy-binding ingest-worker \
  --region=europe-west1 \
  --member="serviceAccount:ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/run.invoker"
 
# Link 3: let Pub/Sub mint tokens for that account only
gcloud iam service-accounts add-iam-policy-binding \
  "ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com" \
  --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountTokenCreator"
 
# Cloud Storage events also need the storage agent to publish
SERVICE_AGENT="$(gcloud storage service-agent --project="$PROJECT_ID")"
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member="serviceAccount:${SERVICE_AGENT}" \
  --role="roles/pubsub.publisher"
 
gcloud eventarc triggers create ingest-uploads \
  --destination-run-service=ingest-worker \
  --destination-run-region=europe-west1 \
  --event-filters="type=google.cloud.storage.object.v1.finalized" \
  --event-filters="bucket=${PROJECT_ID}-uploads" \
  --service-account="ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com"

Do not grant run.invoker at project level out of convenience: that turns a delivery identity into a caller of every service. Bindings are idempotent, so re-running any of these is harmless. If a CI identity creates triggers or subscriptions, it also needs iam.serviceAccounts.actAs on the delivery account, which roles/iam.serviceAccountUser includes. A first trigger in a project can hit a permission delay, and a trigger can take up to two minutes to become fully functional.

The audience: nothing to set on a trigger#

A Standard trigger to Cloud Run has no audience option. The gcloud eventarc triggers create reference has flags for the destination, service account, filters, transport topic and retry attempts, and none for audience. The practical rule is that the receiving service must accept the audience the platform sends, which for a Cloud Run destination is its own run.app URL. Cloud Run's guide adds that custom domains are not supported as aud values.

So the audience only becomes a problem on paths where you choose it: a Pub/Sub push subscription you create yourself, a Cloud Scheduler job, or any caller that knows only the custom domain or load balancer address. Pub/Sub exposes --push-auth-token-audience as an optional override. Set it to a domain name because that is what clients see, and Cloud Run answers 401. There are two fixes. Leave the override empty or set it to the run.app URL, or register the value your callers use as a custom audience:

terminalbash
gcloud run services update ingest-worker \
  --region=europe-west1 \
  --add-custom-audiences="https://events.example.com"

Custom audiences are set on the service and apply to every serving revision. The only removal flag documented is --clear-custom-audiences, which removes all of them, so keep the full list in a script rather than in someone's memory. A private service needs no verification code of its own: Cloud Run has validated the token before the request reaches the container.

Retries make a permanent 403 loud#

Eventarc Standard delivers at least once, so a handler that mutates state must tolerate duplicates; idempotent Pub/Sub consumers covers that half. The failure side is on the retry page: Eventarc's retry documentation says the default message retention is 24 hours, and Pub/Sub backs off exponentially from a 10-second minimum to a 600-second maximum between attempts. A permanent 403 on a busy source therefore means a day of repeated failing deliveries per event, and a growing backlog behind them. Alert on rejected deliveries within the hour, not at the end of the retention window.

The retry policy can be reduced to a single attempt (--max-retry-attempts=1, the only value the flag accepts, and only for Cloud Run destinations). That suits a handler that is not idempotent, but then a transient 503 loses the event. Pair retries with a dead-letter path rather than switching them off.

Two timing facts also catch people during testing. IAM grants take a short while to apply, so re-test with a fresh event rather than a replay. And ID tokens expire about an hour after creation; a hand-made token used for a test call and cached will start failing on its own.

A drill that turns the table into something you have seen#

In a staging project, upload an object and watch the request arrive:

terminalbash
gcloud storage cp ./sample.pdf "gs://${PROJECT_ID}-uploads/test/sample.pdf"
 
gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="ingest-worker"' \
  --limit=5 --freshness=10m

Then break one link at a time and compare the status with the table. Remove the invoker binding and expect a 403. Set a Pub/Sub push subscription's audience override to a value the service does not list and expect a 401. Restore each change before the next. A drill like that takes ten minutes and leaves you with a runbook written from observed behaviour instead of from the page above.

Eventarc is worth these four links when you need its event sources, such as Cloud Storage or audit logs. If one Pub/Sub topic feeds one service, a plain push subscription with an authenticated endpoint has fewer links to keep correct.

Share
All articles →

A one-hour audit procedure for Google Cloud IAM: snapshot the policies, run eight targeted queries, triage what you find, and keep the result diffable.

13 min

Validate the user's JWT once at API Gateway, pass identity downstream as a header, and close the bypass, spoofing and revocation gaps that design opens.

14 min