Eventarc to Cloud Run: the identities behind a 401 or 403
An Eventarc trigger delivering to a private Cloud Run service passes four identity checks. A 401/403 reference table, and the setup that keeps each one correct.
The trigger exists, the event source is firing, and the handler never logs a request. Or the only trace is a 401 or 403 in a log you did not know to open. Both are the same kind of problem: Cloud Run answered before your code ran, and it answered because one of several identities in the delivery path did not match what the service expected. This page is a reference for reading that failure. The table comes first, because that is what you need at the moment something is broken; the chain, the setup commands and the audience rules follow.
Start from the receiving service's log#
The trigger console shows little. The evidence is on the receiving side. Eventarc's troubleshooting page says that if the Cloud Run logs show a non-OK status code, the service is rejecting events, and for Pub/Sub transport it suggests watching the push request count by response code. Filter the service's request log for errors:
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="ingest-worker" AND httpRequest.status>=400' \
--limit=20 \
--freshness=1h \
--format='table(timestamp, httpRequest.status, httpRequest.requestUrl)'If this returns nothing while the source is clearly emitting events, either the request never reached Cloud Run as an authenticated call, or the rejection happened before a request log was written. The next section covers the second case.
Reading the table#
Cloud Run's own troubleshooting page documents the split that this table builds on: a 401 is about the token, a 403 is about permission or a missing token. For an Eventarc trigger the token is always attached by the platform, so a 403 almost always means the permission is missing, and a 401 means the token is not acceptable to this service.
| What you see | Cause | Fix |
|---|---|---|
401 "The request was not authorized to invoke this service" | The token's aud does not match the service URL or a configured custom audience | Make the audience the run.app URL, or register the audience callers use as a custom audience on the service |
401, with a Pub/Sub push subscription you created | --push-auth-token-audience set to a custom domain, which Cloud Run does not accept as aud | Leave the override empty, or set it to the service URL |
401, token present | An access token or refresh token instead of an ID token, an invalid signature, several JWTs or several Authorization headers | Send exactly one ID token as Authorization: Bearer ... |
403 "Your client does not have permission to get URL from this server" | The identity that generated the token lacks run.routes.invoke | Grant roles/run.invoker to the trigger's service account on this service |
403 "The request was not authenticated. Either allow public access or set the proper Authorization header" | No token reached the service | Check that the trigger or subscription actually has a service account; for your own subscription, check push authentication is enabled |
403, and no cloud_run_revision entry in the log | A VPC Service Controls perimeter can block the request before Cloud Run logs it | Read the policy audit logs of the perimeter |
404 instead of 403 | Ingress is internal and the request does not come from an allowed source | Ingress settings, not IAM |
INVALID_ARGUMENT, service account does not exist | The trigger's service account was deleted; events are not delivered | Create a new account, re-grant, and update the trigger |
2xx but nothing happens | The handler | Application logs, not IAM |
5xx | The handler failed; the platform retries | Fix the code, and make it idempotent |
The two rows that cost the most time are the audience ones, because the roles all look correct in the console and the failure carries no hint about the token's contents. When the cause is not obvious, decode the token a test call sends and compare aud and email with what the service and its IAM policy expect.
The four links behind one delivery#
A delivery succeeds only if four things hold at once. They fail independently and look identical from the event source, which is why the table above starts from the status code instead of from the trigger.
- The trigger has an identity. The roles reference says a trigger can use a user-managed service account as its identity, and uses the Compute Engine default service account when none is given. That account is the principal that calls your service.
- That identity may receive and invoke.
roles/eventarc.eventReceiverlets the trigger receive events from providers;roles/run.invokerlets it call the target Cloud Run service. The first is project-wide, the second belongs on the one service. - The transport may mint a token for that identity. Eventarc Standard delivers through Pub/Sub. For authenticated push, the Pub/Sub service agent needs
iam.serviceAccounts.getOpenIdToken, or the Service Account Token Creator role, on the service account, per authentication for push subscriptions. The Cloud Storage trigger quickstart says the explicit grant is only needed if the Pub/Sub service agent was enabled on or before April 8, 2021; otherwise the role is granted by default. - The token is addressed to the receiver. Per the service-to-service guide, the
audclaim must be the receiving service's URL or a configured custom audience.
Links 1 to 3 are about who is allowed. Link 4 is about what the token says it is for.
One delivery identity per target#
The identity choice is the only real design decision in the setup.
| Choice | Consequence |
|---|---|
| Compute Engine default service account | Broad permissions in many projects, shared with everything else |
| One account for all triggers | One set of grants, but a leak or mistake reaches every target |
| One account per target service | One more account per service, and the invoker binding on each service names exactly one caller |
I use the third. The cost is one gcloud iam service-accounts create per service, and the benefit is that the service's IAM policy shows its single caller at a glance. The commands below follow the official Storage trigger quickstart, with a dedicated account in place of the default one. Names and regions are examples.
PROJECT_ID="my-project"
PROJECT_NUMBER="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')"
gcloud services enable eventarc.googleapis.com pubsub.googleapis.com \
run.googleapis.com storage.googleapis.com
gcloud config set eventarc/location europe-west1
# Link 1: the identity
gcloud iam service-accounts create ingest-trigger \
--display-name="Eventarc delivery for ingest-worker"
# Link 2: receive on the project, invoke on one service only
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="serviceAccount:ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/eventarc.eventReceiver"
gcloud run services add-iam-policy-binding ingest-worker \
--region=europe-west1 \
--member="serviceAccount:ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/run.invoker"
# Link 3: let Pub/Sub mint tokens for that account only
gcloud iam service-accounts add-iam-policy-binding \
"ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com" \
--member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" \
--role="roles/iam.serviceAccountTokenCreator"
# Cloud Storage events also need the storage agent to publish
SERVICE_AGENT="$(gcloud storage service-agent --project="$PROJECT_ID")"
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="serviceAccount:${SERVICE_AGENT}" \
--role="roles/pubsub.publisher"
gcloud eventarc triggers create ingest-uploads \
--destination-run-service=ingest-worker \
--destination-run-region=europe-west1 \
--event-filters="type=google.cloud.storage.object.v1.finalized" \
--event-filters="bucket=${PROJECT_ID}-uploads" \
--service-account="ingest-trigger@${PROJECT_ID}.iam.gserviceaccount.com"Do not grant run.invoker at project level out of convenience: that turns a delivery identity into a caller of every service. Bindings are idempotent, so re-running any of these is harmless. If a CI identity creates triggers or subscriptions, it also needs iam.serviceAccounts.actAs on the delivery account, which roles/iam.serviceAccountUser includes. A first trigger in a project can hit a permission delay, and a trigger can take up to two minutes to become fully functional.
The audience: nothing to set on a trigger#
A Standard trigger to Cloud Run has no audience option. The gcloud eventarc triggers create reference has flags for the destination, service account, filters, transport topic and retry attempts, and none for audience. The practical rule is that the receiving service must accept the audience the platform sends, which for a Cloud Run destination is its own run.app URL. Cloud Run's guide adds that custom domains are not supported as aud values.
So the audience only becomes a problem on paths where you choose it: a Pub/Sub push subscription you create yourself, a Cloud Scheduler job, or any caller that knows only the custom domain or load balancer address. Pub/Sub exposes --push-auth-token-audience as an optional override. Set it to a domain name because that is what clients see, and Cloud Run answers 401. There are two fixes. Leave the override empty or set it to the run.app URL, or register the value your callers use as a custom audience:
gcloud run services update ingest-worker \
--region=europe-west1 \
--add-custom-audiences="https://events.example.com"Custom audiences are set on the service and apply to every serving revision. The only removal flag documented is --clear-custom-audiences, which removes all of them, so keep the full list in a script rather than in someone's memory. A private service needs no verification code of its own: Cloud Run has validated the token before the request reaches the container.
Retries make a permanent 403 loud#
Eventarc Standard delivers at least once, so a handler that mutates state must tolerate duplicates; idempotent Pub/Sub consumers covers that half. The failure side is on the retry page: Eventarc's retry documentation says the default message retention is 24 hours, and Pub/Sub backs off exponentially from a 10-second minimum to a 600-second maximum between attempts. A permanent 403 on a busy source therefore means a day of repeated failing deliveries per event, and a growing backlog behind them. Alert on rejected deliveries within the hour, not at the end of the retention window.
The retry policy can be reduced to a single attempt (--max-retry-attempts=1, the only value the flag accepts, and only for Cloud Run destinations). That suits a handler that is not idempotent, but then a transient 503 loses the event. Pair retries with a dead-letter path rather than switching them off.
Two timing facts also catch people during testing. IAM grants take a short while to apply, so re-test with a fresh event rather than a replay. And ID tokens expire about an hour after creation; a hand-made token used for a test call and cached will start failing on its own.
A drill that turns the table into something you have seen#
In a staging project, upload an object and watch the request arrive:
gcloud storage cp ./sample.pdf "gs://${PROJECT_ID}-uploads/test/sample.pdf"
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="ingest-worker"' \
--limit=5 --freshness=10mThen break one link at a time and compare the status with the table. Remove the invoker binding and expect a 403. Set a Pub/Sub push subscription's audience override to a value the service does not list and expect a 401. Restore each change before the next. A drill like that takes ten minutes and leaves you with a runbook written from observed behaviour instead of from the page above.
Eventarc is worth these four links when you need its event sources, such as Cloud Storage or audit logs. If one Pub/Sub topic feeds one service, a plain push subscription with an authenticated endpoint has fewer links to keep correct.
Related articles
All articles →Most small teams should not start with a service perimeter. The organization prerequisite, the friction it adds, and when it pays for itself.
9 min
A one-hour audit procedure for Google Cloud IAM: snapshot the policies, run eight targeted queries, triage what you find, and keep the result diffable.
13 min
Validate the user's JWT once at API Gateway, pass identity downstream as a header, and close the bypass, spoofing and revocation gaps that design opens.
14 min