Google Cloud PERMISSION_DENIED Troubleshooting Guide

A Google Cloud PERMISSION_DENIED error tells you a request was refused, but not which of several independent policy layers refused it. Allow policies, deny policies, Principal Access Boundary policies, VPC Service Controls perimeters, API enablement, quota project settings and plain propagation delay all produce a 403, and the fix for each is different. This guide treats the error as a troubleshooting problem: the symptom and its exact wording, the likely causes from most to least common, how to diagnose each with Policy Troubleshooter and audit logs, the fix, and how to stop it recurring. It is for cloud platform engineers, SREs and security teams supporting both people and service accounts. Error strings and limits were checked against Google Cloud documentation on 11 November 2026.

Key takeaways

Read the whole error, not the status. The reason, domain, permission and error_info_id fields in the error details tell you which layer refused and which permission was missing.
Deny beats allow. IAM checks deny policies before allow policies, so adding a role does nothing if a deny policy at the organisation, folder or project names that permission.
Policy Troubleshooter has blind spots. It evaluates allow, deny and (in beta) Principal Access Boundary policies, but not VPC Service Controls, Cloud Storage ACLs or allow policies on Pub/Sub resources.
Wait before you widen. Allow and deny policy changes typically propagate in about 2 minutes but can take 7 minutes or longer, and group membership changes can take hours.
Never answer a 403 with Owner or Editor. Granting a basic role hides the real cause and leaves a service account far more powerful than its job.

Reading the error

Google APIs return PERMISSION_DENIED as HTTP 403. Google's API design guidance (AIP-193) says services must check permission before checking whether a resource exists, so a 403 can also mean "it may not exist", which the REST message states explicitly. Google documents these forms:

Surface	Exact text	Useful fields
REST API	`Permission 'storage.buckets.list' denied on resource (or it may not exist).` followed by a Troubleshooter URL	HTTP `403`; `ErrorInfo` with `reason`, `domain`, metadata `permission` and `error_info_id`
gcloud CLI	`ERROR: (gcloud.storage.buckets.list) PERMISSION_DENIED:` then "EMAIL_ADDRESS does not have storage.buckets.list access to the Google Cloud project."	`reason: IAM_PERMISSION_DENIED`, `domain: storage.googleapis.com`, `permission`, `error_info_id`
Console	Error page with the resource name, missing permissions and roles that contain them	"Select a role to grant" or "Request a specific role", plus a Policy Troubleshooter link
VPC Service Controls	`Request is prohibited by organization's policy`; for Cloud Storage `Request violates VPC Service Controls.`	`vpcServiceControlsUniqueIdentifier` in the message and audit log

The error_info_id identifies the specific denial so it can be looked up in Policy Troubleshooter. Capture it, the principal, the permission and the full resource name before changing anything.

Causes, most common first

#	Cause	Tell-tale sign	First check
1	No allow policy binding grants the permission	`IAM_PERMISSION_DENIED` naming a permission	Policy Troubleshooter: allow policy section
2	Wrong principal is calling	The email in the error is not the one you expected	Which credentials did ADC or the metadata server supply?
3	Recent change not yet propagated	Grant made minutes ago, or principal added to a group	Change timestamp versus request time
4	Conditional binding not satisfied	Role is present but has a condition (time, resource name, tag)	Troubleshooter condition evaluation
5	Deny policy	Role is present and unconditional	Deny policies on the organisation, folders and project
6	Principal Access Boundary policy	Principal has the role but the resource is outside its boundary	Troubleshooter in beta with PAB evaluation
7	VPC Service Controls perimeter	Different message and a `vpcServiceControlsUniqueIdentifier`	Violation analyzer
8	Quota project or API enablement with user credentials	Local development only; message mentions quota project, API not enabled or unsupported user credentials	ADC quota project

Diagnosing each cause

1 and 4: missing or conditional allow binding

Where to look: Policy Troubleshooter in the console (from the error page link) or gcloud policy-troubleshoot iam with the full resource name, principal and permission. It reports which bindings were checked at each level of the hierarchy and how conditions evaluated.
Who can run it: on the organisation, Security Reviewer (roles/iam.securityReviewer), plus Deny Reviewer (roles/iam.denyReviewer) for deny policies; group membership needs the Workspace groups.read privilege. Without these, results show as Unknown, which is not the same as denied.
Fix: grant the narrowest predefined or custom role containing the named permission, at the lowest resource level that works. The console lists candidate roles; prefer the smallest. If a condition failed, fix the request (resource name, tag) or the condition deliberately.

2: the wrong principal

Where to look: the email in the error. Application Default Credentials follow an order, and GOOGLE_APPLICATION_CREDENTIALS overrides other sources, including Workload Identity Federation for GKE. A forgotten key file therefore makes a pod run as a different service account.
Remember: the gcloud CLI uses gcloud auth login credentials, not ADC, so a command can succeed in the shell while the application fails.
Fix: remove the stray key file or variable and confirm the intended identity. See the Google Cloud Service Account Keys Security Guide.

3: propagation

Google documents allow and deny policy changes as "Typically 2 minutes, potentially 7 minutes or longer", and group membership changes, including nested groups, as "Typically several minutes, potentially hours or longer". Additions usually propagate faster than removals. Retry with exponential backoff for several minutes after a grant before treating the error as a configuration fault.

5: deny policies

How they work: each deny policy is attached to an organisation, folder or project and inherited downwards. Denied permissions use the v2 format SERVICE_FQDN/RESOURCE.ACTION, for example iam.googleapis.com/roles.delete. exceptionPrincipals and exceptionPermissions carve out exemptions.
Where to look: the deny section of Policy Troubleshooter, or list deny policies at each ancestor. An organisation-level denial applies everywhere below it.
Fix: deny policies usually encode a deliberate guardrail. Raise an exception request with the policy owner rather than working around it, and add the principal to exceptionPrincipals only through change control.

6: Principal Access Boundary policies

How they work: PAB policies are bound to principal sets and define which resources those principals are eligible to access. They can only block, never grant. Access is blocked when the principal is subject to at least one PAB policy and none of them includes the resource.
Gotchas: a policy blocks only permissions in its enforcement version; Google advises against latest because it can remove access unexpectedly when a new version ships. Evaluation fails closed, which most often affects newly created principals still propagating; a later retry usually succeeds.
Where to look: run the Troubleshooter in beta (gcloud beta policy-troubleshoot iam), which needs Principal Access Boundary Policy Viewer on the organisation.

7: VPC Service Controls

How to recognise it: the message says the request is prohibited by organisation policy or violates VPC Service Controls, and carries a unique identifier.
Violation reasons: NO_MATCHING_ACCESS_LEVEL (caller IP, device or identity not matched), RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER, NETWORK_NOT_IN_SAME_SERVICE_PERIMETER, SERVICE_NOT_ALLOWED_FROM_VPC.
Fix: look the identifier up in the violation analyzer and add a narrow ingress or egress rule; Google recommends ingress rules over broad access levels. IAM changes will never fix this. See the VPC Service Controls Guide.

8: quota project and user credentials

With local user ADC, some APIs fail because no quota project is set or the API is not enabled in it. Google's fixes are gcloud auth application-default set-quota-project PROJECT, gcloud config set billing/quota_project PROJECT, or the x-goog-user-project header, and the caller needs serviceusage.services.use (in Service Usage Consumer). This is a developer workstation issue, not a production authorisation gap.

Organisation policy is not PERMISSION_DENIED

Organisation Policy constraints such as iam.disableServiceAccountCreation restrict configurations rather than principals. They are enforced by the service and often return a different status and message naming the constraint, so a request blocked by a constraint will not show a missing permission in Policy Troubleshooter. Constraints also support dry-run, where violations are audit-logged but allowed. Treat a constraint block as a design question for the policy owner. See the Google Cloud Organization Policy Guide.

A step-by-step triage procedure

Capture the full error: principal, permission, resource, reason, domain, error_info_id or VPC SC identifier, timestamp.
Branch on the message: VPC SC wording goes to the violation analyzer; quota project wording goes to ADC settings; everything else continues.
Confirm the principal is the one intended. If not, fix credentials, not IAM.
Check timing: if a relevant grant or group change happened recently, retry with backoff for up to 10 minutes.
Run Policy Troubleshooter (beta for PAB). Read allow, deny and boundary results separately.
Apply the smallest fix through infrastructure as code: a narrow role on the lowest resource, a corrected condition, or an exception request.
Verify and roll back: re-run the request; if it still fails, revert the change rather than adding another role.
Record the cause and add a test or policy check that would catch it.

Decision guidance

Finding	Right response	Wrong response
Missing permission for a service account	Custom or narrow predefined role at resource level	Editor on the project
Missing permission for a person, occasional need	Time-bound grant through Privileged Access Manager or a conditional binding with expiry	Standing Owner
Deny policy blocks a legitimate job	Exception agreed with the guardrail owner	Moving the project to a folder without the deny policy
Perimeter blocks a pipeline	Ingress or egress rule scoped to identity and service	Removing the project from the perimeter
Intermittent 403 after changes	Retry with backoff; check group nesting	Repeated delete and re-grant

Small organisations can resolve most cases with Policy Troubleshooter and direct bindings. Large estates with deny policies, PAB and perimeters need a documented escalation path to each guardrail owner and should give the platform team Security Reviewer and Deny Reviewer on the organisation so the Troubleshooter returns complete results.

Where identity fits

Most production 403s involve service accounts, workload identities or agent identities rather than people. Each fix is a permanent change to a non-human identity's privileges, so the pressure of an outage is exactly when over-granting happens. Review the result afterwards with How to Find GCP Service Accounts With Owner or Editor Roles and the Google Cloud IAM and Service Accounts Guide. Federation-specific denials such as iam.serviceAccounts.getAccessToken are covered in the Multi-Cloud Workload Identity Troubleshooting Guide.

Detection and measurement

Audit logs: denied calls appear in Cloud Audit Logs with status code 7 (PERMISSION_DENIED); Data Access logs are off by default for most services, so enable them where denials matter.
Policy changes: alert on SetIamPolicy granting roles/owner or roles/editor, and on deny policy and perimeter changes.
Recommender: use IAM role recommendations to remove permissions granted during incidents but never used.

Metrics:

403 rate per service account, grouped by reason.
Median time from first 403 to resolution.
Number of basic role grants (Owner, Editor) made in response to access tickets.
Share of fixes made through reviewed infrastructure as code.
VPC SC violations per perimeter, split into dry-run and enforced.

Common failure patterns

Editor as a fix: a 403 is "solved" with roles/editor, and the service account keeps it for years. See Top 10 GCP IAM Mistakes.
Granting where the deny is: engineers add roles repeatedly while an organisation deny policy blocks the permission.
Trusting an Unknown result: the Troubleshooter returns Unknown when the operator lacks reviewer roles, and it is read as "allowed".
IAM changes for a perimeter block: VPC SC errors are worked on as IAM problems for hours.
Stray key files: GOOGLE_APPLICATION_CREDENTIALS left in an image makes the workload run as an old, over-privileged service account. Leaked keys of this kind drove the Codecov breach, where a cloud storage key exposed in a build image let attackers alter a CI script.
Group nesting surprises: nested group membership takes longest to propagate, so a fix appears not to work.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev. 5	AC-3 Access Enforcement; AC-6 Least Privilege	Layered policy enforcement; narrow fixes
NIST SP 800-53 Rev. 5	AU-12 Audit Record Generation; AU-6 Audit Record Review	Logging and reviewing denials
NIST SP 800-53 Rev. 5	CM-3 Configuration Change Control	IAM changes through reviewed code
NIST CSF 2.0	PR.AA-05	Access permissions defined, enforced and reviewed
CIS Controls v8.1	6.8 Define and Maintain Role-Based Access Control	Roles sized to the job
ISO/IEC 27001:2022 Annex A	5.15 Access control; 5.18 Access rights	Granting and reviewing access rights
MITRE ATT&CK	T1098.003 Additional Cloud Roles; T1078.004 Cloud Accounts	Over-granting creates the privileges attackers seek

Practitioner checklist

Capture principal, permission, resource, reason and error_info_id before changing anything.
Confirm the calling identity, including any GOOGLE_APPLICATION_CREDENTIALS override.
Wait and retry with backoff for propagation after recent grants or group changes.
Run Policy Troubleshooter with reviewer roles, in beta when PAB policies exist.
Route VPC SC identifiers to the violation analyzer, not to IAM.
Check deny policies at every ancestor before adding roles.
Fix with the narrowest role at the lowest resource, in code, and verify.
Never grant Owner or Editor to resolve a ticket; review incident grants within a week.
Track 403s by reason and basic-role grants as metrics.