A Kubernetes "Forbidden" error means the API server knew who you were and every authoriser declined the request, and the fastest wrong fix, binding cluster-admin, is also the most common one. This guide explains how to read the 403 message, separate authentication failures from authorisation failures, find the missing rule or binding, and fix it with the least permission. It covers users, groups and service accounts, subresources such as logs and exec, the RBAC escalation checks that block role and binding changes, and managed cluster mappings such as Amazon EKS access entries. It is for platform engineers, SREs and security teams who own cluster access for people and workloads.
Key takeaways
- 401 and 403 are different problems. 401 Unauthorized means the API server could not authenticate the caller; 403 Forbidden means it authenticated the caller and no authoriser allowed the request. Fixing RBAC never fixes a 401.
- The message names every attribute you need. User, verb, resource, API group and namespace (or cluster scope) are all in the text; a Role rule must match all of them exactly.
- Check identity with whoami, permission with can-i.
kubectl auth whoamishows the username and groups the server sees;kubectl auth can-iwith--astests any subject without editing anything. - Scope mistakes cause most 403s. A Role or RoleBinding in the wrong namespace, a namespaced binding for a cluster-scoped resource, or a service account subject without its namespace.
- Escalation checks are working as designed. "attempting to grant RBAC permissions not currently held" is a safety control. Grant the
bindorescalateverb narrowly instead of making the operator cluster-admin.
How Kubernetes decides Forbidden
Each API request passes authentication, then authorisation, then admission. Kubernetes checks the configured authorisers (for example Node, RBAC and Webhook) in sequence. If any authoriser approves or denies, that decision is final. If all have no opinion, the request is denied, and the API server responds with HTTP 403 (Forbidden). RBAC itself never denies: it either allows or returns no opinion. A 403 therefore means no RBAC rule matched, unless a webhook authoriser explicitly denied.
Admission happens after authorisation, so an admission webhook or Pod Security rejection is a different error (usually "admission webhook ... denied the request" or a pod security violation), not an RBAC 403. Check which stage rejected the request before editing roles. See the Kubernetes Admission Control Guide.
The authorisation request attributes are: user, groups, extra, verb, API group, resource, subresource, namespace and resource name. HTTP methods map to verbs as follows.
| HTTP method | RBAC verb | Common trap |
|---|---|---|
| GET, HEAD | get (one object), list (collection), watch | Controllers and informers need list and watch, not only get |
| POST | create | Some subresources such as pods/eviction are created, not updated |
| PUT | update | Server-side apply and many clients use patch instead |
| PATCH | patch | kubectl apply needs get and patch, plus create for new objects |
| DELETE | delete (one), deletecollection (many) | kubectl delete --all may need deletecollection |
The Kubernetes documentation warns that get, list and watch can all return full object content, so granting list on Secrets exposes every Secret value in scope.
Reading the error message
The API server builds the reason from the request attributes. kubectl prints it after Error from server (Forbidden):. The forms, from the API server source, are:
| Message pattern | Meaning | What to check |
|---|---|---|
pods is forbidden: User "jane" cannot list resource "pods" in API group "" in the namespace "dev" | Namespaced request with no matching rule | Role or ClusterRole bound by a RoleBinding in dev, core API group "" |
... cannot list resource "nodes" in API group "" at the cluster scope | Cluster-scoped resource or all-namespaces request | Needs a ClusterRoleBinding; a RoleBinding cannot grant cluster scope |
pods "web-0" is forbidden: User ... cannot get resource "pods/log" ... | Subresource missing | Rule needs pods/log explicitly; pods does not include it |
User "jane" cannot get path "/metrics" | Non-resource URL | ClusterRole rule with nonResourceURLs, bound by a ClusterRoleBinding |
User "system:serviceaccount:ci:deployer" cannot ... | Service account caller | Binding subject kind ServiceAccount with name deployer and namespace ci |
User "system:anonymous" cannot ... | Credentials missing or not presented | Really an authentication problem: kubeconfig, token or client certificate |
Read the API group carefully. Deployments are in apps, Ingresses in networking.k8s.io, CronJobs in batch, and custom resources in their CRD's group. A rule with apiGroups: [""] only covers the core group.
Step-by-step diagnosis
- Separate 401 from 403. "Unauthorized" or
system:anonymousin the message means authentication failed: an expired token, the wrong kubeconfig context, or (on EKS) an IAM principal with no access entry. Fix credentials first. - Confirm the identity the server sees. Run
kubectl auth whoami, which shows username and groups through the SelfSubjectReview API. Group membership from an OIDC provider or cloud mapping is often not what the user expects. - Reproduce with can-i.
kubectl auth can-i list pods -n devfor yourself, orkubectl auth can-i list pods -n target --as system:serviceaccount:dev:dev-safor a workload. Add--subresource=logfor subresources.kubectl auth can-i --list -n dev --as ...lists what the subject can do in a namespace. Impersonation requires theimpersonateverb, so cluster operators run these checks. - Find the bindings for the subject. List RoleBindings in the namespace and ClusterRoleBindings, and search subjects for the user, each of its groups, and for service accounts the
system:serviceaccountsandsystem:serviceaccounts:<namespace>groups. - Read the referenced role. Check every rule for the exact API group, resource (including subresource), verb and, if present,
resourceNames. - Check the audit log. Audit events carry the annotations
authorization.k8s.io/decision(alloworforbid) andauthorization.k8s.io/reason. With a policy atMetadatalevel or above you can see the user, groups, verb and resource of every forbidden request. See the Kubernetes Audit Log Detection Guide. - Fix narrowly and re-test. Add the missing rule or binding, re-run
can-i, then the original command.
Causes and fixes by symptom
| Symptom | Likely cause (most to least common) | Fix |
|---|---|---|
| Works in one namespace, Forbidden in another | RoleBinding exists only in the first namespace | Create a RoleBinding in each namespace, referencing a shared ClusterRole |
| Pod's service account is Forbidden | Pod runs as default because serviceAccountName is unset; binding names the wrong namespace | Set serviceAccountName; bind the subject with its namespace |
kubectl logs or exec fails, get pods works | Subresource not in the rule | Add pods/log or pods/exec; check the verb against your version's documentation with can-i --subresource |
Rule with resourceNames still Forbidden | create and deletecollection cannot be restricted by name; list and watch need a matching metadata.name field selector | Grant those verbs without resourceNames, or use field selectors |
Custom resource Forbidden for edit users | CRD's group not aggregated into default roles | Create a ClusterRole labelled rbac.authorization.k8s.io/aggregate-to-edit: "true" |
| Changing a binding's role fails validation | roleRef is immutable | Delete and recreate the binding, or use kubectl auth reconcile |
| "attempting to grant RBAC permissions not currently held" | Escalation prevention: creator lacks the permissions in the role or binding | Grant bind on the specific role, or escalate on roles, narrowly |
| Edits to aggregated or default roles disappear | Aggregation controller and auto-reconciliation overwrite them | Edit the source ClusterRoles, not the aggregated or default ones |
Escalation errors in detail
RBAC prevents users from escalating privileges by editing roles or bindings, and this is enforced at the API level even when the RBAC authoriser is not in use. You can create or update a role only if you already hold all its permissions at the same scope, or hold the escalate verb on roles or clusterroles. You can create or update a binding only if you hold all the permissions of the referenced role at the binding's scope, or hold the bind verb on that role. The Kubernetes documentation shows a pattern where a namespace owner can bind only admin, edit and view by listing them in resourceNames on the bind rule. That is the right answer to a delegation request, not a cluster-admin grant.
Managed clusters and cloud identity mapping
On managed services the username and groups come from a cloud identity mapping, so the 403 may be caused upstream of RBAC.
- Amazon EKS access entries: the cluster authentication mode must be
API_AND_CONFIG_MAPorAPIto use access entries. An entry can carry Kubernetes groups (used by your RBAC bindings) and associated access policies such asAmazonEKSViewPolicy,AmazonEKSEditPolicy,AmazonEKSAdminPolicyandAmazonEKSClusterAdminPolicy, each scoped to the cluster or to named namespaces. A 403 in one namespace often means the policy is namespace-scoped; check withaws eks list-associated-access-policies. Group names cannot start withsystem:,eks:,aws:,amazon:oriam:. See the EKS Identity Hardening Guide. - Migrating from the aws-auth ConfigMap: only the cluster creator's entry is migrated automatically; other mapped roles need access entries before switching to
APImode, or they lose access. - OIDC and other providers: group claims may be prefixed or missing, so the group your binding names never reaches the API server.
kubectl auth whoamisettles it. See the Azure Kubernetes Service Security Hardening Guide for the Entra ID case.
Choosing the right fix
| Need | Use | Avoid |
|---|---|---|
| Same access in many namespaces | One ClusterRole, a RoleBinding per namespace | ClusterRoleBinding, which grants every namespace including future ones |
| Read-only for a team | Built-in view via RoleBinding | view does not show Secrets; do not add Secrets to work around it |
| Workload calling the API | Dedicated service account, Role with exact verbs | Binding to the namespace's default service account or to system:serviceaccounts |
| Operator or controller across the cluster | ClusterRole scoped to its API groups and resources | cluster-admin or * verbs and resources |
| Delegated namespace administration | bind on admin, edit, view via resourceNames | Granting escalate broadly |
Small clusters can manage this with a few ClusterRoles applied per namespace from Git. Large estates should template RBAC per tenant, apply it with kubectl auth reconcile (test with --dry-run=client first), and review drift on a schedule. See the Kubernetes RBAC Guide for role design.
Where identity fits
Most 403s in production come from non-human identities: service accounts used by controllers, CI deployers and operators. Their permissions grow by accretion because each 403 is fixed under time pressure. A service account with a broad binding is a ready-made privilege escalation path for anyone who can run a pod that mounts it. Treat each service account as an identity with an owner and a defined set of API calls, and treat unexpected 403s from it as a possible sign of compromise: a stolen token probing the API matches ATT&CK T1613 Container and Resource Discovery, and adding cluster role bindings for persistence matches T1098.006 Additional Container Cluster Roles. See the Kubernetes NHI Security Guide and the Kubernetes Service Account Token Guide.
Detection and measurement
- Forbidden bursts: alert on audit events with
authorization.k8s.io/decision: forbidgrouped by user, especially service accounts makinglistcalls across many resource types or namespaces. - Privilege changes: alert on create or update of ClusterRoleBindings, bindings to
cluster-admin, and roles with*verbs or resources,escalate,bindorimpersonate. - Membership of system:masters: it bypasses RBAC and webhook authorisation entirely and cannot be fixed by removing bindings, so track any credential issued in that group.
| Metric | What it shows | Healthy direction |
|---|---|---|
| Median time to resolve a 403 ticket | Whether teams can self-diagnose with can-i and whoami | Falling |
Subjects bound to cluster-admin | Over-privilege from past fixes | Small, named, reviewed |
| Service accounts with ClusterRoleBindings | Workloads with cluster-wide reach | Falling |
| Forbidden events from service accounts outside deploy windows | Possible token misuse or broken releases | Investigated within one day |
| RBAC objects not managed from Git | Manual hotfix drift | Zero |
Common failure patterns
- Binding cluster-admin to make it work: the error disappears and the workload can now read every Secret and create privileged pods.
- Broad grants to mounted service accounts: in the Kubeflow cryptomining attacks, exposed dashboards and a mounted service account let attackers deploy mining workloads into ML clusters.
- Binding the default service account: every pod in the namespace that does not set
serviceAccountNameinherits the grant. - Adding Secrets to view: working around the deliberate exclusion of Secrets from
viewturns read-only users into credential readers. - Fixing 401s with RBAC: creating bindings for a user whose token is expired or whose group claim is missing, then wondering why nothing changes.
- Leaving the hotfix in place: a temporary ClusterRoleBinding made during an incident and never removed or committed to Git.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | AC-3 Access Enforcement; AC-6 Least Privilege; AC-6(5) Privileged Accounts | RBAC enforcement; narrow fixes; control of cluster-admin and system:masters |
| NIST SP 800-53 Rev 5 | AC-2 Account Management; AU-6 Audit Record Review, Analysis, and Reporting | Service account ownership; review of forbidden audit events |
| NIST CSF 2.0 | PR.AA-05; DE.CM-03 | Least-privilege permissions; monitoring of API activity |
| MITRE ATT&CK v20 | T1613 Container and Resource Discovery; T1098.006 Additional Container Cluster Roles | Forbidden bursts during discovery; binding changes for persistence |
Practitioner checklist
- Confirm the error is 403 Forbidden, not 401 Unauthorized or an admission rejection.
- Run
kubectl auth whoamito see the username and groups the API server receives. - Reproduce with
kubectl auth can-i, using--as,--subresourceand-nto match the failing request. - Match the rule exactly: API group, resource, subresource, verb, namespace and resource names.
- Use RoleBindings to ClusterRoles for multi-namespace access; reserve ClusterRoleBindings for cluster-scoped needs.
- Give each workload its own service account and bind it with its namespace.
- Answer escalation errors with narrow
bindorescalategrants, never cluster-admin. - On EKS, check the authentication mode, access entry groups and access policy scope.
- Commit every RBAC fix to Git and apply with
kubectl auth reconcile. - Alert on forbidden bursts from service accounts and on new cluster-admin bindings.
Standards and references
- Kubernetes: Authorization
- Kubernetes: Using RBAC Authorization
- Kubernetes: Authenticating
- Kubernetes: Auditing
- Kubernetes: Audit Annotations
- Kubernetes source: API server forbidden message format
- AWS: Grant IAM users access to Kubernetes with EKS access entries
Related NHI Mgmt Group resources: Kubernetes RBAC Guide · Kubernetes NHI Security Guide · Kubernetes Audit Log Detection Guide · AWS AccessDenied Troubleshooting Guide