HashiCorp Vault Permission Denied Troubleshooting Guide

A HashiCorp Vault "permission denied" (HTTP 403) tells you the request was refused, but not whether the token was wrong, the policy path was wrong, the namespace was wrong or the login itself never succeeded. Because Vault is deny by default, almost every misconfiguration ends in the same message, and teams lose hours widening policies until the error disappears. This guide separates the causes from most to least common, shows how to prove each one with vault token capabilities, token lookup and the audit log, and gives the fix and the prevention for each. It is for platform engineers, Vault operators and application teams using Vault from CI/CD, Kubernetes and services, on self-managed Vault Community or Enterprise and on HCP Vault.

Key takeaways

• Find out which request failed and with which token. A 403 at login is an auth method problem; a 403 after login is a policy, path or namespace problem. They need different fixes.
• Ask Vault, do not guess. vault token capabilities <path> returns the capabilities the token has on a path; deny confirms a policy gap without any trial and error.
• KV version 2 changes the policy path. The CLI path secret/app is served by the API at secret/data/app and secret/metadata/app, so a policy written for the CLI path matches nothing.
• Token policies are fixed at creation. Editing a policy's contents takes effect on the next request, but adding a new policy to a role does not change tokens already issued; the client must log in again.
• Fix with the narrowest grant that works. Use the audit log and -output-policy to write the exact path and capability, never path "*" or the root token.

How Vault decides: a 30-second model

Every request carries a token (or none). Vault resolves the token's policies, which are the token policies set at creation plus identity policies computed from the token's entity and groups, then evaluates the request path and HTTP verb against those policies. HashiCorp documents that policies are deny by default, that deny overrides every other capability including sudo, and that capabilities map to HTTP verbs rather than to what the operation does: generating database credentials is a GET, so it needs read, not create. Where several path rules match, the most specific wins, using HashiCorp's priority rules (an earlier wildcard, a trailing *, more + segments, or a shorter path all lower priority); identical patterns in different policies are unioned.

Step one: classify the failure

HashiCorp's troubleshooting tutorial draws the same line: "missing client token" means no valid token was supplied, which is a different problem from a policy denial.

Likely causes, most to least common

1. Policy path does not match the API path

The single most common cause. With the KV version 2 secrets engine, data lives under <mount>/data/ and listing and version information under <mount>/metadata/, as HashiCorp's KV v2 setup documentation shows. A policy with path "secret/app/*" grants nothing on secret/data/app/config. Other path mistakes: a missing trailing * (paths match exactly unless they end in a glob), a * placed mid-path (it is only valid as the last character and is not a regular expression; use + for a single segment), the wrong mount name, and a templated path such as secret/data/{{identity.entity.id}}/* that renders differently from what you expect.

Diagnose: run vault token capabilities secret/data/app/config as the failing identity. deny confirms the gap. Read the policy with vault policy read <name> and compare the path character by character with the request path in the audit log.

Fix: write the API path. For KV v2, a read-only application typically needs read on <mount>/data/<app>/* and, if it lists, list on <mount>/metadata/<app>/*. HashiCorp's KV v2 documentation describes running a vault kv command with -output-policy to print the minimal policy that command needs, which is the quickest way to get it right. Note that KV policies do not support allowed_parameters, denied_parameters or required_parameters.

2. Wrong capability for the verb

The path matches, but the capability does not: read granted where the client writes, update missing for an engine endpoint that uses POST, patch missing for vault kv patch, or sudo missing on a root-protected path such as sys/audit, sys/auth/:path, sys/raw or sys/leases/revoke-prefix/:prefix. A deny in any attached policy, including one attached through an identity group, also wins over every grant.

Diagnose: vault token capabilities lists what the token holds on the path; compare with the verb in the audit log entry for the failed request. Search all attached policies, not just the one you expect, for a deny.

Fix: add the specific capability on the specific path. For root-protected endpoints add sudo only to an operator policy, not to application policies.

3. The token was issued before the fix

HashiCorp documents that token policies are set when the token is created and cannot be changed afterwards; to change them you revoke the token and get a new one. Editing the contents of an existing policy does apply on the next request, because policies are parsed on each use. So adding a missing path to a policy the token already has works immediately, but attaching a new policy to an auth role (for example token_policies on a Kubernetes or AppRole role) has no effect until the client logs in again. Identity policies, by contrast, follow entity and group membership changes.

Diagnose: vault token lookup (or auth/token/lookup-self) shows policies, identity_policies and the creation time. If the policy you just attached is not in the list, this is the cause.

Fix: trigger a fresh login, restart the agent or sidecar, or wait for the token to renew through a new login. Shorten token TTLs where clients regularly hold stale policy sets.

4. Wrong namespace

On Vault Enterprise and HCP Vault, a namespace is an isolated environment with its own mounts, auth methods, policies and tokens. The same path in the wrong namespace either does not exist or is not covered by the token's policies, and both show up as permission denied. The namespace is selected per request with the X-Vault-Namespace header or by prefixing the path; the CLI reads the VAULT_NAMESPACE environment variable or the -namespace flag. On HCP Vault, clients usually need the admin namespace set.

Diagnose: check the namespace in the audit log entry for the request and in the token lookup output. A token issued in one namespace used against another is a common pattern in CI jobs that set the variable only for some steps.

Fix: set the namespace consistently in the client, and write policies in the namespace that owns the path. Parent namespaces can assert policies on identities in child namespaces through group policy application; keep that deliberate and documented.

5. The token is not reaching Vault, or it is the wrong token

The client sends no token, an expired token, a token from another cluster, or a different token than you think: a stale VAULT_TOKEN exported in a shell, a cached ~/.vault-token, or an agent sink file that has not been refreshed. Expired or revoked tokens produce permission denied as well.

Diagnose: run vault token lookup with exactly the same environment as the failing process. Check the token's TTL and whether it is an orphan whose parent was revoked.

Fix: remove stray token variables and files, use Vault Agent or the platform integration to manage renewal, and handle renewal failures in the application.

6. The login itself is denied

Here the 403 comes from the auth method. For the Kubernetes auth method, HashiCorp's documentation points to role and config parameters: the service account and namespace must match bound_service_account_names and bound_service_account_namespaces; the audience, if set on the role, must match the JWT; the reviewer token used for TokenReview needs the system:auth-delegator ClusterRole; and kubernetes_host and kubernetes_ca_cert must be right. Short-lived Kubernetes tokens fail once the pod or service account is deleted or the token expires, and Vault versions before 1.9.3 read the local reviewer token once and failed after it rotated. For AppRole, check the role ID, a secret ID that has not expired or exceeded its use count, and any CIDR binding. For JWT and OIDC roles, check bound claims, audience and issuer.

Diagnose: vault read auth/<mount>/role/<role> and vault read auth/<mount>/config, then compare with the claims in the client's JWT (decode it locally, never paste it into an online decoder).

Fix: correct the binding or configuration, keeping bindings as narrow as they were. Widening a binding to * to make login work lets any service account in the cluster read that role's secrets. See the Kubernetes Secrets Security Guide.

Reading the audit log

HashiCorp's troubleshooting tutorial recommends querying the audit log for entries with a non-empty error field, where policy refusals appear as "permission denied", and reviewing the server log for ERR and nearby WARN lines (journalctl -b --no-pager -u vault on systemd hosts). Each audit entry records the request path, operation, namespace and the policies on the token, which is enough to write the exact grant. Sensitive string values, including client tokens, are written as HMAC-SHA256 hashes by default; use the sys/audit-hash endpoint to hash a known value and search for it. The tutorial also describes enabling a temporary audit device with log_raw=true to reproduce a problem; treat that as a last resort on a controlled system, because it writes secrets in clear text, and disable it immediately afterwards. Vault refuses to serve requests if it cannot write to any enabled audit device, so HashiCorp recommends at least two devices.

For a wider review of tokens and policies, see How to audit HashiCorp Vault tokens, policies and auth methods.

Safe fix workflow

1. Reproduce with the failing identity, not with an admin token, and capture the request path, verb and namespace.
2. Confirm the gap with vault token capabilities and vault token lookup.
3. Draft the smallest change: one path, the needed capabilities, in the right namespace. Generate it with -output-policy where available.
4. Test in a non-production namespace or cluster with a test token created from the policy, as HashiCorp's tutorial suggests.
5. Apply through version control (Terraform Vault provider or a reviewed policy repository) so the previous policy is one revert away.
6. Force fresh tokens if you attached a new policy rather than editing an existing one.
7. Roll back by restoring the previous policy version; tokens pick up the restored contents on their next request.

Where identity fits

Most Vault clients are non-human identities: pipelines, pods and services. A permission denied is often the first sign that one of them has no clear owner or a policy nobody understands. The pressure during an outage is to grant path "*" or hand out a root token, which turns a single compromised workload into a compromise of every secret Vault holds; HashiCorp itself recommends revoking root tokens in production. Pin each workload to an entity, use templated policies keyed on entity IDs rather than names, and keep policy per workload. See the HashiCorp Vault Guide and the Credential Vaulting for AI Agents Guide.

Detection and measurement

• Ship audit logs to the SIEM and alert on bursts of permission denied from one token accessor or one source address, which can indicate a credential being used to probe paths.
• Alert on policy and auth changes: writes to sys/policy/ or sys/policies/acl/, sys/auth/ and auth role paths.
• Track wildcard policies and any use of tokens with the root policy.

Common failure patterns

• Writing KV v1 paths for a KV v2 mount, then adding secret/* to "fix" it, which grants far more than intended.
• Debugging with the root token, which proves nothing about the failing identity and leaves a root token in shell history.
• Forgetting the default policy: tokens created with -no-default-policy lose self-lookup and renewal paths, and then fail in confusing ways.
• Leaving log_raw on after troubleshooting, so secrets sit in clear text in the audit log.
• Widening Kubernetes role bindings to every service account or namespace to fix one login.
• Putting secrets in key names: HashiCorp notes that LIST results are not filtered by policy, so names are visible to anyone with list.

Framework mapping

Practitioner checklist

• Decide first whether the 403 is at login or after login.
• Run vault token capabilities and vault token lookup as the failing identity.
• For KV v2, write policies on data/ and metadata/ paths.
• Check every attached policy, including identity policies, for deny.
• Confirm the namespace in the client, the token and the audit entry.
• Force a new login after attaching a new policy to a role.
• Keep auth role bindings narrow when fixing login failures.
• Manage policies in version control and test with a token built from the policy.
• Run at least two audit devices and never leave log_raw enabled.

Standards and references

• HashiCorp: Vault policies
• HashiCorp: vault token capabilities command
• HashiCorp: Troubleshoot Vault tutorial
• HashiCorp: Set up the KV v2 plugin
• HashiCorp: Vault namespaces
• HashiCorp: Vault audit devices
• HashiCorp: Kubernetes auth method
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls

Related NHI Mgmt Group resources: HashiCorp Vault Guide, How to audit HashiCorp Vault tokens, policies and auth methods, Kubernetes Secrets Security Guide, HashiCorp Vault Associate.