Workload identity federation removes static cloud keys, but when it fails the error messages are terse, differ by cloud, and tempt engineers to "fix" the outage by loosening a trust policy or putting a long-lived key back. This guide is a diagnostic reference for the three major clouds and their Kubernetes integrations: AWS (AssumeRoleWithWebIdentity, IRSA and EKS Pod Identity), Google Cloud (Workload Identity Federation and Workload Identity Federation for GKE) and Microsoft Entra (federated identity credentials and AKS workload identity). It explains where each exchange can break, maps the real error strings to causes, and shows how to fix each one without weakening the trust boundary. It is for platform, cloud and DevSecOps engineers. The concepts and design choices are covered in the Cloud Workload Identity Guide, which this page does not repeat. Error strings and limits were checked against AWS, Google Cloud and Microsoft documentation on 28 October 2026.
Key takeaways
- Every federation failure is one of five things: the token never reached the workload, the cloud cannot validate the issuer, a claim does not match the trust, the permission grant is missing or not yet propagated, or the resulting token is used somewhere it is not accepted.
- Decode the token before changing anything. Compare
iss,sub,aud,expandiatin the presented token with the trust configuration, character for character. Entra matching is case-sensitive. - Propagation delays cause real errors. Entra returns AADSTS70021 shortly after a credential is added, and Google Cloud IAM grants usually take about two minutes but can take seven or more. Retry with backoff before you redesign.
- Never fix a mismatch with a wildcard. Broadening
subto a namespace, organisation or repository wildcard turns an outage into an exposure. Fix the claim or the workload instead. - Static keys silently win. On EKS, credentials found earlier in the SDK chain (such as static keys in environment variables) take precedence over Pod Identity, so "it works" can mean federation is not being used at all.
How a federated exchange breaks: the five failure points
All three clouds follow the same shape: a workload obtains a signed JWT from an issuer (Kubernetes API server, CI platform, another cloud), presents it to a security token service, and receives a short-lived cloud credential if the issuer, subject and audience satisfy a trust configuration and the identity holds a permission grant.
| Failure point | Typical symptom | First check |
|---|---|---|
| 1. Token acquisition (workload side) | File not found, empty token, SDK falls back to another credential, timeouts at pod start | Is the projected token mounted and the expected environment variable present in the container? |
| 2. Issuer validation | Cloud cannot reach discovery document or JWKS, unsupported key format, invalid token | Can the cloud reach {issuer}/.well-known/openid-configuration and the jwks_uri from the internet? |
| 3. Claim matching | "No matching federated identity record", IDPRejectedClaim, attribute condition rejection | Decode the token and diff iss, sub, aud against the trust exactly |
| 4. Authorisation | Exchange succeeds but next call is denied, iam.serviceAccounts.getAccessToken denied | Is the role binding or role trust present, and has it had time to propagate? |
| 5. Token use | 401 from a specific API, scope errors, session too large | Does the target API accept federated tokens, and is the scope or audience format right? |
Work through them in order. Most time is lost by jumping to step 4 and adding permissions when the real fault is at step 1 or 3.
Reading the token: the universal first step
- Capture the presented token safely. Read it from the workload's token file or CI step in a non-production run, decode the payload locally, and never paste production tokens into online decoders. Treat the token as a secret until it expires.
- Compare
iss. It must equal the configured issuer exactly, including scheme and trailing path. Entra notes that leading or trailing whitespace in the issuer blocks the exchange. - Compare
sub. Kubernetes service account tokens usesystem:serviceaccount:<namespace>:<name>. CI platforms encode repository, branch, environment or pipeline into the subject, and a renamed branch or environment changes it. - Compare
aud. Each cloud expects a specific audience:sts.amazonaws.comfor IRSA,pods.eks.amazonaws.comfor EKS Pod Identity, and one value (recommendedapi://AzureADTokenExchange) for Entra federated identity credentials. Google Cloud providers accept the audience configured on the provider. - Check
expandiat. An expired or not-yet-valid token fails as ExpiredToken or InvalidIdentityToken on AWS. Check the node or runner clock ifiatis in the future. - Check the signing algorithm in the header. AWS STS accepts RSA (RS256, RS384, RS512) and ECDSA (ES256, ES384, ES512) signatures; Microsoft says only RS256 issuers have been tested.
AWS: AssumeRoleWithWebIdentity, IRSA and EKS Pod Identity
Error reference
| Error | HTTP | Meaning (AWS STS API reference) | Common real cause and fix |
|---|---|---|---|
InvalidIdentityToken | 400 | The token could not be validated | OIDC provider not registered in IAM for this issuer, or issuer URL differs from the token's iss. Register the exact issuer as an IAM OIDC provider. |
ExpiredToken | 400 | The web identity token is expired or not valid | Cached token reused after expiry, or clock skew. Re-read the token file on each exchange. |
IDPRejectedClaim | 403 | Authentication failed, possibly an invalid, expired or revoked claim | Usually the role trust conditions on :sub or :aud do not match. Fix the trust condition to the exact value. |
IDPCommunicationError | 400 | AWS could not reach the identity provider | Often transient. Retry a limited number of times; check the issuer's public endpoints. |
PackedPolicyTooLarge | 400 | Session policies and tags exceed the size limit | Too many session tags or a large inline session policy. Trim them. |
RegionDisabled | 403 | STS not activated in that region for the account | Activate the regional endpoint in IAM account settings or use an enabled region. |
AccessDenied on the role | 403 | Not an STS error code in the table, returned when the trust policy does not allow the caller | Trust policy principal or action wrong (sts:AssumeRoleWithWebIdentity for IRSA). |
Also check that DurationSeconds is between 900 and 43,200 and does not exceed the role's maximum session duration, and that the role session name is 2 to 64 characters from [\w+=,.@-].
IRSA checks
- Trust principal:
arn:aws:iam::<account>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<ID>, the cluster issuer withouthttps://. - Conditions:
<provider>:audequalssts.amazonaws.comand<provider>:subequalssystem:serviceaccount:<namespace>:<name>underStringEquals. A typo in the namespace is the most common IRSA fault. - Annotation: the service account carries
eks.amazonaws.com/role-arn; pods created before the annotation do not pick it up until restarted.
EKS Pod Identity checks
- Agent: the EKS Pod Identity Agent runs on the node and serves credentials at
http://169.254.170.23/v1/credentials, calling the EKS AuthAssumeRoleForPodIdentityaction. - Injection: the pod should contain
AWS_CONTAINER_CREDENTIALS_FULL_URIandAWS_CONTAINER_AUTHORIZATION_TOKEN_FILE, with a projected token of audiencepods.eks.amazonaws.com. - Credential chain: static keys in environment variables or a mounted credentials file are found first. Remove them, then confirm the caller identity is the Pod Identity role session.
For trust policy design and cross-account patterns see the AWS STS and AssumeRole Guide and the EKS Identity Hardening Guide.
Google Cloud: Workload Identity Federation and GKE
Error reference
| Error text (Google Cloud documentation) | Cause | Fix |
|---|---|---|
"Error connecting to the given credential's issuer." (invalid_grant) | Google cannot fetch the issuer's discovery document or JWKS | Make the endpoints publicly reachable and OIDC-compliant; confirm the token's iss |
| "Input JWK is not in a valid json format" | Uploaded JWK contains x5c or x5t | Remove those fields and reconfigure the provider |
"The size of mapped attribute google.subject exceeds the 127 bytes limit." (invalid_request) | Mapped subject too long (common with long CI subjects) | Shorten with extract() in the mapping |
| "Org Policy violated for value: '{PROVIDER}'." | constraints/iam.workloadIdentityPoolProviders does not allow that issuer | Add the provider to the allowlist through policy change control |
| "Request had invalid authentication credentials." (401) | A federated STS token was sent to an API that does not accept federated tokens | Use service account impersonation (GenerateAccessToken) for that API |
"The given AWS request doesn't contain all the required headers." (invalid_grant) | AWS-to-Google exchange with wrong signed headers | Include only the documented headers with an empty body |
quota_exceeded, rate limit on sts.googleapis.com/requests (429) | Too many token exchanges | Cache credentials until near expiry; request a quota increase if needed |
| "Permission 'iam.serviceAccounts.getAccessToken' denied" (403) | Missing roles/iam.workloadIdentityUser binding, or propagation delay | Add the binding on the service account; allow minutes for propagation |
Mapping and condition checks
- Attribute mapping:
google.subjectis required for OIDC and SAML providers and must be unique; up to 50 customattribute.NAMEmappings are allowed. - Attribute conditions: a CEL expression that must return true. A rejection after a CI change usually means the condition references a claim value (repository, ref, workflow) that changed. Fix the workload or update the condition deliberately; do not delete it.
- Principal identifiers:
principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/VALUEuses the project number, not the project ID. Using the ID is a frequent silent mismatch.
Workload Identity Federation for GKE checks
- Node pool: confirm the cluster has a workload pool and the node pool's
workloadMetadataConfig.modeis the GKE metadata server mode. - Start-up race: the GKE metadata server needs a few seconds before it accepts requests from a new pod. Add retries or an init container; with Istio, use
holdApplicationUntilProxyStartsor exclude169.254.169.254/32from redirection. - Linked service accounts: the IAM service account needs
roles/iam.workloadIdentityUserforPROJECT_ID.svc.id.goog[NAMESPACE/KSA], and the Kubernetes service account needs theiam.gke.io/gcp-service-accountannotation. If a binding vanished, thesetIamPolicyaudit log shows who removed it. - Scale:
gke-metadata-serverpods can be OOM-killed in clusters with more than 3,000 Kubernetes service accounts.
See the GKE Identity Hardening Guide for the hardening side.
Microsoft Entra: federated identity credentials and AKS
Error and limit reference
| Error or limit | Meaning | Fix |
|---|---|---|
| AADSTS70021 "No matching federated identity record found for presented assertion." | No credential matches the token's issuer, subject and audience, or the credential has not propagated yet | Wait and retry; then diff the three values case-sensitively |
| AADSTS700212 (AKS) | Wrong projected token used: identity binding tokens use audience api://AKSIdentityBinding, direct federation needs api://AzureADTokenExchange | Mount and read the token file with the correct audience |
| AADSTS50013 InvalidAssertion | Assertion invalid: wrong issuer, expired or malformed | Check exp, issuer and token format |
| "Federated identity credentials must have exactly one audience" (400) | More than one audience configured | Use a single audience |
| "Issuer and subject combination already exists" (400) | Duplicate issuer and subject on the same identity | Reuse the existing credential |
| 409 Conflict | Concurrent writes of credentials under one user-assigned identity | Create sequentially (ARM "mode": "serial" or dependsOn) |
| 20 credentials per app or user-assigned managed identity | Hard limit | Consolidate subjects or split identities by workload |
| No wildcards in any credential property | Exact match only | One credential per distinct subject |
Entra stores only the first 100 signing keys from an external issuer's discovery endpoint, so an issuer publishing more may fail intermittently.
AKS workload identity checks
- Pod label:
azure.workload.identity/use: "true"must be on the pod template. Without it the webhook does not inject configuration and pods can fail after a restart. - Service account annotations:
azure.workload.identity/client-idand optionallyazure.workload.identity/tenant-id; restart pods after changing them. - Token file: read
AZURE_FEDERATED_TOKEN_FILEat runtime and re-read it on each exchange, because Kubernetes refreshes it in place. - Scopes: request v2 scopes such as
https://management.azure.com/.default; a raw resource URI can fail.
See the AKS Identity Hardening Guide.
Cross-cloud federation: when one cloud's identity calls another
- AWS to Google Cloud: the AWS provider maps
google.subjectto the caller's ARN by default. Assumed-role ARNs include the session name, so use an attribute condition on the role rather than binding a single session ARN. - Kubernetes to another cloud: the cluster's issuer must be publicly discoverable for the remote cloud to validate it. Private clusters often need the issuer discovery documents published to a public endpoint.
- CI to cloud: CI subjects change with branch, environment and event type. Pin trust to repository plus a protected branch or environment. See the GitHub Actions OIDC Federation Hardening Guide.
- Chained exchanges: when a token from one exchange is used for the next, check each hop separately and keep the session name or subject traceable to the original workload.
Logging and detection while you troubleshoot
| Cloud | Where exchanges are logged | What to alert on |
|---|---|---|
| AWS | CloudTrail AssumeRoleWithWebIdentity and Pod Identity session events, with role session name and session tags | Exchanges from an unexpected issuer or subject; trust policy edits (UpdateAssumeRolePolicy) |
| Google Cloud | Cloud Audit Logs for the Security Token Service and IAM (setIamPolicy), with google.subject in the principal | New pool providers, attribute condition changes, new workloadIdentityUser bindings |
| Microsoft Entra | Service principal and managed identity sign-in logs; audit logs for federated credential changes | New federated identity credentials, repeated AADSTS70021 from unknown issuers |
Map detections to MITRE ATT&CK T1078.004 Cloud Accounts and T1550.001 Application Access Token. Metrics that show the federation estate is healthy:
- Exchange failure rate per workload, with failures grouped by error code.
- Mean time to resolve federation incidents.
- Number of trust policies or credentials using wildcard or namespace-wide subjects (target: zero or documented exceptions).
- Workloads still holding static cloud keys alongside a federation path.
- Trust configuration changes made outside infrastructure as code.
Common failure patterns
- Loosening the subject to end an outage: replacing an exact
subwithsystem:serviceaccount:ns:*or a repository-wide wildcard lets every workload in scope assume the role. Fix the claim, and record any temporary widening as an exception with an expiry. - Reintroducing a static key "temporarily": the key stays for years, and on EKS it silently takes precedence over federation. Detect keys on workloads that also have a federation path.
- Caching the projected token: applications read the token once at start and fail at expiry with ExpiredToken or AADSTS50013. Re-read the file for every exchange.
- Project ID instead of project number: Google Cloud principal identifiers silently fail to match.
- Treating propagation as a misconfiguration: engineers delete and recreate correct credentials during the propagation window, causing 409 conflicts or further delay. Add retries with backoff first.
- Unbounded CI trust: trust bound to an organisation rather than a repository and protected branch lets any compromised workflow mint cloud tokens.
Lessons from incidents
In the Megalodon GitHub Actions attack, injected workflows harvested CI secrets from thousands of repositories, including long-lived AWS, Google Cloud and Azure credentials and GitHub Actions OIDC tokens. Federation removes the stored keys, but a compromised workflow can still request tokens. The trust conditions you debug are your security boundary: keep subject conditions tied to specific repositories, branches or environments, and alert on exchanges from runs you did not expect.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev. 5 | IA-9 Service Identification and Authentication | Workloads authenticate with federated identity rather than shared secrets |
| NIST SP 800-53 Rev. 5 | IA-5 Authenticator Management | Removing static keys that federation replaces |
| NIST SP 800-53 Rev. 5 | AC-6 Least Privilege | Exact subject conditions instead of wildcards |
| NIST SP 800-53 Rev. 5 | AU-12 Audit Record Generation | Logging exchanges and trust changes |
| NIST CSF 2.0 | PR.AA-01, PR.AA-05 | Managing service credentials and enforcing access policy |
| MITRE ATT&CK | T1078.004, T1550.001, T1528 | Abuse of cloud accounts and application access tokens |
Practitioner checklist
- Decode the presented token and diff
iss,sub,aud,expagainst the trust before changing anything. - Confirm the token file and injected environment variables exist in the running container.
- Confirm the issuer discovery document and JWKS are reachable by the target cloud.
- Allow for propagation: retry with backoff for several minutes after creating credentials or bindings.
- Use exact subject conditions; document and expire any temporary widening.
- Remove static keys from workloads once federation works, and verify the caller identity.
- Re-read projected tokens on every exchange; never cache them across expiry.
- Manage trust policies, pools and federated credentials in infrastructure as code, and alert on changes made outside it.
- Track exchange failure rates by error code per workload.
Standards and references
- AWS: AssumeRoleWithWebIdentity API reference
- AWS: How EKS Pod Identity works
- Google Cloud: Troubleshoot Workload Identity Federation
- Google Cloud: Troubleshoot authentication issues in GKE
- Microsoft Learn: Important considerations and restrictions for federated identity credentials
- Microsoft Learn: Use Microsoft Entra Workload ID with AKS
- NIST: SP 800-53 Rev. 5
- MITRE ATT&CK: Application Access Token
Related NHI Mgmt Group resources: Cloud Workload Identity Guide · Workload Identity Federation Guide · SPIFFE and SPIRE Guide · OAuth Token Exchange and Delegation Guide