Join our Newsletter — 33% off our NHI Course
Home› Guides› Multi-Cloud Workload Identity Troubleshooting Guide: Fixing Federation, Token…
Guide Cloud & Workload Identity

Multi-Cloud Workload Identity Troubleshooting Guide: Fixing Federation, Token and Trust Errors

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 28 October 2026 14 min read
On this page

Workload identity federation removes static cloud keys, but when it fails the error messages are terse, differ by cloud, and tempt engineers to "fix" the outage by loosening a trust policy or putting a long-lived key back. This guide is a diagnostic reference for the three major clouds and their Kubernetes integrations: AWS (AssumeRoleWithWebIdentity, IRSA and EKS Pod Identity), Google Cloud (Workload Identity Federation and Workload Identity Federation for GKE) and Microsoft Entra (federated identity credentials and AKS workload identity). It explains where each exchange can break, maps the real error strings to causes, and shows how to fix each one without weakening the trust boundary. It is for platform, cloud and DevSecOps engineers. The concepts and design choices are covered in the Cloud Workload Identity Guide, which this page does not repeat. Error strings and limits were checked against AWS, Google Cloud and Microsoft documentation on 28 October 2026.

Key takeaways

  • Every federation failure is one of five things: the token never reached the workload, the cloud cannot validate the issuer, a claim does not match the trust, the permission grant is missing or not yet propagated, or the resulting token is used somewhere it is not accepted.
  • Decode the token before changing anything. Compare iss, sub, aud, exp and iat in the presented token with the trust configuration, character for character. Entra matching is case-sensitive.
  • Propagation delays cause real errors. Entra returns AADSTS70021 shortly after a credential is added, and Google Cloud IAM grants usually take about two minutes but can take seven or more. Retry with backoff before you redesign.
  • Never fix a mismatch with a wildcard. Broadening sub to a namespace, organisation or repository wildcard turns an outage into an exposure. Fix the claim or the workload instead.
  • Static keys silently win. On EKS, credentials found earlier in the SDK chain (such as static keys in environment variables) take precedence over Pod Identity, so "it works" can mean federation is not being used at all.

How a federated exchange breaks: the five failure points

All three clouds follow the same shape: a workload obtains a signed JWT from an issuer (Kubernetes API server, CI platform, another cloud), presents it to a security token service, and receives a short-lived cloud credential if the issuer, subject and audience satisfy a trust configuration and the identity holds a permission grant.

Failure pointTypical symptomFirst check
1. Token acquisition (workload side)File not found, empty token, SDK falls back to another credential, timeouts at pod startIs the projected token mounted and the expected environment variable present in the container?
2. Issuer validationCloud cannot reach discovery document or JWKS, unsupported key format, invalid tokenCan the cloud reach {issuer}/.well-known/openid-configuration and the jwks_uri from the internet?
3. Claim matching"No matching federated identity record", IDPRejectedClaim, attribute condition rejectionDecode the token and diff iss, sub, aud against the trust exactly
4. AuthorisationExchange succeeds but next call is denied, iam.serviceAccounts.getAccessToken deniedIs the role binding or role trust present, and has it had time to propagate?
5. Token use401 from a specific API, scope errors, session too largeDoes the target API accept federated tokens, and is the scope or audience format right?

Work through them in order. Most time is lost by jumping to step 4 and adding permissions when the real fault is at step 1 or 3.

Reading the token: the universal first step

  1. Capture the presented token safely. Read it from the workload's token file or CI step in a non-production run, decode the payload locally, and never paste production tokens into online decoders. Treat the token as a secret until it expires.
  2. Compare iss. It must equal the configured issuer exactly, including scheme and trailing path. Entra notes that leading or trailing whitespace in the issuer blocks the exchange.
  3. Compare sub. Kubernetes service account tokens use system:serviceaccount:<namespace>:<name>. CI platforms encode repository, branch, environment or pipeline into the subject, and a renamed branch or environment changes it.
  4. Compare aud. Each cloud expects a specific audience: sts.amazonaws.com for IRSA, pods.eks.amazonaws.com for EKS Pod Identity, and one value (recommended api://AzureADTokenExchange) for Entra federated identity credentials. Google Cloud providers accept the audience configured on the provider.
  5. Check exp and iat. An expired or not-yet-valid token fails as ExpiredToken or InvalidIdentityToken on AWS. Check the node or runner clock if iat is in the future.
  6. Check the signing algorithm in the header. AWS STS accepts RSA (RS256, RS384, RS512) and ECDSA (ES256, ES384, ES512) signatures; Microsoft says only RS256 issuers have been tested.

AWS: AssumeRoleWithWebIdentity, IRSA and EKS Pod Identity

Error reference

ErrorHTTPMeaning (AWS STS API reference)Common real cause and fix
InvalidIdentityToken400The token could not be validatedOIDC provider not registered in IAM for this issuer, or issuer URL differs from the token's iss. Register the exact issuer as an IAM OIDC provider.
ExpiredToken400The web identity token is expired or not validCached token reused after expiry, or clock skew. Re-read the token file on each exchange.
IDPRejectedClaim403Authentication failed, possibly an invalid, expired or revoked claimUsually the role trust conditions on :sub or :aud do not match. Fix the trust condition to the exact value.
IDPCommunicationError400AWS could not reach the identity providerOften transient. Retry a limited number of times; check the issuer's public endpoints.
PackedPolicyTooLarge400Session policies and tags exceed the size limitToo many session tags or a large inline session policy. Trim them.
RegionDisabled403STS not activated in that region for the accountActivate the regional endpoint in IAM account settings or use an enabled region.
AccessDenied on the role403Not an STS error code in the table, returned when the trust policy does not allow the callerTrust policy principal or action wrong (sts:AssumeRoleWithWebIdentity for IRSA).

Also check that DurationSeconds is between 900 and 43,200 and does not exceed the role's maximum session duration, and that the role session name is 2 to 64 characters from [\w+=,.@-].

IRSA checks

  • Trust principal: arn:aws:iam::<account>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<ID>, the cluster issuer without https://.
  • Conditions: <provider>:aud equals sts.amazonaws.com and <provider>:sub equals system:serviceaccount:<namespace>:<name> under StringEquals. A typo in the namespace is the most common IRSA fault.
  • Annotation: the service account carries eks.amazonaws.com/role-arn; pods created before the annotation do not pick it up until restarted.

EKS Pod Identity checks

  • Agent: the EKS Pod Identity Agent runs on the node and serves credentials at http://169.254.170.23/v1/credentials, calling the EKS Auth AssumeRoleForPodIdentity action.
  • Injection: the pod should contain AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE, with a projected token of audience pods.eks.amazonaws.com.
  • Credential chain: static keys in environment variables or a mounted credentials file are found first. Remove them, then confirm the caller identity is the Pod Identity role session.

For trust policy design and cross-account patterns see the AWS STS and AssumeRole Guide and the EKS Identity Hardening Guide.

Google Cloud: Workload Identity Federation and GKE

Error reference

Error text (Google Cloud documentation)CauseFix
"Error connecting to the given credential's issuer." (invalid_grant)Google cannot fetch the issuer's discovery document or JWKSMake the endpoints publicly reachable and OIDC-compliant; confirm the token's iss
"Input JWK is not in a valid json format"Uploaded JWK contains x5c or x5tRemove those fields and reconfigure the provider
"The size of mapped attribute google.subject exceeds the 127 bytes limit." (invalid_request)Mapped subject too long (common with long CI subjects)Shorten with extract() in the mapping
"Org Policy violated for value: '{PROVIDER}'."constraints/iam.workloadIdentityPoolProviders does not allow that issuerAdd the provider to the allowlist through policy change control
"Request had invalid authentication credentials." (401)A federated STS token was sent to an API that does not accept federated tokensUse service account impersonation (GenerateAccessToken) for that API
"The given AWS request doesn't contain all the required headers." (invalid_grant)AWS-to-Google exchange with wrong signed headersInclude only the documented headers with an empty body
quota_exceeded, rate limit on sts.googleapis.com/requests (429)Too many token exchangesCache credentials until near expiry; request a quota increase if needed
"Permission 'iam.serviceAccounts.getAccessToken' denied" (403)Missing roles/iam.workloadIdentityUser binding, or propagation delayAdd the binding on the service account; allow minutes for propagation

Mapping and condition checks

  • Attribute mapping: google.subject is required for OIDC and SAML providers and must be unique; up to 50 custom attribute.NAME mappings are allowed.
  • Attribute conditions: a CEL expression that must return true. A rejection after a CI change usually means the condition references a claim value (repository, ref, workflow) that changed. Fix the workload or update the condition deliberately; do not delete it.
  • Principal identifiers: principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/VALUE uses the project number, not the project ID. Using the ID is a frequent silent mismatch.

Workload Identity Federation for GKE checks

  • Node pool: confirm the cluster has a workload pool and the node pool's workloadMetadataConfig.mode is the GKE metadata server mode.
  • Start-up race: the GKE metadata server needs a few seconds before it accepts requests from a new pod. Add retries or an init container; with Istio, use holdApplicationUntilProxyStarts or exclude 169.254.169.254/32 from redirection.
  • Linked service accounts: the IAM service account needs roles/iam.workloadIdentityUser for PROJECT_ID.svc.id.goog[NAMESPACE/KSA], and the Kubernetes service account needs the iam.gke.io/gcp-service-account annotation. If a binding vanished, the setIamPolicy audit log shows who removed it.
  • Scale: gke-metadata-server pods can be OOM-killed in clusters with more than 3,000 Kubernetes service accounts.

See the GKE Identity Hardening Guide for the hardening side.

Microsoft Entra: federated identity credentials and AKS

Error and limit reference

Error or limitMeaningFix
AADSTS70021 "No matching federated identity record found for presented assertion."No credential matches the token's issuer, subject and audience, or the credential has not propagated yetWait and retry; then diff the three values case-sensitively
AADSTS700212 (AKS)Wrong projected token used: identity binding tokens use audience api://AKSIdentityBinding, direct federation needs api://AzureADTokenExchangeMount and read the token file with the correct audience
AADSTS50013 InvalidAssertionAssertion invalid: wrong issuer, expired or malformedCheck exp, issuer and token format
"Federated identity credentials must have exactly one audience" (400)More than one audience configuredUse a single audience
"Issuer and subject combination already exists" (400)Duplicate issuer and subject on the same identityReuse the existing credential
409 ConflictConcurrent writes of credentials under one user-assigned identityCreate sequentially (ARM "mode": "serial" or dependsOn)
20 credentials per app or user-assigned managed identityHard limitConsolidate subjects or split identities by workload
No wildcards in any credential propertyExact match onlyOne credential per distinct subject

Entra stores only the first 100 signing keys from an external issuer's discovery endpoint, so an issuer publishing more may fail intermittently.

AKS workload identity checks

  • Pod label: azure.workload.identity/use: "true" must be on the pod template. Without it the webhook does not inject configuration and pods can fail after a restart.
  • Service account annotations: azure.workload.identity/client-id and optionally azure.workload.identity/tenant-id; restart pods after changing them.
  • Token file: read AZURE_FEDERATED_TOKEN_FILE at runtime and re-read it on each exchange, because Kubernetes refreshes it in place.
  • Scopes: request v2 scopes such as https://management.azure.com/.default; a raw resource URI can fail.

See the AKS Identity Hardening Guide.

Cross-cloud federation: when one cloud's identity calls another

  • AWS to Google Cloud: the AWS provider maps google.subject to the caller's ARN by default. Assumed-role ARNs include the session name, so use an attribute condition on the role rather than binding a single session ARN.
  • Kubernetes to another cloud: the cluster's issuer must be publicly discoverable for the remote cloud to validate it. Private clusters often need the issuer discovery documents published to a public endpoint.
  • CI to cloud: CI subjects change with branch, environment and event type. Pin trust to repository plus a protected branch or environment. See the GitHub Actions OIDC Federation Hardening Guide.
  • Chained exchanges: when a token from one exchange is used for the next, check each hop separately and keep the session name or subject traceable to the original workload.

Logging and detection while you troubleshoot

CloudWhere exchanges are loggedWhat to alert on
AWSCloudTrail AssumeRoleWithWebIdentity and Pod Identity session events, with role session name and session tagsExchanges from an unexpected issuer or subject; trust policy edits (UpdateAssumeRolePolicy)
Google CloudCloud Audit Logs for the Security Token Service and IAM (setIamPolicy), with google.subject in the principalNew pool providers, attribute condition changes, new workloadIdentityUser bindings
Microsoft EntraService principal and managed identity sign-in logs; audit logs for federated credential changesNew federated identity credentials, repeated AADSTS70021 from unknown issuers

Map detections to MITRE ATT&CK T1078.004 Cloud Accounts and T1550.001 Application Access Token. Metrics that show the federation estate is healthy:

  • Exchange failure rate per workload, with failures grouped by error code.
  • Mean time to resolve federation incidents.
  • Number of trust policies or credentials using wildcard or namespace-wide subjects (target: zero or documented exceptions).
  • Workloads still holding static cloud keys alongside a federation path.
  • Trust configuration changes made outside infrastructure as code.

Common failure patterns

  • Loosening the subject to end an outage: replacing an exact sub with system:serviceaccount:ns:* or a repository-wide wildcard lets every workload in scope assume the role. Fix the claim, and record any temporary widening as an exception with an expiry.
  • Reintroducing a static key "temporarily": the key stays for years, and on EKS it silently takes precedence over federation. Detect keys on workloads that also have a federation path.
  • Caching the projected token: applications read the token once at start and fail at expiry with ExpiredToken or AADSTS50013. Re-read the file for every exchange.
  • Project ID instead of project number: Google Cloud principal identifiers silently fail to match.
  • Treating propagation as a misconfiguration: engineers delete and recreate correct credentials during the propagation window, causing 409 conflicts or further delay. Add retries with backoff first.
  • Unbounded CI trust: trust bound to an organisation rather than a repository and protected branch lets any compromised workflow mint cloud tokens.

Lessons from incidents

In the Megalodon GitHub Actions attack, injected workflows harvested CI secrets from thousands of repositories, including long-lived AWS, Google Cloud and Azure credentials and GitHub Actions OIDC tokens. Federation removes the stored keys, but a compromised workflow can still request tokens. The trust conditions you debug are your security boundary: keep subject conditions tied to specific repositories, branches or environments, and alert on exchanges from runs you did not expect.

Framework mapping

FrameworkReferenceRelevance
NIST SP 800-53 Rev. 5IA-9 Service Identification and AuthenticationWorkloads authenticate with federated identity rather than shared secrets
NIST SP 800-53 Rev. 5IA-5 Authenticator ManagementRemoving static keys that federation replaces
NIST SP 800-53 Rev. 5AC-6 Least PrivilegeExact subject conditions instead of wildcards
NIST SP 800-53 Rev. 5AU-12 Audit Record GenerationLogging exchanges and trust changes
NIST CSF 2.0PR.AA-01, PR.AA-05Managing service credentials and enforcing access policy
MITRE ATT&CKT1078.004, T1550.001, T1528Abuse of cloud accounts and application access tokens

Practitioner checklist

  • Decode the presented token and diff iss, sub, aud, exp against the trust before changing anything.
  • Confirm the token file and injected environment variables exist in the running container.
  • Confirm the issuer discovery document and JWKS are reachable by the target cloud.
  • Allow for propagation: retry with backoff for several minutes after creating credentials or bindings.
  • Use exact subject conditions; document and expire any temporary widening.
  • Remove static keys from workloads once federation works, and verify the caller identity.
  • Re-read projected tokens on every exchange; never cache them across expiry.
  • Manage trust policies, pools and federated credentials in infrastructure as code, and alert on changes made outside it.
  • Track exchange failure rates by error code per workload.

Standards and references

Related NHI Mgmt Group resources: Cloud Workload Identity Guide · Workload Identity Federation Guide · SPIFFE and SPIRE Guide · OAuth Token Exchange and Delegation Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 28 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.