How to move GKE workloads from keys to Workload Identity Federation: remove key Secrets without downtime

A service account key mounted into a GKE Pod as a Kubernetes Secret is a private key that, by default, never expires and works from anywhere, so anyone who can read that Secret, the image or the node holds the service account indefinitely. Workload Identity Federation for GKE replaces the key with a short-lived token that the GKE metadata server obtains for the Pod's own Kubernetes ServiceAccount. The migration has traps that the generic key replacement process does not cover: switching a node pool to the GKE metadata server cuts running Pods off from the node's service account, and Application Default Credentials keep using the key file for as long as GOOGLE_APPLICATION_CREDENTIALS is set. This guide covers the GKE-specific cutover, cluster by cluster and workload by workload. It is for platform engineers and cloud security teams running GKE Standard or Autopilot. The project-wide key inventory is in How to inventory Google Cloud service account keys and replace them with federation, and the grant models are explained in the GKE Identity Hardening Guide.

Key takeaways

• The key wins until you remove it. ADC checks GOOGLE_APPLICATION_CREDENTIALS before the metadata server, so enabling federation changes nothing for a Pod that still points at a mounted key file.
• Node pool changes are the risky step. Updating a Standard node pool to --workload-metadata=GKE_METADATA takes effect immediately for running workloads, which can lose access to the node service account; a new node pool is the safer path.
• Disable the key before deleting it. Disabling is reversible; it is your rollback for the whole migration.
• Identity is namespace plus ServiceAccount name, per project. Every cluster sharing the project's workload identity pool can claim the same identity, so the migration is also a chance to separate clusters of different trust.
• Close the door afterwards. The iam.disableServiceAccountKeyCreation constraint stops new keys, and it is already enforced by default for organisations created on or after 3 May 2024.

When you need this

• Key files in Secrets or images: workloads read a JSON key from a mounted Secret, a ConfigMap, a CSI volume or a file baked into the image. Keys in images are the worst case: in the Codecov breach an error in the Docker image creation process let the attacker extract a Google Cloud Storage key and use it to alter a script that harvested CI secrets for about two months.
• Workloads using the node service account: Pods inherit the node's (often Compute Engine default) service account, which Google notes can over-provision permissions and is inappropriate for multi-tenant clusters.
• Key rotation pain: you are rotating keys by hand across namespaces, or audit findings flag keys older than your policy allows.
• Clusters adopted from another team where nobody knows which Secrets hold keys.

Before you start

Key, node service account and federation compared

Step-by-step

Steps 1 to 3 are read-only. Steps 4 to 7 are additive or reversible. Steps 8 and 9 are destructive.

1. Find key Secrets and their consumers. Service account key files are JSON with "type": "service_account". Search Secrets for that content and list the Pods that mount or reference them, and every container with a GOOGLE_APPLICATION_CREDENTIALS environment variable:

   kubectl get secrets -A -o json | jq -r '.items[] | . as $s | (.data // {}) | to_entries[]
     | select((.value | @base64d | test("\"type\": *\"service_account\"")))
     | [$s.metadata.namespace, $s.metadata.name, .key] | @tsv'
   kubectl get pods -A -o json | jq -r '.items[] | .metadata.namespace as $n | .metadata.name as $p
     | .spec.containers[] | select(any(.env[]?; .name=="GOOGLE_APPLICATION_CREDENTIALS")) | "\($n)\t\($p)\t\(.name)"'

   Also check images and Helm values for embedded key files; secret scanning on registries finds those.
2. Map each key to its service account and usage. Decode the client_email and private_key_id fields (do not print the private key) and match them to gcloud iam service-accounts keys list --iam-account=SA_EMAIL. Use the key usage evidence described in the live key runbook to confirm the key is used only by this workload and not also by a laptop or another system.
3. Decide the grant model per workload. Grant IAM roles directly to the Kubernetes ServiceAccount principal (the default for new work), or link the ServiceAccount to the existing IAM service account where a Google Cloud API does not accept federated tokens or you want to keep the service account's existing grants unchanged during the move. Linking is often the quickest migration path because the permissions stay where they are.
4. Enable federation on the cluster (Standard only).

   gcloud container clusters update CLUSTER_NAME \
     --location=LOCATION --workload-pool=PROJECT_ID.svc.id.goog

   Existing node pools are not affected by this command; node pools created afterwards use the GKE metadata server by default.
5. Move workloads onto GKE metadata server nodes. The safer route is a new node pool created with --workload-metadata=GKE_METADATA, then moving workloads across with cordon and drain as usual. Updating an existing pool in place works but enables the GKE metadata server immediately, so any Pod on that pool relying on the node service account loses it. Pods relying on the node service account need a grant of their own before they move.
6. Grant access to the Kubernetes ServiceAccount (additive). Direct grant:

   gcloud projects add-iam-policy-binding projects/PROJECT_ID --role=ROLE \
     --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME \
     --condition=None

   Or link to the existing IAM service account, which needs both the binding and the annotation:

   gcloud iam service-accounts add-iam-policy-binding SA_NAME@SA_PROJECT.iam.gserviceaccount.com \
     --role=roles/iam.workloadIdentityUser \
     --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
   kubectl annotate serviceaccount KSA_NAME -n NAMESPACE \
     iam.gke.io/gcp-service-account=SA_NAME@SA_PROJECT.iam.gserviceaccount.com

   Prefer resource-level grants (a bucket, a topic) over project-level roles.
7. Cut the workload over. Set serviceAccountName to the Kubernetes ServiceAccount, remove the GOOGLE_APPLICATION_CREDENTIALS variable and the Secret volume mount, and roll out in a pilot namespace. On Standard clusters with mixed node pools, add the nodeSelector iam.gke.io/gke-metadata-server-enabled: "true" so the Pod lands on federated nodes (Autopilot rejects this selector). Inside a new Pod, confirm that a call to the target API succeeds without the key, for example Google's test of listing objects in a bucket the ServiceAccount has been granted.
8. Disable the key (reversible). After the workload has run on federation for its full cycle:

   gcloud iam service-accounts keys disable KEY_ID \
     --iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com --project=PROJECT_ID

   Disabling does not revoke short-lived credentials already issued from the key, and disabled keys cannot be toggled in the console, only with gcloud or the API.
9. Delete the key and the Secret (destructive). When nothing has failed during the disabled period, delete the key with gcloud iam service-accounts keys delete KEY_ID --iam-account=SA_EMAIL, then delete the Kubernetes Secret and any copies in CI variables, Helm values or vaults.

Decision guidance

Small estates on Autopilot can skip the cluster and node pool steps and migrate workload by workload. Large Standard estates should plan node pool replacement as its own change, watch the per-node limits (500 concurrent metadata server connections; memory pressure above about 3,000 Kubernetes ServiceAccounts) and the token exchange quota of 6,000 requests per minute.

Verify it worked

• No key paths left: re-run the step 1 queries; the Secret and the environment variable are gone from the workload.
• API access works: the workload's own operations succeed from a fresh Pod, including after a restart (to cover the metadata server's start-up window).
• Key unused: the key list for the service account no longer shows the key; before deletion, the disabled key produced no failures.
• Negative test: a Pod using a different ServiceAccount in the same namespace cannot reach the resource.

Rollback and common errors

• Workload fails after cutover: while the key is only disabled, re-enable it with gcloud iam service-accounts keys enable and restore the Secret mount. After deletion there is no rollback; create a new key only as a recorded emergency measure.
• Still using the key: GOOGLE_APPLICATION_CREDENTIALS is still set, often by a base image or Helm chart default.
• Authentication fails right after Pod start: the GKE metadata server takes a few seconds to accept requests on a new Pod; add retries or a start-up check.
• Permission denied with the link model: the annotation or the roles/iam.workloadIdentityUser binding is missing; both are required.
• Timeouts to the metadata server: a network policy blocks 169.254.169.252:988 (or 169.254.169.254:80 with Dataplane V2), or the per-node connection limit is producing HTTP 499 responses.
• Other Pods broke during the node pool update: they relied on the node service account. Give them their own grants, or move them back to a pool with --workload-metadata=GCE_METADATA temporarily.

Common failure patterns

• Enabling federation and declaring victory: the cluster has a workload pool but every Pod still reads its key file.
• Keys left enabled "for DR": the disaster recovery runbook keeps the key, which remains the easiest credential to steal.
• Namespace-wide principal sets for convenience: granting principalSet for a whole namespace gives every Pod there the access, and identity sameness extends it to other clusters in the project.
• Over-privileged node service account left in place: hostNetwork Pods and anything that escapes to the node still get its permissions.
• No organisation policy afterwards: a developer creates a new key the following week. Enforce iam.disableServiceAccountKeyCreation and iam.disableServiceAccountKeyUpload. See the Google Cloud Organization Policy Guide.

Detection and measurement

• Kubernetes Secrets containing service account keys, per cluster (target zero).
• Containers with GOOGLE_APPLICATION_CREDENTIALS set (target zero outside approved exceptions).
• User-managed keys on service accounts used by GKE workloads, and their age.
• Standard node pools still on GCE_METADATA (target zero, or documented).
• Key creation attempts denied by organisation policy, as a signal of teams still depending on keys. Audit logging is covered in the Google Cloud Audit Logs and Log Sinks Guide.

Framework mapping

Practitioner checklist

• Find every Secret holding a service account key and every container setting GOOGLE_APPLICATION_CREDENTIALS.
• Map each key to its service account and confirm no other consumer uses it.
• Enable the workload pool on Standard clusters and move workloads to GKE_METADATA node pools, preferably new ones.
• Allow metadata server egress in network policies before cutover.
• Grant roles to the Kubernetes ServiceAccount principal, or link it with both the binding and the annotation.
• Remove the key mount and environment variable, pilot first, and add start-up retries.
• Disable the key for a full cycle, then delete it and every copy of the Secret.
• Enforce organisation policy to block key creation and upload.

Standards and references

• Google Cloud: Authenticate to Google Cloud APIs from GKE workloads
• Google Cloud: About Workload Identity Federation for GKE
• Google Cloud: How Application Default Credentials works
• Google Cloud: Disable and enable service account keys
• Google Cloud: Service account credentials
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls
• NIST: Cybersecurity Framework 2.0

Related NHI Mgmt Group resources: How to inventory Google Cloud service account keys and replace them with federation · GKE Identity Hardening Guide · Google Cloud Service Account Keys Security Guide · Workload Identity Federation Guide