How to rotate a Kubernetes service account token and detect old token use: Rotation by token type, with audit checks

"Rotate the service account token" means three different operations in Kubernetes, and choosing the wrong one either leaves the exposed token valid or takes down every workload using the account. Secret-based tokens never expire until the Secret is deleted, short-lived TokenRequest tokens cannot be revoked one by one, and pod-bound projected tokens die with their Pod. This runbook gives the rotation procedure for each token type, the signing key rotation that invalidates every token at once, and the audit log, label and metric checks that show whether an old token is still being used. It is for cluster administrators and platform engineers on Kubernetes v1.29 or later, where legacy token tracking and clean-up are available. For how the token types work and how to design them out, see the Kubernetes Service Account Token Guide.

Key takeaways

• Identify the token type first. A Secret of type kubernetes.io/service-account-token, a token from kubectl create token, and a projected pod token are rotated in different ways.
• Secret-based tokens are rotated by deleting the Secret. kubectl delete secret invalidates that token; create a new one only if a consumer truly needs a long-lived token.
• Short-lived tokens have no individual revocation. Kubernetes documents that you must delete and recreate the ServiceAccount (new UID) or, for pod-bound tokens, delete the Pod.
• Signing key rotation is the cluster-wide reset. Keep the old public key in --service-account-key-file while new tokens are signed with the new key, then remove it.
• Prove the old token is dead. Use the kubernetes.io/legacy-token-last-used label, the authentication.k8s.io/legacy-token-invalidated audit annotation and the credential ID (JTI) in audit events.

When you need this

The Kubeflow cryptomining attacks show why the pod case matters: attackers ran code in notebook servers and used the mounted service account to deploy mining containers. A confirmed cluster compromise needs more than token rotation; use the Kubernetes Cluster Compromise Playbook.

Before you start

• Audit logging is on: the API server only logs events when --audit-policy-file is set, with a backend such as --audit-log-path. A policy at the Metadata level records the requesting user, timestamp, resource and verb, which is enough for this runbook. Managed services expose control plane audit logs through their own logging settings.
• Inventory the account: record the ServiceAccount UID, RoleBindings and ClusterRoleBindings, Pods using it and any Secrets referencing it, so a recreate is exact.

  kubectl -n <ns> get serviceaccount <sa> -o yaml
  kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
    -L kubernetes.io/legacy-token-last-used,kubernetes.io/legacy-token-invalid-since
  kubectl get rolebindings,clusterrolebindings -A -o wide | grep <sa>

• Know your consumers: in-cluster Pods reload projected tokens automatically if the application rereads the file; external consumers (CI, monitoring, operators outside the cluster) need a new credential delivered.
• Change window: recreating a ServiceAccount or rotating the signing key affects every workload using it. Plan a rolling restart.

Step-by-step

Step 1 is read-only. Steps 2 to 4 are destructive for the targeted token or account. Step 5 is a control plane change.

1. Decode the token to identify it (read-only). A projected or TokenRequest token is a JWT with exp, iat, iss, jti and a kubernetes.io claim naming the namespace, ServiceAccount and, when bound, the Pod and Node. A legacy Secret-based token has no expiry. You can also submit it in a TokenReview; the response's user.extra carries authentication.kubernetes.io/credential-id (for example JTI=...) and the bound pod and node names. Handle the token as a live secret while you do this.
2. Rotate a Secret-based token. Find the Secret, then delete it.

   kubectl -n <ns> get serviceaccount/<sa> -o yaml     # secrets: lists auto-generated token Secrets
   kubectl -n <ns> delete secret/<token-secret-name>

   If the consumer cannot yet move to short-lived tokens, create a new manual token Secret with the kubernetes.io/service-account.name annotation and deliver it through your secret store. Do not reference manually created Secrets in the ServiceAccount's secrets field, or the cleaner will treat them as auto-generated and remove them when unused.
3. Rotate short-lived tokens for an external consumer. Issue a new token with a bounded lifetime and move the consumer to it:

   kubectl -n <ns> create token <sa> --duration=1h
   kubectl -n <ns> create token <sa> --bound-object-kind Node --bound-object-name <node> --bound-object-uid <uid>

   The issued duration can differ from the request, capped by --service-account-max-token-expiration. To kill an exposed token before it expires, delete and recreate the ServiceAccount; its new UID invalidates all tokens created for it, so re-apply its bindings from your inventory and restart its Pods.
4. Rotate pod-bound tokens. Deleting a Pod expires its bound tokens. For a Deployment, a rollout replaces every Pod:

   kubectl -n <ns> rollout restart deployment/<name>

   Routine rotation needs no action: the kubelet requests a new token when the current one is older than 80 per cent of its TTL or older than 24 hours, and the application must reread the file.
5. Rotate the signing key (cluster-wide). Generate a new key pair. Add the new public key alongside the old one in --service-account-key-file (the flag accepts multiple keys and can be repeated), point --service-account-signing-key-file on the API server (and --service-account-private-key-file on kube-controller-manager if legacy Secrets are still issued) at the new private key, and restart the control plane components. New tokens are signed with the new key while old ones still verify. After every Pod has restarted and the longest token lifetime has passed, remove the old public key and restart again; any token signed by the old key now fails. If you also change --service-account-issuer, list both issuers during the transition, because the first is used to sign and all are accepted. Managed Kubernetes services control these flags themselves, so use the provider's documented credential rotation instead.

Verify it worked

Search the audit log for the ServiceAccount username and the old credential ID after each step. Successful requests with the old JTI after rotation mean the step did not take effect; failed requests from an unexpected source address mean someone else holds the old token and the incident is not over.

Rollback and common errors

• A consumer broke after legacy invalidation: Kubernetes documents that removing the kubernetes.io/legacy-token-invalid-since label temporarily allows the token again. Use this only while you move the consumer.
• Deleted Secret was still needed: deletion cannot be undone. Create a new token for the consumer; the old value will never work again.
• Recreated ServiceAccount lost permissions: bindings reference the ServiceAccount by name, so they usually still apply, but anything keyed on the old UID will not. Re-apply from the inventory and restart Pods so they get tokens for the new UID.
• Application fails an hour after deploy: it reads the projected token once at start-up. Fix the client to reread the file, or use a current client library.
• Signing key removed too early: Pods with tokens signed by the old key get 401 errors. Put the old public key back in --service-account-key-file, restart the API server, then restart workloads before trying again.
• Tokens appear to live a year: with --service-account-extend-token-expiration (default true), admission-injected tokens can be extended up to a year to ease migration. Account for this when deciding when an old key is safe to remove.

Hardening afterwards

• Remove long-lived tokens: Kubernetes recommends TokenRequest short-lived tokens over manually created Secrets. External systems should use workload identity federation where they support it. See the Workload Identity Federation Guide.
• Stop automounting where unused: set automountServiceAccountToken: false on ServiceAccounts or Pods that never call the API.
• Cap lifetimes: set --service-account-max-token-expiration to a value your consumers can live with, and plan to turn off expiration extension once migration is complete.
• Alert on token creation: watch audit events for create on serviceaccounts/token and on Secrets of type kubernetes.io/service-account-token. See the Kubernetes Audit Log Detection Guide.
• Least privilege on the account: a rotated token with cluster-admin is still cluster-admin. See the Kubernetes RBAC Guide.

Single cluster or fleet

On one self-managed cluster, an exposed Secret-based token is a five minute fix and a signing key rotation is a planned evening with two control plane restarts. Across a fleet, the expensive part is knowing which clusters hold Secret-based tokens and which external systems consume them. Collect the token Secret inventory and the legacy token labels from every cluster centrally, give each external consumer an owner, and practise the ServiceAccount recreate on a non-production cluster so the bindings re-apply cleanly. On managed services, test the provider's credential rotation in a staging cluster before you need it in an incident, because it usually restarts or replaces nodes.

Metrics

• Secret-based service account tokens in the cluster, auto-generated and manual (target: zero, with documented exceptions).
• invalid_legacy_auto_token_uses_total rate (target: zero after migration).
• Maximum token lifetime allowed by --service-account-max-token-expiration.
• Time from token exposure report to invalidation (target: minutes for Secret-based tokens).

Framework mapping

Practitioner checklist

• Confirm API server audit logging is enabled at Metadata level or above.
• Decode the exposed token to find its type, ServiceAccount, JTI and binding.
• Delete Secret-based tokens; never reference manual tokens in the ServiceAccount secrets field.
• Recreate the ServiceAccount to kill an unexpired TokenRequest token, then re-apply bindings.
• Delete or roll Pods to expire pod-bound tokens.
• Rotate the signing key with an overlap period when the key or control plane is suspect.
• Search the audit log for the old JTI and 401s from the ServiceAccount afterwards.

Standards and references

• Kubernetes: Managing Service Accounts
• Kubernetes: Configure Service Accounts for Pods
• Kubernetes: Service Accounts
• Kubernetes: kube-apiserver command line reference
• Kubernetes: Auditing
• Kubernetes: Authenticating

Related NHI Mgmt Group resources: Kubernetes Service Account Token Guide · Kubernetes Audit Log Detection Guide · Kubernetes NHI Security Guide · Kubernetes Cluster Compromise Playbook