Most Kubernetes workloads still prove who they are with bearer tokens or long-lived certificates copied into Secrets, and anyone who reads the Secret becomes the workload. Kubernetes 1.37, released on 26 August 2026, made two APIs stable and on by default that change this: PodCertificateRequest with the podCertificate projected volume, which has the kubelet generate a private key on the node and obtain a short-lived certificate for each pod, and ClusterTrustBundle, which distributes CA certificates to pods as an automatically updated file. This guide explains how the two work, how they compare with cert-manager, SPIRE and service account tokens, how to build and govern a signer, and how to roll them out and monitor them. It is for platform engineers, PKI owners and security architects running Kubernetes.
Key takeaways
- The private key never leaves the node. The kubelet generates it, keeps it in memory and a tmpfs projected volume, and sends only a public key to the API server, so issuance involves no bearer token when kubelet and signer use their own client certificates.
- Kubernetes provides the plumbing, not the CA. There is no built-in signer for pod certificates; you deploy a signer controller (your own, a mesh, or an external CA bridge) and it decides what to issue.
- Short lifetimes are the default. Certificates default to 24 hours, the accepted range is 1 hour to 91 days, and the kubelet refreshes them automatically; your application must reload the file when it changes.
- ClusterTrustBundles are public inside the cluster. Every service account can read them by default, so they are for trust anchors only, and writing a signer-linked bundle requires the attest verb on that signer.
- The signer is your new tier-zero component. Whoever holds sign on a signer name can mint identities for any pod that requests from it; treat that RBAC like cluster-admin.
What the two APIs do
Both live in the certificates.k8s.io API group alongside the older CertificateSigningRequest (CSR) API, which has been stable since 1.19 but leaves delivery to workloads "as an exercise for the reader", in the words of the Kubernetes enhancement proposal KEP-4317.
PodCertificateRequest and the podCertificate volume
A pod author adds a podCertificate source to a projected volume. For each such volume the kubelet generates a key pair, creates a PodCertificateRequest (PCR) naming the pod, its UID, its service account and UID, its node and UID, the signer, and a stub PKCS#10 request containing only the public key, then waits for the signer to respond. The API server, through the NodeRestriction admission plugin, only accepts a PCR that corresponds to a real pod currently running on that node, so a compromised node cannot request certificates for pods elsewhere.
- No approval step: unlike CSRs, a PCR has no separate approval phase. The signer controller sets exactly one condition: Issued, Denied or Failed. Once set, status is immutable, and the spec is immutable from creation.
- Issuance output: the signer writes
status.certificateChain, copies notBefore and notAfter into status, and suggestsstatus.beginRefreshAt, which tells the kubelet when to start renewing. - Cleanup: kube-controller-manager deletes PCRs older than 15 minutes, so every issuance flow must finish inside that window.
- Annotations are untrusted:
userAnnotationsin the pod spec are copied intospec.unverifiedUserAnnotations. The documentation is explicit that signers must verify this data and should deny keys they do not recognise.
ClusterTrustBundle and its projection
A ClusterTrustBundle (CTB) is a cluster-scoped object holding one or more PEM CERTIFICATE blocks that the API server validates as X.509. A clusterTrustBundle projected volume source writes the selected bundles into a file, deduplicated and normalised, and keeps it current as bundles change. This makes CA rotation a matter of updating an object rather than rebuilding images or restarting pods.
| CTB mode | How it is named | Who can write it | Typical use |
|---|---|---|---|
| Signer-linked | spec.signerName set; name must start with the signer name with slashes replaced by colons, for example example.com:mysigner:2026-11 | Only principals allowed the custom attest verb on that signer in certificates.k8s.io | Trust anchors for a signer that issues pod certificates; pods select by signer name plus label selector |
| Signer-unlinked | No signerName; name must not contain a colon | Whoever RBAC allows to create or update ClusterTrustBundles | General cluster configuration, such as a corporate root CA for outbound TLS |
The kube-controller-manager also uses a CTB to publish the CA bundle for the built-in kubernetes.io/kube-apiserver-serving signer, which gives workloads a clean way to trust the API server.
Feature status and version support
| Feature gate | Alpha | Beta (off by default) | Stable, on by default |
|---|---|---|---|
| ClusterTrustBundle | 1.27 to 1.32 | 1.33 to 1.36 | 1.37 |
| ClusterTrustBundleProjection | 1.29 to 1.32 | 1.33 to 1.36 | 1.37 |
| PodCertificateRequest | 1.34 | 1.35 to 1.36 | 1.37 |
On 1.33 to 1.36 the features exist but must be switched on in the API server, controller manager and kubelet, and managed services may not let you set the gates. Before planning, confirm your provider's 1.37 availability and whether it exposes the certificates.k8s.io v1 resources for these kinds. Kubernetes 1.36 reaches end of life on 28 June 2027 according to the project schedule.
Choosing a workload identity mechanism
| Mechanism | Credential | Key location | Rotation | Best fit |
|---|---|---|---|---|
| Projected service account token | Bound JWT (bearer) | No key; token in tmpfs | Kubelet refreshes | Calling the API server and cloud workload identity federation |
| podCertificate volume | X.509 certificate plus private key | Generated by kubelet on the node | Kubelet refreshes from beginRefreshAt | mTLS between services, external services that accept client certificates, SPIFFE-style IDs from a compatible signer |
| cert-manager Certificate | X.509 in a Kubernetes Secret | Generated by cert-manager, stored in etcd | Controller renews and updates the Secret | Ingress and server certificates, especially public ACME; anything that must survive pod restarts |
| SPIRE agent (Workload API) | X.509-SVID or JWT-SVID | Agent on node, delivered over a socket | Agent rotates | Multi-platform identity across Kubernetes, VMs and clouds with federation |
| Static Secret mounted into pod | Certificate and key or API key | etcd and every reader of the Secret | Manual | Nothing new: migrate away |
- Choose podCertificate when: you want per-pod client or server certificates with keys that never reach etcd, and your services can reload a file. It avoids a node agent and a socket.
- Keep cert-manager when: the certificate belongs to a Service or Ingress rather than a pod, needs a public CA, or must be shared. See the cert-manager security guide.
- Keep SPIRE when: you need identity across non-Kubernetes platforms, attestation beyond pod and node, or trust domain federation. A signer can issue certificates carrying a SPIFFE ID as a URI SAN, but federation and the Workload API are SPIRE's job. See the SPIFFE and SPIRE Guide.
- Small clusters: one signer backed by a dedicated intermediate CA is enough. Large estates: one signer name per trust purpose (mesh, database client, egress) so RBAC and blast radius stay separate.
Building and governing a signer
Because Kubernetes ships no pod certificate signer, the security of the whole scheme rests on the controller you deploy. It needs update on podcertificaterequests/status and sign on the resource signers with the resource name <domain>/<path>; to publish its CA it also needs attest on the same signer.
- Name signers under your own domain: for example
certs.example.com/mesh. Never reusekubernetes.io/names. - Derive identity from verified fields only: service account name and UID, namespace and pod UID in the PCR are accurate because the API server enforces them. Build the subject or SAN from those, and treat
unverifiedUserAnnotationsas input to check, never as identity. - Write an issuance policy: which namespaces and service accounts may obtain certificates from this signer, which key types (prefer ECDSAP256 or ED25519 where clients support it), maximum lifetime, and which key usages. Deny anything else with a clear reason code.
- Keep the CA key out of the cluster where possible: the KEP's support for PKCS#10 stubs is there so external CAs, such as HashiCorp Vault, can issue. If the CA key must live in-cluster, isolate the signer namespace, pin it to dedicated nodes and back the key with a KMS.
- Know the limits: keys are held in kubelet memory and tmpfs, not an HSM; this is a non-goal of the design. If kubelet restarts, keys are lost and every pod on the node requests new certificates at once, so size the signer for bursts.
- Rotate the CA with overlap: publish the new root in a signer-linked CTB, wait until every consumer has it, switch issuance, then remove the old root after the longest certificate lifetime has passed. See Certificate trust stores and root CA management.
Where identity fits
A pod certificate is a non-human identity credential, and the controls that apply to service accounts now apply to the certificate that represents them. The service account becomes the identity anchor: a pod running as payments/api gets a certificate that says so, so namespace and service account sprawl translate directly into certificate sprawl. Inventory which service accounts can obtain certificates from each signer, retire unused ones, and remember that whoever can create pods under a service account can obtain its certificate. See the Kubernetes Service Account Token Guide and the Kubernetes NHI Security Guide.
How to implement
- Inventory current credentials: list Secrets of type kubernetes.io/tls and Opaque Secrets holding PEM keys, mounted service account tokens used for service-to-service calls, and which workloads read them. These are your migration candidates.
- Confirm platform support: run 1.37 or enable the three gates on a test cluster; check that NodeRestriction is enabled on the API server, because node scoping of PCRs depends on it.
- Stand up the signer in a test cluster: issue with a dedicated intermediate, publish its root in a signer-linked CTB, and apply the issuance policy in deny-by-default form.
- Pilot one service pair: mount a podCertificate volume using
credentialBundlePath(a single file avoids reading a key and certificate from different rotations) and a clusterTrustBundle volume selected by signer name, then turn on mTLS between the two services. - Test rotation and failure: set the lifetime to one hour to force several renewals, restart the kubelet, stop the signer, and confirm pods stay healthy until expiry and alert as it approaches. Pods with a required CTB that cannot be found will not start; set
optional: trueonly where an empty trust file is safe. - Roll out by namespace: expand with admission policy that requires the podCertificate volume for selected namespaces. See the Kubernetes Admission Control Guide.
- Retire static Secrets: delete the old TLS Secrets once traffic uses the new certificates, and revoke or let expire the certificates they held.
- Roll back cleanly: keep the old Secret mounts in a disabled manifest version for one release; rollback is a deployment change because the two credential sources are independent.
Detection and measurement
Kubernetes audit logs record every create and status update on podcertificaterequests and every write to clustertrustbundles. See the Kubernetes Audit Log Detection Guide.
| Signal | Why it matters |
|---|---|
| create or update on clustertrustbundles by anyone other than the signer controller | Adding a root to a trust bundle lets the holder of that root impersonate any service that trusts it |
| New RoleBinding or ClusterRoleBinding granting sign or attest on signers | Creates a second party able to mint identities or trust anchors |
| update on podcertificaterequests/status from an unexpected principal | Something other than the signer is issuing |
| Spike in PCR creation from one node | Kubelet crash loop, or a node creating pods to harvest certificates |
| Denied or Failed conditions by reason | Policy violations or a broken signer |
Metrics to track:
- Share of service-to-service connections authenticated with pod certificates rather than static Secrets or bearer tokens.
- Number of TLS Secrets remaining in migrated namespaces (target zero).
- Median issued certificate lifetime (target 24 hours or less).
- Issuance latency at the 95th percentile, well inside the 15-minute PCR window.
- Principals holding sign or attest per signer (target one each).
Common failure patterns
- Apps that never reload: the kubelet rotates the file but the process keeps the old certificate in memory and fails at expiry. Test with one-hour lifetimes before production.
- Trusting user annotations: a signer that puts
unverifiedUserAnnotationsinto the SAN lets any pod author choose its own identity. - Over-broad attest or sign grants: a wildcard
example.com/*grant to a CI service account lets a pipeline compromise mint identities and trust anchors cluster-wide. - Putting secrets in a CTB: bundles are world-readable to every service account by default; they are for public certificates only.
- Leaving the old Secrets in place: the migration only helps if the static credentials go. Exposed Kubernetes Secrets have leaked high-value keys before, as in the SAP artifact repository key exposure, and a long-lived stolen certificate was enough for the Mimecast certificate compromise.
- Signer as single point of failure: with 24-hour certificates, a signer down for a day becomes a cluster-wide outage. Run it highly available and alert well before expiry.
Framework mapping
| Framework | Reference | How pod certificates and CTBs help |
|---|---|---|
| NIST SP 800-53 Rev 5 | IA-5(2) Public Key-based Authentication; IA-9 Service Identification and Authentication; SC-12 Cryptographic Key Establishment and Management; SC-17 Public Key Infrastructure Certificates; SC-8 Transmission Confidentiality and Integrity; AC-6 Least Privilege; AU-12 Audit Record Generation | Per-pod key generation, short-lived certificates from a governed CA, mTLS between services, least-privilege signer RBAC, audited issuance |
| NIST CSF 2.0 | PR.AA-01, PR.AA-03, PR.DS-02 | Service identities managed and authenticated; data in transit protected |
| CIS Controls v8.1 | 3.10 Encrypt Sensitive Data in Transit | mTLS with automatically rotated certificates |
| ISO/IEC 27001:2022 Annex A | A.5.17 Authentication information; A.8.24 Use of cryptography | Credential lifecycle and key management for workloads |
| MITRE ATT&CK v20 | T1552.004 Private Keys; T1552.007 Container API; T1528 Steal Application Access Token; T1649 Steal or Forge Authentication Certificates; T1557 Adversary-in-the-Middle | Keys stay on the node and expire within hours; fewer bearer tokens to steal; forged certificates need the signer, which is monitored |
Practitioner checklist
- Confirm cluster version 1.37, or the three feature gates and NodeRestriction on earlier versions.
- Name each signer under your own domain, one per trust purpose.
- Grant sign, attest and podcertificaterequests/status update to the signer controller only, and alert on any new grant.
- Build certificate identity from service account, namespace and pod UID; never from unverified annotations.
- Set maximum lifetime to 24 hours or less and test reload with one-hour certificates.
- Use credentialBundlePath so key and chain are read atomically.
- Publish roots in signer-linked ClusterTrustBundles and rotate CAs with overlap.
- Monitor CTB writes, PCR volume per node and Denied reasons in audit logs.
- Delete static TLS Secrets after migration and track the count to zero.
Standards and references
- Kubernetes: Certificates and Certificate Signing Requests (PodCertificateRequests and ClusterTrustBundles)
- Kubernetes: Projected Volumes (podCertificate and clusterTrustBundle sources)
- Kubernetes Enhancements: KEP-4317 Pod Certificates
- Kubernetes Enhancements: KEP-3257 Cluster Trust Bundles
- Kubernetes: Feature Gates
- IETF: RFC 2986, PKCS #10 Certification Request Syntax
- IETF: RFC 5280, Internet X.509 PKI Certificate and CRL Profile
Related NHI Mgmt Group resources: SPIFFE and SPIRE Guide · Kubernetes Service Account Token Guide · cert-manager security guide · Kubernetes Workload Identity Checklist · Kubernetes Secrets Security Guide