If your service mesh is compromised: containment, credential rotation and rebuild playbook

A service mesh is the identity and traffic authority for every workload it fronts, so an attacker who controls its control plane, its certificate authority or its proxy injection can impersonate any service, read traffic in clear text and rewrite routing without touching application code. This playbook covers how a mesh compromise shows up, the triage questions for the first hours, containment in priority order, rotation of the mesh trust chain and workload credentials, the decision to rebuild, the evidence to keep and when notification rules apply. It is vendor-neutral and written for platform engineering, cloud security and incident response teams running a sidecar or sidecarless mesh on a container platform. Prevention and hardening are covered in the Service Mesh Security Guide and Securing the service mesh itself; this page is about what to do once you believe the mesh itself is in the attacker's hands.

Key takeaways

• The mesh CA is the crown jewel. If the signing key that issues workload certificates was readable, every identity in the trust domain is forgeable until you replace the root or intermediate and every workload holds a new certificate.
• Short lifetimes do not revoke anything. SPIFFE notes that changing or revoking an assertion on a live system waits for existing SVIDs to expire, so trust bundle replacement, not waiting, is the containment step for a stolen signing key.
• Policy is persistence. A permissive authentication mode, a wildcard authorization rule, a new egress route or a changed injection webhook keeps the door open after credentials rotate, so diff all mesh configuration against source control.
• Freeze before you rotate. Export control plane audit records, mesh configuration and proxy access logs first; a rebuild or root rotation destroys the state you need to scope the incident.
• Rotate in a planned order. Close administrative access to the control plane, then replace the trust chain, then restart workloads, then rotate application secrets that crossed the mesh in a form the attacker could read.

How the incident shows up

Mesh compromises are rarely announced by a single alert. The signals are usually unexplained configuration changes in the cluster audit log, certificates you did not expect, or traffic that takes a path the routing table should not allow. Most begin with a stolen platform credential (MITRE ATT&CK T1078.004 Cloud Accounts), a container escape on a node (T1611 Escape to Host) or a compromised CI pipeline that deploys mesh configuration.

Collection and alerting for these sources are covered in the Kubernetes Audit Log Detection Guide. If the platform itself is compromised beyond the mesh, run the Kubernetes Cluster Compromise Playbook in parallel.

Triage questions

1. Which plane is affected? Control plane (configuration and policy distribution), identity plane (the CA and its keys), data plane (proxies on nodes or beside pods) or the gateways at the edge. Each needs a different response, and a control plane compromise usually implies the others.
2. Could the attacker read the CA signing key? Where is it stored: a platform secret, an external KMS or HSM, or an upstream enterprise CA? Who and what could read it during the window? A yes or "unknown" moves you straight to trust chain replacement.
3. What is the trust domain blast radius? List every cluster and federated trust domain that accepts the compromised bundle. SPIFFE's trust domain specification warns that a compromised peer trust domain can assert anything about its callers, so federation partners are in scope.
4. Which identities could administer the mesh? Platform role bindings that can create or modify mesh policy, webhooks, gateways and the CA secret, including CI service accounts and operators.
5. What crossed the mesh in a readable form? If the attacker held proxy keys or controlled routing, tokens, API keys and session cookies in headers may have been captured. List which services pass bearer credentials between them.
6. Is mesh audit and telemetry intact? Confirm platform audit logging, proxy access logs and CA issuance logs were running and have not been reduced or redirected.
7. Is personal or regulated data in the affected services? This decides the notification path and its clock.

Containment priorities

Work top down and record each action with a time. Coordinate the first four steps into one window so the attacker cannot re-enter through a path you have not yet closed.

1. Preserve evidence: export platform audit logs, all mesh configuration objects, CA issuance records and a sample of proxy configuration and certificates from affected workloads to storage the platform administrators cannot alter.
2. Lock administrative access: remove or suspend role bindings that can change mesh configuration, webhooks or CA secrets, except a named response group using break-glass accounts. Disable CI pipelines that deploy mesh configuration until their credentials are rotated.
3. Re-assert security policy: reapply strict mutual authentication across namespaces and a default-deny authorization baseline from source control. Remove unknown egress routes, external service entries and mirroring rules.
4. Restrict egress at the network layer: because the mesh egress gateway may be attacker controlled, apply platform network policies or cloud firewall rules that limit outbound traffic from workloads and gateways to known destinations while you investigate.
5. Revert the injection path: restore the sidecar injection webhook and proxy image references to known good, pinned by digest, and block deployment of unsigned proxy images through admission control (see the Kubernetes Admission Control Guide).
6. Isolate suspect workloads: label and quarantine pods that ran attacker code or carried forged identities, keeping them for forensics rather than deleting them.

Containment decision: trust chain or not

Credential rotation and trust chain replacement

Rotation in a mesh is a sequence, because proxies must trust the new chain before workloads present certificates signed by it. The SPIFFE trust domain and bundle specification describes the mechanism: keys are added and revoked by publishing a new bundle with new keys included and revoked keys omitted, and each bundle carries a spiffe_sequence number that must increase whenever the contents change. In an emergency the order is:

1. Generate new signing material in an HSM or cloud KMS, not a platform secret. Constrain intermediates with X.509 name constraints to your trust domain; the X.509-SVID specification describes name constraints as a way to limit the blast radius of a compromised signing certificate.
2. Publish a bundle containing the new root and, for a short overlap, the old one, only if you must avoid an outage and you have no evidence of forged certificates. If forgery is likely, publish a bundle without the old root and accept the outage window.
3. Confirm propagation by checking the bundle sequence number reported by proxies across clusters. The specification notes that the refresh hint governs how quickly a key redaction propagates, and suggests clients default to a low refresh interval, such as five minutes, when no hint is set.
4. Restart or reissue every workload so each holds a leaf certificate from the new chain, then remove the old root from the bundle and publish again.
5. Rotate the control plane's own credentials: its platform service account tokens, any cloud credentials used to reach a KMS or upstream CA, and gateway TLS keys for public listeners. The internal PKI side of this work is covered in the If your internal PKI and certificate authority is compromised playbook, and public certificates in the Leaked TLS Private Key Response.
6. Rotate what travelled through the mesh: OAuth tokens, API keys, database passwords and JWT-SVIDs passed in headers between services. The JWT-SVID specification points out that interception of a bearer token grants the attacker its full privileges until it expires. Use the Secrets Rotation Runbook Template to track each one.

Eradication and rebuild

• Configuration diff: compare every mesh resource (authentication policy, authorization policy, routing, gateways, external services, telemetry and extension or filter configuration) against the Git source of truth. Delete what has no owner and change record.
• Proxy extensions: review any custom filters, scripts or WebAssembly modules loaded into proxies; these run inside every request path and are an easy place to hide credential capture.
• Webhooks and operators: check mutating and validating admission webhooks, mesh operators and their service accounts. An operator with cluster-wide rights can reinstall attacker configuration after you remove it.
• Images: rebuild control plane and proxy images from trusted sources, verify signatures and pin by digest.
• Node level: for sidecarless or node proxy designs, a proxy compromise is a node compromise: drain, rebuild the node from a trusted image and reissue the node's identity.

Rebuild or clean in place

• Rebuild the control plane: when the attacker held administrative rights for days, audit logs have gaps, or extensions were modified. Install a fresh control plane from source control into a new namespace or cluster with a new trust chain, and migrate workloads in waves.
• Clean in place: when access was short, logs are complete for the whole window and the CA key was never readable.
• Small teams: a single cluster with one mesh usually rebuilds faster than it investigates; plan a maintenance window and accept a full restart.
• Large estates: multi-cluster meshes with federation need a cutover plan per trust domain, and a short period where both old and new chains run in separate domains rather than a shared bundle.

Recovery

1. Re-enable change access in stages through pull requests and time-bound elevation, not standing cluster-admin.
2. Move the signing key out of reach: keep it in an HSM or KMS with audit logging and alerting on every signing call outside the control plane's identity.
3. Enforce policy as code: admission policies that reject permissive authentication, wildcard principals and unapproved egress in production namespaces.
4. Re-baseline and watch: rebuild the expected service-to-service graph from telemetry, alert on new edges, and watch the identities, IPs and images seen in the incident for at least 30 days.

Detection and measurement

Evidence to preserve

Hash exports and record chain of custody; see the Evidence Handling and Chain of Custody Guide.

Notification triggers

• Personal data: GDPR Article 33 requires a controller to notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of a personal data breach, unless it is unlikely to result in a risk to people. Interception of service traffic carrying personal data can meet that test even without proven exfiltration.
• Sector and listed company rules: NIS2, financial sector rules and securities disclosure rules may apply; see the Breach Notification Laws by Country Guide.
• Federation partners: tell any organisation whose trust domain accepted your bundle, so it can remove the compromised root.

Common failure patterns

• Waiting for certificates to expire: short lifetimes limit future exposure, but a stolen signing key mints new certificates until it is removed from every bundle.
• Rotating leaves, not the chain: restarting pods reissues certificates from the same compromised intermediate.
• Leaving permissive mode on "temporarily": a namespace in permissive mode accepts plaintext callers and undoes the rotation.
• Forgetting the CI pipeline: the deploy token that pushes mesh configuration reapplies the attacker's change at the next merge.
• Ignoring secrets in transit: tokens that crossed an attacker-controlled proxy stay valid. Rotate every credential that could have crossed the mesh.

Framework mapping

Practitioner checklist

• The mesh CA signing key lives in an HSM or KMS, and every signing call is logged and alerted outside the control plane identity.
• All mesh configuration is in Git, deployed by pipeline, with drift detection.
• Admission policy rejects permissive authentication and wildcard authorization in production.
• A tested root and intermediate rotation runbook exists, with propagation checked by bundle sequence number.
• Network-layer egress controls exist independent of the mesh egress gateway.
• Proxy images and injection webhooks are pinned by digest and signature-verified.
• Platform audit logs, proxy access logs and CA issuance logs go to storage outside platform admin reach.
• Federation partners and their contacts are documented for emergency bundle removal.

Standards and references

• SPIFFE: SPIFFE Trust Domain and Bundle
• SPIFFE: The X.509 SPIFFE Verifiable Identity Document
• SPIFFE: The JWT SPIFFE Verifiable Identity Document
• NIST: SP 800-204A Building Secure Microservices-based Applications Using Service-Mesh Architecture
• NIST: SP 800-204B Attribute-based Access Control for Microservices-based Applications using a Service Mesh
• NIST: SP 800-207A A Zero Trust Architecture Model for Access Control in Cloud-Native Applications in Multi-Cloud Environments
• NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• GDPR: Article 33 Notification of a personal data breach to the supervisory authority

Related NHI Mgmt Group resources: Service Mesh Security Guide, mTLS Guide, Kubernetes Cluster Compromise Playbook, If your internal PKI and certificate authority is compromised, Cloud Incident Response Guide.