Service mesh rollout playbook: phased implementation with security built in

Service meshes are often adopted for traffic management and observability, and their security features are left in permissive mode because nobody planned the trust domain, the certificate authority or who owns authorisation policy. A mesh puts a proxy and a workload certificate in front of every service, which makes it one of the largest non-human identity systems in an organisation and a new control plane with cluster-wide power. This playbook sets out a vendor-neutral, phased rollout for a service mesh on Kubernetes and adjacent platforms: the decisions to make first, the foundations to build before onboarding, how to pilot, how to onboard services in waves, how to reach strict mutual TLS and default-deny without outages, and how to measure progress. It is written for platform engineering, cloud security and architecture teams. The security model itself, including data plane options and policy semantics, is covered in the Service Mesh Security Guide; this page is about running the rollout programme.

Key takeaways

• Trust domain and certificate authority come first. Workload identities, certificate chains and federation all depend on them, and changing a root after production traffic relies on it means a trust bundle migration.
• The control plane is a privileged system. It issues every workload identity and pushes every policy, so harden it, restrict who can change it and log it before the first service joins.
• Onboard in waves, enforce in steps. Permissive mutual TLS, then strict per namespace, then audit-mode authorisation, then default-deny, each with exit criteria and a rollback that does not switch security off.
• Keep network policy underneath the mesh. Layer 3 and 4 Kubernetes NetworkPolicy still matters for traffic that bypasses the proxy and for workloads outside the mesh.
• Measure identity coverage. The rollout is working when a known share of service-to-service traffic is mutually authenticated and covered by explicit allow policies.

Before you start: decisions that are hard to change later

NIST SP 800-204A describes a proxy-based mesh as security infrastructure for microservices, and NIST SP 800-207A positions sidecar proxies, gateways and an application identity system such as SPIFFE as the enforcement platform for identity-based zero trust policy. The decisions below shape that platform; record them in a design record approved by platform and security owners.

Small versus large organisations: a small platform team should run one mesh, one cluster and one trust domain, use the mesh CA chained to an enterprise intermediate, and reach strict mutual TLS and default-deny before adding features. A large organisation needs a mesh platform team that owns the control plane, CA integration and policy templates, with application teams writing namespace-scoped policy through code review, and a federation design before the second cluster joins.

The rollout at a glance

Phase 0: inventory and design

1. Map callers, not just services. For each namespace in scope, list inbound callers, including health checks, monitoring scrapers, batch jobs and anything outside the cluster. These are the callers that break when strict mutual TLS is enforced.
2. Identify workloads that should not join. Databases with their own TLS, stateful systems with client certificate authentication and high-throughput data paths may stay outside the mesh with network policy and their own transport security.
3. Define identity naming and ownership. Agree how workload identities map to Kubernetes service accounts and who may create service accounts in each namespace. Shared default service accounts make identities meaningless; give each workload its own. See the Kubernetes Service Account Token Guide.
4. Set certificate lifetimes. Short-lived workload certificates reduce the value of a stolen key. NHI Mgmt Group suggests workload certificates measured in hours rather than days, with the intermediate CA lifetime and rotation planned separately.

Phase 1: foundations before onboarding

• Control plane access: restrict who can change mesh-wide configuration, CA settings and the mesh's own namespace to the platform team through Kubernetes RBAC, and route those changes through code review. The Kubernetes RBAC Guide covers role design. See also Securing the service mesh itself.
• CA integration: plug in the enterprise intermediate before any production service depends on mesh identities. Keep the signing key in an HSM or KMS where the mesh supports it, and document how to rotate it. The SPIFFE X.509 SVID specification requires exactly one URI SAN per workload certificate and describes name constraints on signing certificates to limit the blast radius of a compromised intermediate, while warning that support for URI name constraints is uneven.
• Admission and injection: control proxy injection with labels set by the platform, and use admission policy to stop workloads from disabling the proxy, requesting privileged capabilities the proxy does not need, or using another namespace's service account. The Kubernetes Admission Control Guide covers the mechanisms.
• Network policy baseline: Kubernetes NetworkPolicy works at layer 3 and 4 and only takes effect when the cluster's network plugin enforces it. Put a default-deny policy and explicit allows beneath the mesh so a workload that bypasses its proxy is still contained.
• Telemetry: send control plane audit events, certificate issuance events, policy changes and proxy access logs (with source and destination identity) to the SIEM. Avoid logging request bodies and authorisation headers.
• Upgrade path: decide how control plane and proxy versions will be upgraded, for example a canary control plane revision with namespaces moved across, and test it in non-production before onboarding production. Proxies that fall behind miss security fixes.

Phases 2 and 3: pilot and onboarding waves

1. Pilot in permissive mode: onboard two or three namespaces that represent your patterns (a web front end, an internal API, a worker). Permissive mode accepts both mutual TLS and plaintext so you can see which callers are not yet in the mesh.
2. Measure cost and behaviour: record added latency, proxy memory and CPU, and connection errors. Check that retries and timeouts set in the mesh do not hide failures or multiply load.
3. Fix the callers: enrol remaining callers in the mesh, or move health checks and scrapers to paths the mesh supports. Record any caller that must stay plaintext, with an owner and an end date.
4. Enforce strict mutual TLS per namespace: once no plaintext callers remain, enforce strict mode for that namespace and watch for connection failures. Move to mesh-wide strict only when every namespace is enforced.
5. Onboard in waves: group namespaces by risk and dependency, low-risk first, and onboard callers before the services they call. Each wave has a change window, an owner on call and a soak period, which NHI Mgmt Group suggests is at least one week per wave.
6. Rollback: the rollback for a namespace is returning it to permissive mode or removing injection for that namespace, not disabling mutual TLS mesh-wide. Keep the namespace policy in code so rollback is a revert.

Phase 4: authorisation and default-deny

NIST SP 800-204B describes mutual authentication between every pair of services plus attribute-based access control as the two core security needs a mesh addresses. Mutual TLS alone proves who is calling; authorisation decides whether they may.

• Write allow policies from observed pairs: use proxy telemetry to list source identity to destination service pairs, confirm them with service owners, and write explicit allow policies. Add method and path conditions for sensitive APIs.
• Test before enforcing: use the mesh's audit or dry-run mode where it has one, or apply policies in a staging copy of the namespace, and review would-be denials.
• Default-deny: apply an allow-nothing baseline per namespace, then mesh-wide, once allow policies cover observed traffic. Keep a tested break-glass allow policy owned by the platform team for incident rollback.
• Egress: route outbound traffic through an egress gateway with an allow list of external destinations, so a compromised workload cannot reach arbitrary hosts.
• End-user identity: where requests carry user tokens, validate them at the ingress gateway or the service and pass identity downstream; the workload certificate identifies the service, not the user.

Where identity fits

Every workload certificate in a mesh is a non-human identity, issued automatically at scale. That brings the usual NHI risks in a new form.

• Service account sprawl: identities derive from Kubernetes service accounts, so unused or shared accounts become unused or shared mesh identities. See How to find unused Kubernetes service accounts to clean up before onboarding.
• CA keys: the intermediate signing key can mint any identity in the trust domain. Protect it like a root of trust and plan its compromise response with If your service mesh is compromised.
• Federation: trusting another trust domain's bundle means accepting any identity that domain asserts. Federate only with domains whose issuance you can audit.
• Non-mesh credentials: a mesh does not remove API keys and database passwords in application config; they still need a secrets manager. The Service Mesh Identity Checklist covers the identity controls in short form.

Detection and measurement

Metrics that show the rollout is working:

• Percentage of in-cluster service-to-service requests carried over mutual TLS.
• Percentage of namespaces in strict mode and under default-deny authorisation.
• Number of open plaintext or mutual TLS exceptions, and how many are past their end date.
• Percentage of workloads with a dedicated service account rather than a shared or default one.
• Control plane and proxy versions behind the latest supported release.
• Percentage of external destinations reached through the egress gateway allow list.

Common failure patterns

• Permissive forever: the mesh is installed for observability and mutual TLS never moves to strict, so any plaintext caller is still accepted.
• Self-signed root in production: the default mesh CA goes live, and replacing it later forces a disruptive trust bundle migration.
• Mutual TLS mistaken for authorisation: every service can call every other service because no allow policies were written.
• Shared default service accounts: many workloads share one identity, so policies cannot tell them apart.
• Rollback by disabling security: an outage is fixed by switching off mutual TLS mesh-wide instead of reverting one namespace.
• No network policy underneath: a workload that skips its proxy can reach anything.

Framework mapping

Practitioner checklist

• Approve a design record covering data plane model, trust domains, CA, identity naming, policy ownership and scope.
• Chain the mesh CA to an enterprise intermediate with protected keys before production onboarding.
• Restrict control plane and mesh-wide configuration changes to the platform team through RBAC and code review.
• Enforce proxy injection and service account rules with admission policy.
• Put default-deny Kubernetes NetworkPolicy beneath the mesh, and confirm the network plugin enforces it.
• Pilot in permissive mode, find plaintext callers, and fix or record each with an owner and end date.
• Enforce strict mutual TLS per namespace, wave by wave, with namespace-level rollback in code.
• Write allow policies from observed identity pairs, test them, then apply default-deny with a tested break-glass policy.
• Route egress through a gateway with an allow list of external destinations.
• Send control plane, issuance, policy and identity-aware access logs to the SIEM and track the metrics above.

Standards and references

• NIST: SP 800-204A, Building Secure Microservices-based Applications Using Service-Mesh Architecture
• NIST: SP 800-204B, Attribute-based Access Control for Microservices-based Applications using a Service Mesh
• NIST: SP 800-207A, A Zero Trust Architecture Model for Access Control in Cloud-Native Applications in Multi-Cloud Environments
• SPIFFE: SPIFFE ID and Trust Domain specification
• SPIFFE: X.509 SVID specification
• Kubernetes: Network Policies
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls

Related NHI Mgmt Group resources: Service Mesh Security Guide · Securing the service mesh itself · Service mesh integration security guide · Service mesh health check · mTLS Guide