Service Mesh Security Guide: Istio, Linkerd and mTLS

Installing a service mesh does not make service-to-service traffic secure: most meshes start in a mode that still accepts plaintext and still allows every caller, and the control plane becomes a certificate authority that most teams never harden. A mesh can give every workload a cryptographic identity, encrypt traffic with mutual TLS and enforce identity-based authorisation, but only once it is configured to do so. This guide covers the Istio service mesh in sidecar and ambient mode, Linkerd, and the security models of Consul, Cilium and Kuma: mesh mTLS modes, workload identity issuance, authorization policy, control plane hardening, multi-cluster trust and observability. It is written for platform engineers, Kubernetes security teams and architects building zero trust between workloads. For the identity foundations underneath, see the Guide to SPIFFE and SPIRE.

Key takeaways

• Permissive is a migration state, not a setting. Istio proxies start in permissive mode and Linkerd's default inbound policy allows all traffic; plan the move to strict mTLS and default-deny from day one.
• mTLS authenticates, policy authorises. Encryption and identity mean nothing for access control until an authorization policy says which identities may call which services.
• The mesh CA is a tier-zero credential. Whoever holds the istiod signing key or Linkerd issuer key can mint any workload identity in the mesh.
• A sidecar is not a boundary for its own pod. An application can bypass its own proxy; the guarantee is that a client cannot bypass another pod's proxy. Keep Kubernetes NetworkPolicy underneath.
• Measure enforcement, not installation. Track the share of connections reported as mutual_tls, workloads under default-deny and certificate expiry, not the number of namespaces with injection enabled.

What a service mesh secures, and what it does not

A service mesh is an infrastructure layer that moves service-to-service communication concerns (identity, encryption, routing, retries, telemetry) out of application code and into proxies managed by a central control plane. NIST SP 800-204A describes it as a way to apply authentication, authorisation, secure communication and availability controls consistently across microservices. The two halves matter separately for security:

• Control plane: distributes configuration and issues workload certificates. In Istio this is istiod, which "offers a gRPC service to take certificate signing requests (CSRs)". In Linkerd the identity component is the CA.
• Data plane: the proxies that terminate mTLS and enforce policy. Istio sidecars and waypoints use the Envoy proxy; Linkerd uses its own lightweight Rust proxy; Istio ambient mode uses a per-node Rust proxy called ztunnel.

A mesh does not replace network controls. Istio's own best practices note that traffic capture is TCP-only, some ports are excluded from capture, and the application can interfere with redirection inside its own pod. It also does not authenticate end users by default: Istio RequestAuthentication rejects invalid JWTs, but requests with no token "are accepted by default" unless an authorization policy requires one. A zero trust service mesh therefore combines mesh mTLS, authorization policy, Kubernetes NetworkPolicy and application-level user authorisation. See the Network Segmentation Guide.

Sidecar and sidecarless data planes

In a sidecar mesh, a sidecar proxy container runs in every application pod and intercepts its traffic. In a sidecarless service mesh, proxies run per node or per namespace instead. Istio supports both (its sidecarless option, ambient mode, is often called ambient mesh), and the two can coexist in one mesh.

The practical rule: if you need L7 authorisation in ambient mode, you need a waypoint for that destination, and a policy written for sidecars will not apply to it until it is rewritten with targetRefs.

Mesh mTLS: strict, permissive and workload identity

Mesh mTLS, sometimes called workload mTLS, gives each workload an X.509 certificate tied to its Kubernetes service account and verifies both sides of every connection. In Istio the agent generates the private key and CSR, istiod signs it, and Envoy receives the certificate through the secret discovery service (SDS); the agent rotates it before expiry. The identity is encoded as a SPIFFE ID of the form spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>, and policies refer to it as a principal such as cluster.local/ns/default/sa/curl.

PeerAuthentication modes in Istio

• PERMISSIVE: "Workloads accept both mutual TLS and plain text traffic." The default, intended for migration.
• STRICT: "Workloads only accept mutual TLS traffic." The target state.
• DISABLE: mTLS off. Istio says "you shouldn't use this mode unless you provide your own security solution". Not supported in ambient mode.
• UNSET: inherits from the parent scope; a mesh-wide policy left unset behaves as permissive.

Istio applies the narrowest matching policy: workload, then namespace, then mesh-wide (a policy without a selector in the root namespace, usually istio-system). Only one mesh-wide and one namespace-wide policy is honoured. Port-level exceptions through portLevelMtls work only with a workload selector and use the container port, not the Service port. Every port-level DISABLE is a hole in strict mTLS and should be on an exceptions register.

Certificate lifetimes and trust roots by mesh

Istio with SPIRE

Istio can take identities from SPIRE instead of istiod by detecting a socket that implements the Envoy SDS API, which Istio and SPIRE recommend mounting with the SPIFFE CSI driver rather than host mounts. The benefit is stronger attestation: SPIRE's node attestation "extends attestation to the physical or virtual hardware on which workloads run". The caveats are operational: SPIRE and Istio must use exactly the same trust domain, and a proxy cannot reach READY without a pre-existing matching registration, which is usually automated with ClusterSPIFFEID resources. See the Guide to SPIFFE and SPIRE.

Service mesh authorization

Strict mTLS proves who is calling; service mesh authorization decides whether they may. Without a policy, Istio "allows all requests". Once a workload has any ALLOW policy, unmatched requests are denied.

Istio AuthorizationPolicy

An Istio authorization policy is enforced by the proxy in front of the destination workload.

• Actions and order: Istio evaluates CUSTOM (an external authoriser), then DENY, then ALLOW. Any matching DENY wins. AUDIT marks requests for an audit plugin without changing the outcome.
• Default-deny: an AuthorizationPolicy with an empty spec: {} "allows nothing and effectively denies all requests" in its namespace; placed in the root namespace it applies mesh-wide. Add explicit ALLOW policies per service.
• Identity fields need mTLS: principals and namespaces come from the peer certificate, so they only work under mTLS. requestPrincipals come from a validated JWT.
• Positive and negative matching: Istio recommends ALLOW with positive fields (paths, values) and DENY with negative fields (notPaths, notValues), so a mismatch fails closed with a 403 rather than slipping past a deny rule.
• Path normalisation: Envoy and the backend may read a path differently, which can bypass path-based rules. Choose the strictest pathNormalization option your applications tolerate (NONE, BASE by default, MERGE_SLASHES, DECODE_AND_MERGE_SLASHES) and test both allowed and denied paths.
• Source IPs: ipBlocks uses the packet source; remoteIpBlocks uses X-Forwarded-For and needs numTrustedProxies configured, or the header can be spoofed.

Equivalent controls in other meshes

Linkerd's config.linkerd.io/default-inbound-policy annotation can override the cluster default per namespace or pod, which is useful for tightening sensitive namespaces first and dangerous if teams can loosen their own.

Hardening the service mesh control plane

The Istio control plane is both configuration server and CA, so a compromise of istiod or its signing key compromises every workload identity. Istio's best practices give specific steps:

• Close debug and plaintext ports: istiod exposes 8080 (debug, disable with ENABLE_DEBUG_ON_HTTP=false, at the cost of some istioctl commands), 15010 (plaintext XDS, disable with --grpcAddr=""), 15012 (mTLS XDS, the production port) and 15014 (debug endpoints requiring tokens with the istio-ca audience). Keep ENABLE_DEBUG_ENDPOINT_AUTH on.
• Use a proper CA chain: replace the self-signed root with an intermediate issued from an offline root in your PKI or an HSM-backed key service, so the root key never sits in a cluster Secret. See the Machine Identity, PKI and Certificate Lifecycle Guide.
• Lock down mesh configuration: restrict who can create or edit PeerAuthentication, AuthorizationPolicy, Gateway and root-namespace objects with Kubernetes RBAC and admission policy. A single permissive policy in the root namespace weakens the whole mesh. Read access to the istio-system or linkerd namespace Secrets is equivalent to CA access.
• Prefer third-party tokens: use jwtPolicy=third-party-jwt so proxies authenticate to istiod with audience-scoped, expiring service account tokens rather than non-expiring first-party tokens. See the Kubernetes NHI Security Guide.
• Remove pod privileges: use the Istio CNI plugin so application pods do not need NET_ADMIN and NET_RAW.
• Harden gateways: restrict Gateway creation, avoid wildcard hosts, use dedicated gateways for sensitive services, and set caCertificates, subjectAltNames and sni together on DestinationRules that originate TLS to external services.
• Treat egress as best-effort unless gated: outboundTrafficPolicy: REGISTRY_ONLY is not a security boundary; route egress through an egress gateway backed by NetworkPolicy.
• Patch and validate: stay on a supported release and the latest patch, run istioctl analyze on every change, avoid alpha features, and set a downstream connection limit, since the default is unlimited and Istio security bulletin 2020-007 showed it can be exploited.

Multi-cluster service mesh trust

Workloads in different clusters can only authenticate each other if they share or exchange roots of trust. Istio defines a trust bundle as "the set of all CA public keys used by an Istio mesh" and states that meshes with different CAs "must exchange the trust bundles", manually or through SPIFFE trust domain federation. Linkerd multi-cluster requires a shared trust anchor, so the default per-install anchor does not work.

• Remote secrets are cluster credentials: in Istio primary-remote and multi-primary models, a remote secret "contains credentials, granting access to the API server in the cluster". Store and rotate it like any privileged NHI credential.
• Namespace sameness: same-named namespaces in different clusters are treated as one, and their endpoints merge. A namespace created in a less trusted cluster can attract traffic meant for a production service.
• Isolation choices: not exchanging remote secrets gives the strongest isolation; cluster or mesh tenancy suits teams that need separate control of configuration.
• Control plane exposure: expose the primary control plane to remote clusters through internal load balancers rather than public addresses.

Choosing a mesh model

• Need L7 policy on most services: Istio sidecar mode, or ambient with waypoints per namespace. Ambient reduces per-pod overhead but concentrates risk in node-level ztunnel.
• Need mTLS and simple service-level policy with minimal tuning: Linkerd's on-by-default mTLS and short-lived certificates suit smaller platform teams.
• Mixed VMs and Kubernetes, or multiple runtimes: Consul service mesh (formerly Consul Connect) or Kuma service mesh, which both span environments beyond a single cluster.
• Already standardised on Cilium networking: Cilium can add mutual authentication, but at beta maturity it should not be the only control for regulated workloads.
• Strong workload attestation requirements: Istio with SPIRE, accepting the registration and trust-domain overhead.

Small teams should pick one mesh, one cluster and one trust domain, and reach strict mTLS plus default-deny before adding features. Large organisations need a mesh platform team that owns the root namespace, CA integration and policy templates, with application teams limited to namespace-scoped AuthorizationPolicy.

Rolling out strict mTLS and default-deny

1. Baseline: enable the mesh in PERMISSIVE mode on a pilot namespace and collect metrics to see which callers are already mTLS and which are plaintext.
2. Fix the CA: integrate an intermediate CA from your PKI (or SPIRE) before production traffic depends on mesh identities, because changing roots later means a trust bundle migration.
3. Inject or enrol all callers: move every client of the pilot namespace into the mesh; record remaining plaintext callers such as health checks or legacy jobs.
4. Enforce STRICT per namespace: apply namespace-level STRICT, watch for connection failures, then apply mesh-wide STRICT in the root namespace. Rollback is reverting the namespace policy to PERMISSIVE, not disabling mTLS.
5. Audit authorisation: write ALLOW policies from observed source_principal to destination_principal pairs, test them with Istio's AUDIT action (which needs an audit plugin) or Linkerd audit mode, and review denied-request counts.
6. Default-deny: apply allow-nothing per namespace, then mesh-wide. Keep a tested break-glass ALLOW policy, owned by the platform team, for incident rollback.
7. Close the gaps: remove port-level DISABLE exceptions, add NetworkPolicy beneath the mesh, and harden the control plane.

Service mesh observability and detection

Mesh telemetry is one of the best sources of east-west visibility in a cluster. Istio's standard metrics (istio_requests_total, istio_tcp_connections_opened_total and others) carry source_principal, destination_principal, response_code, response_flags and connection_security_policy, which is set to mutual_tls when the destination proxy reports a secured connection.

• Alert on: destination-reported traffic without mutual_tls in strict namespaces; spikes in 403 responses after policy changes (possible over-blocking) or from one principal (possible probing); new source_principal values calling sensitive services; changes to root-namespace PeerAuthentication or AuthorizationPolicy; rejected configuration via pilot_total_xds_rejects; Kubernetes audit events reading CA Secrets.
• Metrics: percentage of east-west connections reported as mutual_tls; percentage of namespaces under STRICT and default-deny; number of port-level mTLS exceptions; days until trust anchor, issuer or intermediate expiry; proxy versions within the supported patch window; mean time to revoke or rotate a compromised workload identity.

Common failure patterns

• Permanent permissive mode: the mesh is installed, dashboards show encryption, but workloads still accept plaintext, so any pod that skips the mesh can call them.
• mTLS without authorisation: every workload has an identity, but with no AuthorizationPolicy every identity can call every service. Identity is in place; least privilege is not.
• Exposed ingress gateways: in the Kubeflow cryptomining attacks, users switched the istio-ingressgateway service to a public load balancer, exposing an internal dashboard to the internet.
• Default CA in production: a self-signed mesh root stored as a cluster Secret, readable by anyone with broad Secret access, with no rotation plan and an anchor that silently expires after a year.
• Deny rules with positive matching: DENY policies on specific paths are bypassed through encoding or slash variants because path normalisation was left at the default and never tested.
• Relying on REGISTRY_ONLY for egress: a compromised pod bypasses its own sidecar and reaches the internet directly because no egress gateway or NetworkPolicy backs the mesh setting.

Framework mapping

Practitioner checklist

• Set mesh-wide STRICT mTLS (or the mesh equivalent) and keep a register of every PERMISSIVE namespace and port-level exception with an end date.
• Apply default-deny authorisation mesh-wide and grant access per service by principal, not by IP or namespace alone.
• Write ALLOW rules with positive matching and DENY rules with negative matching, and set and test path normalisation.
• Replace the default self-signed mesh root with an intermediate from an offline or HSM-backed root, and calendar trust anchor and issuer rotation.
• Restrict who can edit root-namespace mesh policy, Gateways and CA Secrets with RBAC and admission control.
• Close istiod debug and plaintext XDS ports, use third-party service account tokens and the Istio CNI plugin.
• Keep Kubernetes NetworkPolicy and an egress gateway beneath the mesh.
• Rewrite L7 policies with targetRefs and deploy waypoints before moving workloads to ambient mode.
• Exchange trust bundles explicitly for multi-cluster meshes and protect remote secrets as privileged credentials.
• Alert on non-mTLS traffic, policy changes and CA Secret access, and stay within the supported patch window.

Standards and references

• NIST: SP 800-204A Building Secure Microservices-based Applications Using Service-Mesh Architecture
• Istio: Security concepts
• Istio: Security best practices
• Istio: Authorization Policy reference
• Istio: Ambient mode overview
• Istio: SPIRE integration
• Linkerd: Automatic mTLS
• Linkerd: Authorization policy reference

Related NHI Mgmt Group resources: Guide to SPIFFE and SPIRE · Kubernetes Security Guide · Kubernetes NHI Security Guide · Zero Trust Identity Guide · Network Segmentation Guide