Coordination Services Security Guide: Hardening etcd, ZooKeeper and Consul

Coordination services such as etcd, Apache ZooKeeper and HashiCorp Consul hold the state that distributed systems trust, including secrets, tokens and service configuration, yet none of them is secure by default. etcd is the backing store for all Kubernetes cluster data, ZooKeeper underpins many data platforms, and Consul stores service catalogues, key-value configuration and ACL tokens. Anyone who can read these stores can usually read the credentials of every workload that depends on them, and anyone who can write to them can change how the system behaves. This guide covers how authentication and authorisation work in each service, how to handle root and management credentials, the machine identities that connect to them, what to log, and a hardening baseline. It is for platform, SRE and identity engineers; Kubernetes Secret handling itself is covered in the Kubernetes Secrets Security Guide.

Key takeaways

• Insecure defaults are the norm. etcd's documentation states that TLS and RBAC are not enabled by default, Consul's security model says Consul is not secure by default, and ZooKeeper's open ACL grants anyone full control of a znode.
• Read access is credential access. These stores hold Kubernetes Secrets, service tokens and configuration with embedded passwords, so treat any client with broad read rights as privileged.
• Authenticate every client with a machine identity. Use client certificates (etcd, ZooKeeper x509, Consul mTLS) or scoped ACL tokens, never shared passwords or anonymous access.
• Root and management credentials need break-glass handling. etcd's root role, ZooKeeper super users and Consul's initial management token bypass all other controls.
• Backups and snapshots are as sensitive as the live store. etcd snapshots contain cluster state and Consul snapshots contain ACL tokens, so protect, encrypt and restrict them.

How identity and access work in coordination services

Two points catch teams out. In etcd, a user created with --no-password can authenticate only through a TLS Common Name, and if a client supplies both a certificate and a username and password, the password credentials take priority. In ZooKeeper, ACLs apply only to the znode they are set on, so a child can be more open than its parent, and digest authentication sends username:password in clear text unless the connection is encrypted.

Admin roles and privileged access

• etcd root: the root user must exist before etcdctl auth enable. The root role gives global read-write access plus authentication configuration and maintenance such as membership changes, defragmentation and snapshots. Grant it to as few identities as possible, use separate roles with prefix-scoped permissions for everything else, and keep the root password or certificate in a vault with break-glass procedures.
• Kubernetes and etcd: in a Kubernetes cluster, the API server should be the only routine etcd client. Direct etcd access bypasses Kubernetes RBAC, admission control and audit logging, so a person or process with etcd client credentials effectively has full control of the cluster. See the Kubernetes RBAC Guide.
• ZooKeeper super users: zookeeper.superUser and the x509 super user bypass ACL checks. The superDigest provider is disabled by default; ZooKeeper's documentation advises using it only on localhost or over an encrypted connection because digest credentials travel in plaintext.
• Never set skipACL in production: ZooKeeper's documentation states that it opens full access to the data tree to everyone.
• Consul management token: created by consul acl bootstrap or set in acl.tokens.initial_management, it links to the global-management policy with unrestricted access. Store it offline or in a vault, create scoped operator tokens for daily administration, and rotate by cloning to a new token and deleting the old one.
• Consul anonymous token: all unauthenticated requests receive the anonymous token's permissions. It cannot be deleted, so keep its policies to the minimum needed, such as read-only service discovery for DNS.

Treat every one of these superuser credentials as a break-glass account: two named custodians, storage in a vault with checkout logging, an alert on every use, and rotation after each use. For broader privileged access practice see the Privileged Access Management Guide.

Integrations, API tokens and service accounts

Almost every client of a coordination service is a non-human identity: API servers, brokers, schedulers, service mesh agents, CI pipelines and configuration tools. Inventory them the same way you would any other machine credential.

• Prefer dynamic credentials: Consul auth methods create tokens from a trusted external system such as Kubernetes service accounts, which avoids static tokens in configuration. Certificates from a short-lived PKI (for example a Vault PKI engine, which Consul's security model mentions for certificate rotation) avoid long-lived client keys. See the HashiCorp Vault Guide.
• Use the audit identifier: Consul tokens have a SecretID used for authentication and an AccessorID intended for audit logging. Record AccessorIDs in your NHI inventory; never log SecretIDs.
• Do not store secrets in plain KV: configuration KV stores are not secret managers. Where a secret must be distributed, store a reference and fetch it from a vault.

• Decommission with the workload: when a service, broker or pipeline is retired, revoke its certificate or token and remove its role or ACL entries in the same change. Orphaned client identities with write access to configuration are an easy persistence route.

See the NHI Ownership Guide and the NHI Offboarding and Decommissioning Guide for ownership and retirement of these identities.

Logging and detection

• Kubernetes audit logs: because Kubernetes reads and writes etcd through the API server, its audit log is where secret reads and changes are recorded. Direct etcd traffic from any host other than the API servers is itself a high-fidelity alert. See the Kubernetes Audit Log Detection Guide.
• Network telemetry: alert on connections to client and peer ports from unexpected sources. etcd's examples use 2379 for clients and 2380 for peers; record the ports your deployment actually uses for each service.
• Authentication failures: collect TLS handshake failures, SASL failures and ACL "permission denied" errors from each service's logs and forward them to the SIEM.
• Privileged operations: alert on etcd auth disable, user and role changes, membership changes and snapshot requests; on Consul ACL token and policy creation, especially tokens linked to global-management; on ZooKeeper ACL changes (setAcl) on sensitive paths.
• Admin interfaces: ZooKeeper's AdminServer supports snapshot and restore commands, both enabled by default. Log and restrict them, or disable the AdminServer if unused.

Metrics that show the controls are working:

1. Percentage of coordination service clusters with TLS and client authentication enforced on client and peer listeners (target 100%).
2. Number of identities holding etcd root, ZooKeeper super user or Consul global-management rights (target: a small, named list).
3. Percentage of client credentials that are short-lived certificates or dynamically issued tokens.
4. Connections to client ports from outside the approved client list per month.
5. Percentage of snapshots stored encrypted with restricted access.

In ATT&CK terms, theft of credentials from orchestration stores relates to T1552.007 Container API and discovery of cluster resources to T1613 Container and Resource Discovery; internet-exposed instances are an initial access route under T1190.

Hardening baseline

Roll changes out in this order: place the service on a private network segment with firewall rules first (no client changes needed), enable TLS on peers, then on clients with a migration window (ZooKeeper's port unification allows mixed TLS and plaintext during the change), then turn on client authentication, and finally enable authorisation with deny-by-default. Test each step in a staging cluster, keep the previous configuration and certificates for rollback, and take a protected snapshot before enabling authentication, because a mistake can lock out the API server.

Common misconfigurations

• Exposed client ports: etcd, ZooKeeper or Consul reachable from the internet or a flat internal network without authentication, letting anyone read every key, including Kubernetes Secrets and service tokens.
• TLS without client authentication: traffic is encrypted but any client can connect, or --auto-tls is used so certificates prove nothing about identity.
• One CA for everything: a single cluster CA issues etcd client certificates and other component certificates, so a certificate stolen from any node is accepted by etcd.
• Permissive defaults left in place: Consul default_policy allow, a broad anonymous token, ZooKeeper znodes created with the open ACL, or etcd authentication never enabled.
• Management credentials in daily use: agents or pipelines configured with the Consul management token or etcd root because scoped tokens were never created.
• Unprotected snapshots: backups copied to shared storage without encryption, giving anyone with storage access the full cluster state. The LastPass breach shows how backups and the DevOps secrets that unlock them become the target.

Framework mapping

Practitioner checklist

• Inventory every etcd, ZooKeeper and Consul cluster, its network exposure and the data it holds.
• Enable TLS on client and peer traffic and require client certificates; remove auto-generated certificates.
• Enable authorisation with deny-by-default: etcd RBAC, ZooKeeper ACLs on every application subtree, Consul ACLs with default_policy deny.
• Restrict etcd network access to the Kubernetes API servers and use a dedicated etcd CA.
• Vault root and management credentials, name their holders and alert on their use.
• Issue each client its own scoped certificate or token, prefer dynamic auth methods, and record Consul AccessorIDs in the NHI inventory.
• Disable or lock down AdminServer, four-letter-word commands, script checks, /metrics and UI exposure.
• Encrypt and restrict snapshots, and enable Kubernetes encryption at rest for Secrets.
• Forward authentication failures and privileged operations to the SIEM and alert on unexpected clients.

Standards and references

• etcd: Transport security model
• etcd: Role-based access control
• Apache ZooKeeper: Programmer's Guide, Access Control Using ACLs
• Apache ZooKeeper: Administrator's Guide
• HashiCorp: Consul security model
• HashiCorp: Consul ACL tokens
• Kubernetes: Encrypting Confidential Data at Rest
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: Kubernetes Secrets Security Guide · Kubernetes NHI Security Guide · HashiCorp Vault Guide · Kubernetes Security Guide