Distributed Systems Security Guide: Consensus, Byzantine Faults and Trust Between Nodes

Modern platforms are distributed systems: databases replicate across zones, Kubernetes stores its state in a consensus cluster, service meshes route between thousands of workloads and peer-to-peer networks carry ledgers, so an attacker who can join, impersonate or isolate a node can corrupt state that every other component trusts. Classic security controls protect a host or a perimeter; distributed systems also need protection of membership, messages, quorum and consistency. This guide draws on the CyBOK Distributed Systems Security knowledge area and turns it into practice: fault models and what they imply, attacks on peer-to-peer and coordinated systems, securing consensus clusters such as etcd, workload identity between nodes, and how zero trust principles apply to machine-to-machine trust. It is for platform, SRE and security architects who design or operate replicated and clustered systems.

Key takeaways

• Choose the fault model deliberately. Crash-fault consensus such as Paxos and Raft assumes nodes fail but do not lie; tolerating malicious nodes requires Byzantine fault tolerance, which CyBOK notes needs more than 3f nodes to tolerate f malicious ones.
• Membership is the security boundary. Sybil, eclipse and whitewashing attacks all exploit cheap or unverified identities, so admission control with certified node identities is the first defence.
• Consensus clusters are crown jewels. etcd's own documentation warns that a cluster without security features enabled can expose its data to any client, and its security features are off by default.
• Identical replicas fail together. CyBOK stresses that replicas sharing the same vulnerability give no protection against an attacker, so diversity and patch independence matter.
• Node-to-node trust is zero trust for machines. NIST SP 800-207's tenet of no implicit trust from network location applies to peers in a cluster: every node and workload should authenticate with a short-lived credential on every connection.

Two classes of distributed system

CyBOK distinguishes decentralised peer-to-peer systems, where peers act as both clients and servers without a central coordinator, from coordinated systems, where resources are orchestrated through coordination mechanisms such as client-server systems, cloud platforms, databases and ledgers. The attacks and defences differ.

Fault models and consensus

Consensus requires agreement, validity and termination, and every design starts from an assumption about how nodes can fail. Getting this assumption wrong is the most common architectural security error in distributed systems.

• Raft's guarantee is about crashes: the Raft project describes consensus making progress when any majority of servers is available, and a cluster that loses more than that stops making progress but never returns an incorrect result. That is a crash-fault guarantee. A compromised member with valid credentials is not a crash; it can write whatever its role allows.
• Byzantine tolerance is expensive: BFT needs more nodes, more messages and diverse implementations. Use it where participants do not trust each other (multi-organisation ledgers), not as a substitute for protecting members of a single-owner cluster.
• CAP trade-offs are attack surface: CyBOK notes that a networked data system can guarantee only two of consistency, availability and partition tolerance, and attacks aim to exploit the trade-off, for example by inducing partitions to cause stale reads or split decisions.
• Commit protocols: two-phase commit can block if the coordinator fails, and three-phase commit is more robust but more sensitive to partitions, so a coordinator under denial of service can stall transactions.

Attacks on peer-to-peer systems

CyBOK lists the attacks that exploit open membership. They are relevant wherever nodes discover each other automatically, including service discovery, gossip protocols and blockchain networks.

• Sybil: many fake identities inserted to outvote or surround honest peers.
• Eclipse: a victim surrounded by malicious peers so it sees only the attacker's view of the network; often built on Sybil identities or routing poisoning. CyBOK notes eclipse attacks can threaten Bitcoin consensus.
• Routing table poisoning and routing attacks: messages dropped, delayed or misdirected through corrupted routing state.
• Pollution and index poisoning: incorrect content or index entries spread to waste resources or mislead.
• Whitewashing: a peer with a bad reputation rejoins under a new identity.
• Churn and DoS: forced joins and leaves, or floods, exhaust bandwidth and destabilise the overlay.

Defences follow from the list: authenticate peers, admit them through a controlled or certificate-backed process, sign node identifiers, use multiple disjoint lookup paths, and monitor for Sybil clusters and churn spikes. CyBOK cautions that these defences weaken once colluding peers reach critical mass. See the Blockchain and Distributed Ledger Security Guide for ledger-specific controls.

Securing consensus clusters: etcd as the worked example

etcd holds the entire state of a Kubernetes cluster, including Secrets. Its documentation is explicit that security features are off by default. The settings below come from the etcd transport security guide.

• Separate CAs: use different CAs for etcd peers, etcd clients and the rest of the cluster, so a certificate issued for one purpose cannot join the consensus group. See the Private CA Design and Operations Guide.
• Network isolation: only the API servers should reach etcd's client port, and only members should reach the peer port.
• Encryption at rest: enable Kubernetes API encryption at rest for Secrets with an external KMS provider, so that disk images and snapshots do not hold plaintext. See the Kubernetes Secrets Security Guide.
• Backups are copies of everything: snapshots contain the full keyspace; encrypt them, restrict who can create and read them, and test restores (NIST SP 800-53 CP-9 and CP-10).
• Odd member counts: three members tolerate one failure and five tolerate two. More members add resilience against crashes but not against a compromised member.

Other coordination services (ZooKeeper, Consul and similar) raise the same questions of authentication, ACLs and quorum. See the Coordination Services Security Guide.

Identity between nodes and workloads

In a distributed system, most authentication is machine to machine. Every replica, broker, sidecar and job is a non-human identity, and the security of state depends on how those identities are issued and verified.

• Workload identity standards: SPIFFE defines open standards for identifying software systems in dynamic environments, with short-lived identity documents (SVIDs, including X.509-SVID and JWT-SVID) issued through a Workload API and trust domains that can federate. Short-lived, automatically rotated identities remove static shared secrets between nodes. See the Cloud Workload Identity Guide.
• Mutual TLS everywhere: peers authenticate each other on every connection, and authorisation decisions use the authenticated identity, not the source IP. See the mTLS Guide.
• Least privilege per role: a read replica, a leader-eligible member and an administrative client need different permissions. Avoid one shared credential for every node.
• Admission as an approval: adding a member to a consensus group changes who decides the state. Treat membership changes as privileged operations with change control and alerts.

Resilience, diversity and recovery

• Diversity: CyBOK notes identical replicas share vulnerabilities. Where the threat model justifies it, vary operating systems, regions or implementations, and stagger patching so one faulty update does not hit every member at once (NIST SP 800-53 SC-29 Heterogeneity).
• Quorum under attack: denial of service against a minority is tolerated; against a majority it halts writes. Place members across failure domains and protect peer links from saturation (SC-5).
• Split-brain protection: rely on protocols that require a strict majority, and never "force" a minority partition to accept writes without a documented recovery procedure.
• Integrity checks: verify snapshot and log integrity before restore (SI-7), and keep offline copies for recovery from malicious writes, which replication faithfully copies to every replica.

Decision guidance

• Crash-fault or Byzantine? If one organisation owns, admits and patches every member, crash-fault consensus with strong member authentication, least privilege and monitoring is usually the right answer, because a compromised member is an access control problem rather than a consensus one. Choose Byzantine fault tolerance when independent parties operate members and none should be able to rewrite shared state alone.
• Managed or self-operated? Managed Kubernetes and database services operate the consensus layer for you; your responsibility shifts to identity, network exposure, encryption keys and backups. Self-operated clusters need the full checklist below, including certificate lifecycle for peers.
• Small organisations: prefer managed control planes, keep the number of distinct clusters low, and concentrate on network exposure and credentials. A single three-member cluster with enforced TLS and tested backups is better than several unprotected ones.
• Large organisations: standardise a hardened cluster pattern (dedicated CAs, workload identity, KMS-held keys, monitoring) and enforce it through infrastructure as code, so that every new cluster starts secure rather than relying on per-team configuration.
• Open membership systems: if you run or depend on P2P networks, plan for Sybil and eclipse resistance from the start: costly or certified identities, diverse peer selection and monitoring of your nodes' view of the network.

Implementation roadmap

1. Inventory clusters: list every consensus or replicated system (etcd, databases, brokers, coordination services, ledgers), its fault model, member count and who can reach its ports.
2. Turn on transport security: enable client and peer TLS with certificate authentication, starting with non-production clusters; verify with a client that lacks a certificate that access is refused.
3. Isolate the network: restrict client and peer ports to known callers and members.
4. Replace shared secrets: move node and workload authentication to short-lived certificates or workload identities with automated rotation.
5. Encrypt at rest and protect backups: enable encryption with keys in a KMS, encrypt snapshots and test restores.
6. Monitor membership and state: alert on member additions and removals, leader changes, unusual write rates and authentication failures.
7. Roll back safely: change one member at a time, keep quorum throughout, and keep the previous certificates trusted until all members run the new ones.

Detection and measurement

• Membership changes and leader elections outside maintenance windows.
• Peer or client TLS handshake failures, which can indicate a misconfigured member or an impostor.
• Reads or writes from identities that do not normally touch a key prefix (compare ATT&CK T1552.007 Container API and T1613 Container and Resource Discovery in Kubernetes environments).
• Network anomalies on peer links: floods (T1498 Network Denial of Service) or interception attempts (T1557 Adversary-in-the-Middle).
• For P2P systems: Sybil clusters, churn spikes and peers whose view of the network diverges from others.

Common failure patterns

• Defaults left on: consensus stores deployed without TLS or client authentication because security features are off by default.
• Crash tolerance mistaken for security: assuming a five-node Raft cluster protects against a compromised node.
• Shared cluster CA: any certificate from the platform CA can join etcd or the database as a peer.
• Exposed control planes: management consoles or APIs reachable from the internet, as in the Tesla Kubernetes cryptojacking incident, where an open console exposed cloud credentials.
• Plaintext backups: snapshots copied to object storage without encryption, carrying every secret the cluster holds.
• Monoculture: every replica patched simultaneously with the same faulty build.

Framework mapping

Practitioner checklist

• Record the fault model of every replicated system, and do not rely on crash-fault consensus to tolerate compromised members.
• Enforce client and peer TLS with certificate authentication on etcd and every other consensus store.
• Use dedicated CAs for peer and client certificates of each consensus group.
• Restrict consensus ports to approved callers and members.
• Issue short-lived, automatically rotated identities to nodes and workloads, and authorise on identity, not IP.
• Encrypt state at rest with KMS-held keys, and encrypt and test backups.
• Treat membership changes as privileged operations with alerts.
• Spread members across failure domains and stagger patching to avoid monoculture failures.

Standards and references

• CyBOK: Distributed Systems Security Knowledge Area v1.0.1
• NIST: SP 800-207 Zero Trust Architecture
• etcd: Transport security model
• Raft: The Raft Consensus Algorithm
• SPIFFE: Overview
• NIST: SP 800-53 Rev. 5
• NIST: Cybersecurity Framework 2.0

Related NHI Mgmt Group resources: Zero Trust Identity Guide · Coordination Services Security Guide · Kubernetes Security Guide · Cloud Workload Identity Guide