Authenticated Encryption and Block Cipher Modes Guide: Choosing AEAD Safely and Avoiding Nonce Reuse

Encryption that only hides data, without proving it has not been changed, is one of the most common and most damaging cryptographic mistakes in production software. Authenticated encryption with associated data (AEAD) is a single primitive that gives confidentiality and integrity together: it either returns the original plaintext or refuses to decrypt. This guide explains what AEAD is, how the main block cipher modes and AEAD schemes work, where real systems have failed, and how to choose and operate AES-GCM, ChaCha20-Poly1305 and nonce misuse resistant options. It is written for developers, security architects and reviewers who make or approve cryptographic choices, and it goes deeper than the Symmetric vs Asymmetric Encryption Guide.

Key takeaways

• Use an AEAD, not a bare mode. AES-GCM, ChaCha20-Poly1305 and AES-CCM authenticate the ciphertext and any associated data; CBC or CTR on their own do not.
• Nonce uniqueness is the whole contract. RFC 5116 requires a distinct nonce for every encryption under a key; reusing one with GCM exposes the XOR of plaintexts and lets an attacker forge messages.
• Every scheme has volume limits. RFC 9325 recommends rekeying AES-GCM in TLS before 2^24.5 records, and RFC 5116 caps a single GCM message at 2^36 minus 31 octets.
• Never release plaintext before the tag verifies. Streaming unverified output and non-constant-time tag comparison turn a sound algorithm into an oracle.
• Prefer a high-level library and fixed protocols. TLS 1.3 permits only AEAD record protection and builds nonces for you, which removes the most common implementation errors.

What it is

A block cipher such as AES transforms one fixed-size block (128 bits for AES) under a key. A mode of operation describes how to use it on messages longer than one block. NIST SP 800-38A specifies five confidentiality-only modes: ECB, CBC, CFB, OFB and CTR. None of them detects tampering. An attacker who cannot read a CTR ciphertext can still flip chosen bits in the plaintext, and an attacker who can submit CBC ciphertexts to a system that reveals padding errors can often decrypt them.

Authenticated encryption closes that gap. RFC 5116 defines the AEAD interface that most modern libraries follow:

• Encrypt inputs: a secret key K, a nonce N, the plaintext P and associated data A. A is authenticated but not encrypted (for example a record header, a database row ID or a protocol version).
• Encrypt output: a ciphertext C, which for the AES-GCM and AES-CCM algorithms in RFC 5116 is exactly 16 octets longer than P because it carries the authentication tag.
• Decrypt inputs and output: K, N, A and C go in; either P comes out or a single FAIL indication. There is no partial output.

That last property is the point. A correct AEAD implementation gives the caller nothing to act on unless the ciphertext, the nonce and the associated data are exactly what the key holder produced.

Confidentiality modes compared

How it works

AES-GCM

GCM, specified in NIST SP 800-38D, combines CTR mode encryption with GHASH, a polynomial hash over a binary field keyed by a value derived from the AES key. The tag is computed over the associated data, the ciphertext and their lengths, then masked with an encrypted counter block derived from the nonce. RFC 5116 registers AEAD_AES_128_GCM and AEAD_AES_256_GCM with 16-octet keys and 32-octet keys respectively, a 12-octet nonce, a 16-octet tag, a maximum plaintext of 2^36 minus 31 octets and associated data up to 2^61 minus 1 octets. SP 800-38D also defines GMAC, which authenticates data without encrypting it. NIST recorded a decision on 6 March 2024 to revise SP 800-38D; check the publication page for the current text before relying on details such as short tag lengths.

ChaCha20-Poly1305

RFC 8439 (Informational, IRTF Crypto Forum Research Group, obsoleting RFC 7539) pairs the ChaCha20 stream cipher with the Poly1305 authenticator. It uses a 256-bit key, a 96-bit nonce and a 32-bit block counter. ChaCha20 block 0 produces a one-time Poly1305 key; encryption starts at block 1; Poly1305 then authenticates the associated data, the ciphertext, padding and both lengths into a 128-bit tag. The 32-bit counter limits one invocation to about 256 GB, which the RFC notes may be too small for some file or disk uses. The RFC states that ChaCha20 is roughly three times as fast as AES in software on platforms without AES hardware and avoids the cache-timing issues of many table-based AES implementations, which is why it is common on mobile and embedded devices.

Nonce construction

RFC 5116 recommends a 12-octet nonce built from a fixed field (identifying the sender or device) and a counter field. Deterministic counters are safest when state is reliable. Random 96-bit nonces are acceptable only while the number of messages under one key stays far below the birthday bound: NIST SP 800-38D limits the number of random-IV invocations per key to 2^32. Protocols remove the choice from developers: TLS 1.3 derives each record nonce from a per-connection IV and the record sequence number, and RFC 9325 (BCP 195) updates RFC 5288 so that TLS 1.2 GCM implementations put the 64-bit sequence number in the explicit nonce.

Nonce misuse resistance

AES-GCM-SIV (RFC 8452, Informational, April 2019) derives per-nonce keys and computes a synthetic IV from the message. If a nonce repeats, an observer learns only whether two messages under the same key and nonce were identical. The RFC still recommends random nonces, notes that encryption needs two passes, and states that plaintext must not be released until the tag is verified.

Operating limits to plan for

Why it matters

Unauthenticated encryption has produced a long line of exploitable flaws. RFC 7457 records how the MAC-then-encrypt design used by TLS up to 1.2 enabled padding oracle attacks: Lucky Thirteen (CVE-2013-0169) used timing differences to decrypt CBC records, POODLE (CVE-2014-3566) broke SSL 3.0 CBC with no available mitigation, and BEAST (CVE-2011-3389) exploited predictable IVs in TLS 1.0 CBC. The fixes it lists are authenticated encryption such as AES-GCM or the encrypt-then-MAC extension in RFC 7366. TLS 1.3 went further and kept only AEAD algorithms for record protection.

The same failure classes appear outside TLS: application-level token encryption that uses CBC without a MAC, cookies encrypted with CTR that can be bit-flipped to change a user ID, and services that reuse a GCM nonce after a restart because the counter was held in memory. Adversaries who can observe or alter traffic (MITRE ATT&CK T1557 Adversary-in-the-Middle) or force weaker settings (T1600 Weaken Encryption) benefit directly from these mistakes. For machine identities the stakes are concrete: API tokens, session cookies and encrypted secrets are frequently protected with a symmetric key held by a service, so a malleable ciphertext can become a forged credential. See the Cryptographic Protocol Failures and Misuse Guide for the wider bug class.

Choosing a scheme

• Servers and clients with AES hardware: AES-128-GCM or AES-256-GCM, which are the suites RFC 9325 recommends for TLS 1.2 and are mandatory or common in TLS 1.3.
• Mobile, embedded or mixed fleets without AES acceleration: ChaCha20-Poly1305, offered alongside AES-GCM so each side can negotiate the fastest safe option.
• Nonce state you cannot guarantee (stateless workers, many independent encryptors under one key, restored virtual machine snapshots): AES-GCM-SIV, or derive a fresh key per message or per writer so nonce collisions cannot cross keys.
• Disk and block storage: AEAD tags do not fit fixed-size sectors, so storage encryption usually uses a length-preserving mode such as XTS with integrity handled elsewhere. See the Database and Storage Encryption Guide.
• Regulated environments: AES-GCM and AES-CCM are NIST-specified modes; confirm the module and mode are covered by your validated cryptographic module before choosing ChaCha20-Poly1305. See the FIPS 140-3 and Cryptographic Module Validation Guide.

Implementing and reviewing safely

1. Inventory first. Find every place that calls a cipher directly: grep for CBC, CTR, ECB and raw AES calls, and record them in a cryptographic inventory. See the Cryptographic Inventory and CBOM Guide.
2. Replace with a high-level AEAD API that generates nonces and returns a single opaque ciphertext, rather than one that asks callers for an IV.
3. Bind context in the associated data: tenant ID, record ID, key version and purpose, so a ciphertext cannot be moved to another row or reused as a different token type.
4. Version the format (a leading byte or key ID) so you can rotate algorithms and keys later. See the Crypto-Agility Guide.
5. Test with known-answer vectors from RFC 8439 and the NIST CAVP files, plus negative tests: flipped ciphertext bits, altered associated data and truncated tags must all fail.
6. Roll out with dual decrypt: new writes use the AEAD format, reads accept both formats, a background job re-encrypts old data, then the legacy path is removed. Rollback is simply re-enabling the legacy read path while keeping the new writes.

Detection and measurement

• Log decryption failures by key ID and caller. A spike in tag failures is either corruption or someone probing; RFC 9325 sets a DTLS 1.2 integrity limit of 2^28 failed records for AES-GCM, after which the connection should close.
• Count encryptions per key and alert well before the limits in the table above.
• Metrics: percentage of cryptographic call sites using an approved AEAD; number of remaining CBC or ECB call sites; TLS endpoints still negotiating CBC suites without encrypt-then-MAC; keys exceeding 50% of their usage limit; mean time to rotate a key after a limit alert.

Framework mapping

Common misconceptions

• "AES-256 means it is secure." The key size says nothing about the mode. AES-256 in ECB or unauthenticated CBC is still broken in the ways above.
• "Encryption implies integrity." CTR and CBC ciphertexts can be altered in predictable ways. Only an authenticator checked before use gives integrity.
• "A random nonce is always safe." Random 96-bit nonces collide after enough messages; SP 800-38D caps random-IV use at 2^32 invocations per key. High-volume services need counters, per-message keys or SIV.
• "Adding a MAC fixes CBC." Only if it is encrypt-then-MAC with separate keys and constant-time checking. MAC-then-encrypt is exactly what Lucky Thirteen exploited.
• "Truncating the tag saves space safely." RFC 8439 says Poly1305 tags must not be truncated, and short GCM tags weaken forgery resistance. Keep 128-bit tags.
• "GCM-SIV lets us ignore nonces." It limits the damage of reuse; it does not remove the recommendation to use unique nonces.

Related concepts

• Key exchange: AEAD keys in TLS and SSH come from an ephemeral Diffie-Hellman exchange. See the Key Exchange and Diffie-Hellman Guide.
• Key management and envelope encryption: data keys wrapped by a KMS key, with rotation tied to usage limits. See the Cryptographic Key Management Guide.
• TLS record protection: see the How TLS Works Guide and the TLS Server Configuration Baseline Guide.
• Post-quantum: symmetric AEAD is not broken by known quantum algorithms in the way RSA and elliptic curve key exchange are; 256-bit keys are a common choice for long-lived data. See the Post-Quantum Cryptography Migration Guide.

Practitioner checklist

• Inventory every direct cipher call and flag ECB, CBC and CTR used without an authenticator.
• Standardise on AES-GCM and ChaCha20-Poly1305 through a vetted high-level library.
• Define the nonce strategy per key (counter, random within limits, or SIV) and document it.
• Put tenant, record and purpose identifiers into associated data.
• Keep 128-bit tags and constant-time tag comparison; never release unverified plaintext.
• Track encryptions per key and rotate before the limits in RFC 5116, SP 800-38D and RFC 9325.
• Disable TLS CBC suites, or require encrypt-then-MAC where they must remain.
• Add negative tests for tampered ciphertext, associated data and tags to CI.
• Version ciphertext formats so algorithms and keys can be changed without data loss.

Standards and references

• IETF: RFC 5116 An Interface and Algorithms for Authenticated Encryption
• NIST: SP 800-38D Galois/Counter Mode (GCM) and GMAC
• NIST: SP 800-38A Block Cipher Modes of Operation
• IRTF: RFC 8439 ChaCha20 and Poly1305 for IETF Protocols
• IRTF: RFC 8452 AES-GCM-SIV Nonce Misuse-Resistant Authenticated Encryption
• IETF: RFC 9325 Recommendations for Secure Use of TLS and DTLS (BCP 195)
• IETF: RFC 7457 Summarizing Known Attacks on TLS and DTLS
• IETF: RFC 8446 The Transport Layer Security Protocol Version 1.3

Related NHI Mgmt Group resources: Symmetric vs Asymmetric Encryption Guide · Cryptographic Protocol Failures and Misuse Guide · Cryptographic Key Management Guide · Encryption Guide