Most application teams that "use KMS" still get envelope encryption wrong in the details: plaintext data keys left in memory or logs, no binding between ciphertext and context, one data key reused across tenants, and decrypt rights granted to every workload. Envelope encryption is the pattern behind almost every cloud encryption service: data is encrypted locally with a data encryption key (DEK), and the DEK is encrypted (wrapped) by a key encryption key (KEK) that never leaves a key management service or HSM. This guide covers the application side of that pattern: choosing between raw KMS calls and encryption SDKs, binding ciphertext to context, data key caching, rotation and re-wrapping, multi-region and multi-tenant designs, and the identity controls that decide who can decrypt. It is for developers, security architects and platform teams building field, object or message encryption on AWS KMS, Google Cloud KMS, Azure Key Vault or HashiCorp Vault. For the wider encryption picture see the Encryption Guide.
Key takeaways
- The KMS permission is the real access control. Whoever can call decrypt or unwrap on the KEK can read the data, so the design of key policies and workload identities matters more than the cipher.
- Prefer an encryption SDK over hand-rolled KMS calls. Libraries such as the AWS Encryption SDK and Tink's KMS envelope AEAD handle DEK generation, nonce management, the message format and key commitment for you.
- Always bind ciphertext to context. Use an encryption context or associated data (tenant, record ID, purpose) so a ciphertext copied to another record or tenant fails to decrypt, and use it in key policy conditions.
- Use a fresh DEK per object or write, and cache only with limits. Google's guidance is a new DEK each time you write data; the AWS Encryption SDK generates one per encryption by default and treats caching as an optimisation to use cautiously.
- Rotate the KEK, re-wrap the DEKs, and keep keys away from the data. KEK rotation does not re-encrypt data; plan re-wrapping explicitly, and never store an unwrapped key next to its ciphertext.
How envelope encryption works
A KMS is built to protect a small number of keys and perform a small number of operations per second, not to encrypt gigabytes. Envelope encryption splits the work:
- The application asks the KMS for a new DEK (for example AWS KMS
GenerateDataKeywithKeySpecAES_256) or generates one locally with a cryptographically secure generator. - The KMS returns the DEK in two forms: plaintext, and wrapped under the KEK (AWS calls these
PlaintextandCiphertextBlob). - The application encrypts the data locally with an authenticated cipher, typically AES-256-GCM, passing context as associated data.
- It stores the wrapped DEK alongside the ciphertext and erases the plaintext DEK from memory as soon as possible, as the AWS API reference instructs.
- To decrypt, the application sends the wrapped DEK (and the same context) to the KMS, receives the plaintext DEK, decrypts locally, and erases the DEK again.
Benefits follow directly: bulk data never crosses the network to the KMS, the KEK never leaves the HSM boundary, each object can have its own key, and access can be revoked by changing who may unwrap. Google's Cloud KMS guidance adds four rules worth adopting as policy: generate DEKs locally, keep them encrypted at rest and close to the data, never store a plaintext DEK, and do not reuse a DEK across different users' data. See the Symmetric vs Asymmetric Encryption Guide for the primitives involved.
Key hierarchy terms
| Term | What it is | Where it lives | Typical lifetime |
|---|---|---|---|
| Root or HSM key | Provider key protecting KMS key material | Provider HSMs | Managed by provider |
| KEK (KMS key, CMK, Key Vault key, Vault transit key) | Wraps and unwraps DEKs | KMS or HSM, non-exportable | Years, with versioned rotation |
| Intermediate or branch key | Optional middle tier that wraps many DEKs to cut KMS calls (for example the AWS KMS Hierarchical keyring) | Wrapped by KEK, cached in memory | Hours to days |
| DEK | Encrypts the data | Wrapped next to the ciphertext; plaintext only in memory | One object, write or message |
Integration options compared
| Option | How it works | Strengths | Watch for |
|---|---|---|---|
| Direct KMS encrypt | Send plaintext to the KMS Encrypt API | Simplest; no key ever in app memory | Small payloads only (AWS KMS accepts up to 4 KB), a KMS call per operation, latency |
| Hand-rolled envelope | GenerateDataKey plus your own AES-GCM code and storage format | Full control | Nonce handling, format versioning, context binding and key erasure all become your bugs |
| AWS Encryption SDK | Keyrings wrap DEKs; portable message carries encrypted DEKs, algorithm ID, context and signature | Key commitment, signing, multiple wrapping keys, caching options | Choose commitment policy and keyring deliberately |
Tink KmsEnvelopeAead | Random DEK per encryption, wrapped by a remote KEK, concatenated with ciphertext | Misuse-resistant API, multi-language | One KMS RPC on every encrypt and decrypt; AEAD only. Alternative: a KMS-encrypted keyset |
Azure Key Vault wrapKey/unwrapKey | App wraps its DEK with a vault key (needs keys/wrapKey) | Works with HSM-backed keys | Use RSA-OAEP-256; Microsoft marks RSA-OAEP (SHA-1) and RSA1_5 as not recommended |
| HashiCorp Vault Transit | POST /transit/datakey/wrapped/:name or /plaintext/; rewrap for rotation | ACLs can separate who gets plaintext; rewrap never returns plaintext | Vault availability becomes a data availability dependency |
| Database or storage native encryption | Provider does envelope encryption for you | No code | Protects against disk theft, not against a caller with query access. See the Database and Storage Encryption Guide |
Decision guidance
- Secrets and short values under a few KB: direct KMS encrypt or a secrets manager is fine.
- Fields, documents and messages in an application: use an encryption SDK, not a hand-rolled envelope. See the Crypto Libraries Guide.
- High-volume workloads: use an intermediate key tier (Hierarchical keyring) or bounded data key caching rather than calling the KMS for every record.
- Multi-cloud or customer-held keys: abstract the wrap call behind a keyring or KMS client interface so the KEK provider can change; see the BYOK, HYOK and External Key Management Guide.
- Small organisations: start with provider-managed storage encryption plus one SDK pattern for the few truly sensitive fields; large organisations should publish a single approved envelope library and message format.
Binding ciphertext to context
An authenticated cipher detects tampering, but without context it cannot tell whether a valid ciphertext belongs to this record. Encryption context (AWS) or associated data (Tink, AES-GCM generally) solves that.
- What to put in it: stable, non-secret identifiers such as tenant ID, table and record ID, object path or purpose. AWS documents that encryption context is not secret and appears in plaintext in CloudTrail logs, so never include personal data.
- Exact match on decrypt: AWS KMS requires the same, case-sensitive context to decrypt, otherwise it returns
InvalidCiphertextException. If a file whose path is used as context is renamed, decryption fails, which is the point. - Policy conditions: AWS key policies and IAM can require context with the
kms:EncryptionContext:<key>andkms:EncryptionContextKeyscondition keys, and grants supportEncryptionContextEqualsandEncryptionContextSubset. A role allowed to decrypt only whentenantequals its own tenant cannot unwrap another tenant's DEKs even if it holds them. - Verify on the way out: the AWS Encryption SDK stores context in the message header; outside the CLI your code should check the returned context contains what you expect. Newer SDK versions offer a required encryption context CMM that refuses to encrypt without it.
- Key commitment and signatures: the AWS Encryption SDK default suite uses AES-GCM with HKDF, key commitment and an ECDSA signature. Commitment guarantees a ciphertext decrypts to only one plaintext; the signature stops a caller with only decrypt rights from forging new ciphertexts. AWS notes signing adds nothing with keyrings that cannot separate encryptors from decryptors, such as Raw AES and the Hierarchical keyring.
Performance, caching and nonce limits
Every unwrap is a KMS call with latency, cost and a request quota. There are three ways to cut the number of calls, each a security trade-off:
| Approach | Effect | Risk | Guard rails |
|---|---|---|---|
| Data key caching (AWS Encryption SDK caching CMM) | Reuses a DEK across messages | More data under one key; a memory compromise exposes more | Set the caching CMM's security thresholds to the minimum that meets performance goals; cache per tenant; use only when needed |
| Intermediate key tier (Hierarchical keyring) | A cached branch key derives per-message keys locally | Branch key in memory; ciphertext readable only with the same keyring type | Short cache TTL, branch key per tenant, rotation schedule |
| KMS-encrypted keyset (Tink) | Keyset unwrapped once at start-up | Long-lived keys in process memory | Restart and rotate regularly; restrict who can read the encrypted keyset |
When the same key encrypts many messages, nonce limits matter. AES-GCM with random 96-bit nonces should not exceed 232 encryptions under one key (NIST SP 800-38D), and OWASP ASVS 5.0 requirement V11.3.4 says nonces must not be used for more than one key and data-element pair. A fresh DEK per object sidesteps the problem; caching reintroduces it, which is why limits on messages per cached key are not optional. See the Authenticated Encryption and Block Cipher Modes Guide.
Rotation, re-wrapping and migration
- KEK rotation is not data re-encryption: enabling automatic rotation on a KMS key creates new key material for future wraps while old material stays available to unwrap existing DEKs. Data stays readable and the old DEKs stay as they were.
- Re-wrap when you must retire a KEK version: unwrap each DEK and wrap it under the new version. Vault's
/transit/rewrap/:namedoes this without returning plaintext, so it can be delegated to a batch job. After re-wrapping, Vault'smin_decryption_versionstops old ciphertext versions from being decrypted. - Re-encrypt only when the DEK itself may be exposed: a compromised application host that held plaintext DEKs needs data re-encryption, not just re-wrapping.
- Version the format: store algorithm, KEK identifier and version with each record so you can migrate algorithms. ASVS V11.2.2 asks for crypto agility, including the ability to replace keys and re-encrypt data.
- Commitment migration (AWS Encryption SDK): move readers to
ForbidEncryptAllowDecryptorRequireEncryptAllowDecryptfirst, then writers, and only switch toRequireEncryptRequireDecrypt(the 2.0.x default) once all old ciphertexts are gone. - Deletion is irreversible: scheduling a KEK for deletion destroys every DEK it wraps. Disable first, watch for failed calls, and keep the deletion waiting period. Vault defaults
deletion_allowedto false for the same reason.
Multi-tenant and multi-region designs
| Pattern | Isolation | Cost and complexity | Use when |
|---|---|---|---|
| One KEK, context per tenant | Policy conditions on tenant context | Low | Many small tenants, shared service |
| KEK per tenant | Separate key policies, per-tenant revocation and audit | Medium (key count, quotas) | Regulated or enterprise tenants, customer-managed key offers |
| Branch key per tenant under a shared KEK | Cryptographic separation with fewer KMS keys | Medium | High-volume multi-tenant SaaS |
| Multiple wrapping keys per DEK | Same DEK wrapped under keys in two regions or providers | Higher storage, more principals | Disaster recovery and cross-region reads |
The AWS Encryption SDK encrypts the data key under every wrapping key you specify and stores all the copies in the message, so a ciphertext stays decryptable in a second region if the first is unavailable. Each extra wrapping key is another principal set that can decrypt: review it like any other access grant.
Where identity fits
Envelope encryption turns data access into a question of which non-human identities may call Decrypt, unwrapKey or datakey/plaintext. Apply least privilege to those calls:
- Separate encrypt and decrypt identities: ingest services need
GenerateDataKeyorEncrypt, reporting jobs needDecrypt; few things need both. Vault can expose onlydatakey/wrappedto untrusted producers. - Use workload identity, not static keys: IAM roles, managed identities or Kubernetes service account federation instead of long-lived access keys. The Codefinger S3 Ransomware campaign used stolen long-term AWS keys to re-encrypt buckets with attacker-held SSE-C keys, a reminder that whoever controls the encryption call controls the data.
- Keep key custody separate from data custody: in the GoTo Breach attackers took encrypted backups together with the key for some of them. A wrapped DEK is safe beside its data; an unwrapped key, or credentials that can unwrap, are not.
- Use enclaves for the most sensitive decrypts: AWS KMS can return a DEK encrypted to a Nitro Enclave's attested public key (
Recipientparameter withRSAES_OAEP_SHA_256), so the plaintext DEK is never visible to the parent instance.
See the AWS KMS Security Guide, the Cloud KMS and Cloud HSM Guide and, for container platforms, the Kubernetes Secrets Security Guide.
Implementing envelope encryption
- Classify the data: decide which fields or objects need application-level encryption (ASVS V14.1.1 asks for classification into protection levels).
- Pick one library and one message format: publish it as the approved pattern with code samples; ban hand-rolled AES code in reviews.
- Design keys and context: decide KEK granularity (per environment, service or tenant) and a context schema, and encode the context rule in key policies.
- Separate identities: create distinct roles for encrypt, decrypt and key administration; no human holds routine decrypt rights in production.
- Pilot on one service: encrypt new writes, read both old plaintext and new ciphertext, and measure KMS latency and quota use before adding caching.
- Backfill: re-encrypt historic data with a batch job using a dedicated identity, then remove the plaintext read path.
- Rollback plan: keep the KEK enabled and the dual-read path for one release cycle; never schedule key deletion as part of a rollback.
Detection and measurement
- Log every KMS call: CloudTrail, Cloud Audit Logs, Azure Key Vault diagnostic logs or Vault audit devices. Alert on decrypt calls from unexpected principals, on context values a principal should never use, and on bursts of decrypts.
- Watch for failures: spikes in
InvalidCiphertextExceptionor access denied on decrypt can mean tampering, copied ciphertext or misconfigured context. - Key lifecycle events: key policy changes, disable, schedule deletion and new grants should page the key owners.
Metrics:
- Share of sensitive fields encrypted through the approved library.
- Number of principals with decrypt rights per KEK (falling).
- Percentage of KMS calls that carry an encryption context.
- Maximum messages or bytes per cached DEK versus the configured limit.
- Time to re-wrap all DEKs after a KEK version is retired.
Common failure patterns
- Storing the plaintext DEK: writing the
Plaintextfield to the database, a log line or a debug dump instead of the wrapped copy. - No context: ciphertexts can be swapped between rows or tenants and still decrypt.
- Sensitive data in context: email addresses or account numbers end up in plaintext audit logs.
- Unbounded caching: one DEK reused for millions of records across tenants, approaching nonce limits and maximising the blast radius.
- Everyone can decrypt: a wildcard
kms:DecryptonResource: "*"in a shared role makes encryption a formality. - Deleting a KEK to "clean up": every object wrapped by it becomes unrecoverable, including backups.
Framework mapping
| Control area | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | OWASP ASVS 5.0 | Other |
|---|---|---|---|---|
| Key hierarchy and lifecycle | SC-12, SC-12(1) | PR.DS-01 | V11.1.1, V11.1.2, V13.3.4 | ISO/IEC 27001:2022 A.8.24 |
| Authenticated encryption, nonces | SC-13, SC-28(1) | PR.DS-01 | V11.3.2, V11.3.3, V11.3.4 | NIST SP 800-38D |
| KEK in KMS or HSM | SC-12 | PR.DS-01 | V13.3.1, V13.3.3 | ATT&CK T1552.004 Private Keys |
| Decrypt rights and workload identity | AC-3, AC-6 | PR.AA-05 | V13.3.2 | ATT&CK T1078.004 Cloud Accounts |
| Audit of key use | AU-12 | DE.CM-09 | n/a | ATT&CK T1530, T1486 |
Practitioner checklist
- One approved envelope library and message format is published, and hand-rolled AES code is rejected in review.
- Every encryption call passes a non-secret context, and key policies require it.
- Plaintext DEKs exist only in memory and are erased after use; only wrapped DEKs are stored.
- A fresh DEK is used per object or write, or caching runs with explicit, documented security thresholds.
- Encrypt, decrypt and key administration are separate workload identities with no static access keys.
- KEK rotation is enabled, and a tested re-wrap job exists for retiring key versions.
- Key deletion requires disable-first, a waiting period and two-person approval.
- All KMS calls are logged and decrypts by unexpected principals or contexts raise alerts.
- Multi-region wrapping keys and their principals are reviewed like any other access grant.
Standards and references
- AWS: KMS GenerateDataKey API reference
- AWS: KMS encryption context
- AWS: Encryption SDK concepts
- Google Cloud: Envelope encryption
- Google: Tink client-side encryption with a KMS
- Microsoft Learn: Key Vault Wrap Key
- HashiCorp: Vault Transit secrets engine API
- OWASP: Application Security Verification Standard 5.0
Related NHI Mgmt Group resources: Cryptographic Key Management Guide · Secrets Management Guide · HSM Deployment and Operations Guide · Cryptographic Key Management Policy Template