Secrets Encryption on AWS with Kubernetes and Terraform

Adam Cheriki. Entro Security

Blog Article by Entro Security

Modern cloud infrastructures leverage multiple services and tools, requiring secure access to sensitive information. Secrets management, including the management of non-human identities (NHIs), becomes a complex, multi-layered challenge in AWS environments using Kubernetes for orchestration and Terraform for infrastructure-as-code. Inadequate secrets encryption or misconfiguration at any level can result in costly security breaches.

This technical guide shares practical steps to properly implement secrets encryption in AWS while working with Kubernetes and Terraform. We’ll begin with the fundamentals: what constitutes a secret and how basic secret key encryption works. From there, we’ll explore AWS’s native tools—KMS and Secrets Manager—and how they form the foundation of a robust strategy for both Kubernetes secrets encryption and Terraform secrets encryption.

Understanding the basics

In information security, secrets refer to confidential digital credentials used to authenticate systems or access protected resources. These credentials may include API keys, database credentials, encryption keys, OAuth tokens, SSH keys, etc. Many of these secrets are associated with non-human identities—service accounts, CI/CD tools, applications, and other entities that require programmatic access to systems and data. The increasing number of these NHIs in automated environments significantly increases the complexity of secrets management.

When we talk about encrypting these secrets, we essentially encode this information so that only authorized parties (human or non-human) can access it. This process involves two main steps:

  • Encryption: Converting the plaintext secret into unreadable ciphertext

  • Decryption: Reverting the ciphertext to its original, readable form

In AWS environments, this encryption can be applied to data at rest (e.g., in S3 buckets), data in transit (using SSL/TLS), and application secrets (using AWS Secrets Manager).

The AWS approach: KMS and Secrets Manager

AWS provides two key services for secrets storage and encryption:

  • Key Management Service (KMS): This creates and controls the encryption keys.

  • Secrets Manager: This stores and manages the secrets themselves.

AWS Key Management Service (KMS)

AWS KMS is a managed service used for creating and controlling cryptographic keys for encrypting data. With KMS, you can manage keys across various AWS services and applications in a centralized way. The primary functions of KMS include:

  • Key creation and management: Users can create, rotate, disable, and delete keys.

  • Encryption and decryption: KMS can encrypt data directly or generate data keys for client-side encryption.

  • Access control: KMS integrates with AWS Identity and Access Management (IAM) to control key usage permissions.

  • Audit and compliance: It logs key usage through AWS CloudTrail and helps organizations maintain compliance and security audits.

AWS Secrets Manager

AWS Secrets Manager is designed to store, manage, and retrieve secrets securely. Its key features include:

  • Secret storage: Secrets Manager encrypts secrets at rest and in transit to ensure their confidentiality across the infrastructure.

  • Automatic rotation: It can automatically rotate secrets for supported services, which is particularly useful for long-lived secrets often associated with service accounts and other non-human identities.

  • Access management: Similar to KMS, Secrets Manager uses IAM policies to ensure only authorized access to secrets.

  • Audit capabilities: It integrates with AWS CloudTrail to track access and changes to secrets

Integration of AWS KMS and AWS Secrets Manager

The integration between AWS KMS and AWS Secrets Manager is primarily centered around the concept of envelope encryption. This process involves steps to ensure that secrets are securely stored and retrieved.

Step 1: Secret creation

When a new secret is created in Secrets Manager, it first generates a unique data key using AWS KMS. This data key is a 256-bit Advanced Encryption Standard (AES) symmetric key.

Step 2: Encryption process

The data key is encrypted using a KMS key managed within KMS. This encrypted data key is stored in the secret’s metadata. The plaintext data key is then used to encrypt the secret value (e.g., a password or API key). After encryption, the plaintext data key is removed from memory to prevent exposure.

Step 3: Secret retrieval

Secrets Manager retrieves the encrypted data key from the secret’s metadata. It then calls AWS KMS to decrypt the data key using the KMS key. Finally, the decrypted data key is used to decrypt the secret value, which is returned in plaintext to the application.

This approach ensures that the sensitive data is never exposed in plaintext during storage or retrieval, as AWS KMS securely manages the encryption and decryption processes.

These AWS services provide a strong foundation for secrets management within the AWS environment. In practice, many organizations build upon these basics with a sophisticated platform-agnostic secrets management tool to create comprehensive strategies that span their entire infrastructure.

Kubernetes integration with AWS KMS for secrets encryption

Kubernetes has become a popular platform for deploying and managing containerized applications, which involve many non-human identities interacting with each other and with external services. However, the platform’s default approach to secrets management is not strong enough.

The challenge of Kubernetes secrets encryption

Kubernetes secrets management is a critical aspect of container security. Kubernetes secrets are stored as base64-encoded strings in etcd, its distributed key-value store. This does facilitate basic obfuscation, but more is needed to ensure strong encryption. Anyone accessing the etcd database can decode and read these secrets.

Security teams have begun comparing HashiCorp Vault vs Kubernetes secrets, as Vault offers advanced features like dynamic secrets and fine-grained access controls beyond Kubernetes’ native capabilities. Even then, comprehensive platform-agnostic secrets management solutions are increasingly becoming popular.

EKS secrets encryption

Amazon EKS (Elastic Kubernetes Service) offers a “secrets encryption” feature that integrates AWS KMS with your Kubernetes cluster. This lets you encrypt secrets at rest using your own KMS keys.

How EKS secrets encryption works

When you enable secrets encryption for your EKS cluster, here’s what happens behind the scenes:

  • You specify a KMS key to use for encryption.

  • EKS configures the Kubernetes API server to use this KMS key to encrypt all new secrets.

  • When a secret is created or updated, the Kubernetes API server calls AWS KMS to encrypt the secret before storing it in etcd.

  • When a secret is read, the API server calls AWS KMS to decrypt the secret before returning it to the authorized user or application.

This process ensures that your secrets are always encrypted at rest in etcd, adding an additional layer of security to your Kubernetes workloads.

Enabling secrets encryption on an existing EKS cluster

To enable secrets encryption on an existing EKS cluster, you have four primary methods. Each method has its own advantages and use cases:

1. Using eksctl

The eksctl command-line tool provides a straightforward method for enabling secrets encryption:

eksctl utils enable-secrets-encryption \
–cluster my-cluster \
–key-arn arn:aws:kms:region-code:account:key/key

This method is ideal for users who are comfortable with command-line interfaces and want a quick, scriptable solution.

2. Using a YAML Configuration File

For those who prefer a declarative approach, you can use a YAML configuration file with eksctl:

Create a file named kms-cluster.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
name: my-cluster
region: region-code

secretsEncryption:
keyARN: arn:aws:kms:region-code:account:key/key

Apply the configuration:

eksctl utils enable-secrets-encryption -f kms-cluster.yaml

This method is useful for version control and maintaining consistency across multiple clusters.

3. Using the AWS Management Console

For those who prefer a graphical interface:

  • Open the Amazon EKS console.

  • Select your cluster.

  • In the “Overview” tab, find the “Secrets encryption” section and choose “Enable”.

  • Select your KMS key from the dropdown and confirm.

As a super user-friendly method, this doesn’t need you to have any command-line knowledge. This means it’s suitable for quick, one-off changes.

4. Using the AWS CLI

For users who prefer the AWS CLI:

aws eks associate-encryption-config \
–cluster-name my-cluster \
–encryption-config ‘[{“resources”:[“secrets”],”provider”:{“keyArn”:”arn:aws:kms:region-code:account:key/key”}}]’

This method offers fine-grained control and is excellent for automation scripts interacting with multiple AWS services.

You can find more details about the process here.

With Kubernetes secrets now securely managed using AWS KMS, let’s focus on another critical aspect of modern infrastructure management: Infrastructure as Code (IaC). Terraform is a widely-used IaC tool, and integrating it with AWS Secrets Manager can significantly enhance the security of your infrastructure provisioning process.

Managing Terraform secrets with AWS Secrets Manager

When working with Terraform, securely managing secrets like database credentials is necessary because hardcoding these secrets in your Terraform files exposes them to anyone with access to your version control system.

AWS Secrets Manager helps keep your secrets separate from your Terraform code while still using them to provision and manage your infrastructure.

Here’s how to implement AWS Secrets Manager in your Terraform configurations:

1. First, store your secret in AWS Secrets Manager:

  • Log into the AWS Secrets Manager UI.

  • Click “Store a new secret.”

  • Enter your secrets in JSON format (e.g., database username and password).

  • Assign a unique name to the secret (e.g., “prod/db/credentials”).

  • Click “Next” and “Store” to save.

2. Next, in your Terraform code, use the aws_secretsmanager_secret_version data source to access the secret:

data “aws_secretsmanager_secret_version” “db_credentials” {
secret_id = “prod/db/credentials”
}

3. After retrieving the secret, you must parse its contents. AWS Secrets Manager stores secrets as strings, so if you’ve stored your secret in JSON format, you’ll need to decode it. Terraform provides the jsondecode function for this purpose. Here’s how to use it:

locals {
db_credentials = jsondecode(
data.aws_secretsmanager_secret_version.db_credentials.secret_string
)
}

This step converts the JSON string into a Terraform object so that you can access individual fields of your secret.

4. Now that you’ve parsed the secret, you can use its values in your resource configurations. For example, if you’re setting up a database instance:

resource “aws_db_instance” “production” {
engine = “mysql”
engine_version = “5.7”
instance_class = “db.t3.micro”
name = “production_db”

username = local.db_credentials.username
password = local.db_credentials.password

# Other configuration…
}

This approach helps keep your sensitive information secure and separate from your Terraform code. The secret is stored safely in AWS Secrets Manager and only accessed when needed during Terraform operations. Check out the full how-to guide here.

Remember to configure appropriate IAM permissions for your Terraform execution environment to access the secrets in AWS Secrets Manager. With appropriate IAM policies in place, you can ensure that only authorized users and systems can retrieve secrets.

Why secrets managers fall short in complex, multi-environment infrastructures

It’s great that cloud providers like AWS offer native secrets management services, but these often fall short in heterogeneous environments, especially when they have thousands of NHIs that require access across multiple cloud platforms and services. Here’s where traditional secrets managers struggle:

  • Limited cross-platform compatibility: Native cloud solutions often work best within their own ecosystems, not in multi-cloud or hybrid environments.

  • Lack of centralized management: Organizations with diverse infrastructures struggle to maintain consistent secrets management practices across different platforms.

  • Insufficient NHI lifecycle management: Many solutions don’t adequately address the complete lifecycle of non-human identities, from creation to rotation to decommissioning.

  • Complex integration requirements: Integrating secrets management across various tools and services requires significant custom development.

  • Scalability issues: As the number of secrets and NHIs grows, native solutions may struggle to maintain performance and ease of management.

Entro addresses these challenges by providing a unified, platform-agnostic approach to secrets and non-human identities management. Learn more about Entro’s sophisticated features for advanced secrets and NHIs management here.