Amazon MSK Security: Client authentication, network isolation and connector roles explained

An Amazon MSK cluster is only as private as its listeners: the authentication methods you turn on, the ports your security groups open and whether public access is enabled decide who can produce to and read from every topic. Amazon Managed Streaming for Apache Kafka runs the brokers, but the client authentication model, authorisation rules, network exposure, encryption choices and connector roles are customer-side settings. This guide covers IAM access control, SASL/SCRAM with AWS Secrets Manager, mutual TLS with AWS Private CA, public access, MSK Connect execution roles, logging and a hardening baseline. It is for platform engineers, data engineers and cloud security reviewers running MSK Provisioned or MSK Serverless. Product-neutral Kafka principles, including ACL design, are in the Kafka Security Guide, and client credential hygiene is in the Kafka Credentials Security Guide.

This is an independent guide from NHI Mgmt Group, based on AWS's public documentation for Amazon MSK as of November 2026. It is not sponsored or reviewed by AWS. Product features change, so confirm settings against AWS's current documentation.

Key takeaways

• Pick one primary client authentication method per cluster. IAM access control (port 9098) puts authentication and topic-level authorisation in IAM policy; SASL/SCRAM (9096) and mutual TLS (9094) rely on Kafka ACLs for authorisation.
• Close the default door before enforcing ACLs. With SCRAM or mTLS, define ACLs first and then set allow.everyone.if.no.acl.found to false; AWS warns that skipping this step on a public cluster can expose it without restrictions.
• Keep TLS on everywhere. Client-to-broker TLS only is the default setting and in-cluster encryption is on by default; keep both, and use TLS 1.3 through a TLS security policy where clients support it.
• Treat public access as an exception. It needs authenticated access, in-cluster encryption and no plaintext; add security group rules for named source addresses only.
• Connector and client roles are non-human identities. Scope MSK Connect execution roles and producer roles to named topic and group ARNs, with confused deputy conditions on the trust policy.

How identity and access work in Amazon MSK

MSK has two separate access planes. The control plane (the kafka: API, such as CreateCluster, UpdateSecurity and UpdateConnectivity) is always governed by IAM. The data plane, where clients connect to brokers to produce and consume, is governed by whichever client authentication methods you enable on the cluster.

IPv6 listeners on dual-stack clusters use the 20000 range (20094 TLS, 20096 SCRAM, 20098 IAM). MSK Serverless clusters use IAM access control for clients.

IAM access control

With IAM access control, MSK checks both who the client is and whether it may perform the requested Kafka operation. It works for Java clients and for non-Java clients such as Python, Go, JavaScript and .NET (non-Java clients need Kafka 2.7.1 or later). Two points matter for policy design:

• Kafka ACLs do not apply to IAM identities: you can still call the ACL APIs, but for IAM principals only IAM policy counts, and allow.everyone.if.no.acl.found has no effect on IAM-authenticated access.
• Actions have dependencies: every action requires kafka-cluster:Connect, WriteData also requires DescribeTopic, and ReadData requires DescribeTopic and AlterGroup on the consumer group.

A producer role therefore needs Connect (and WriteDataIdempotently if it uses idempotent writes) on the cluster, plus DescribeTopic and WriteData on its own topics. A consumer role needs Connect on the cluster, DescribeTopic and ReadData on its topics, and DescribeGroup and AlterGroup on its own consumer group. Wildcards can appear after :topic/ and the other resource prefixes; use them for naming conventions such as orders-*, never as kafka-cluster:* on a whole cluster for an application. For policy structure, see the AWS IAM Policies and Roles Guide.

SASL/SCRAM with Secrets Manager

MSK stores SCRAM credentials in AWS Secrets Manager and syncs them to the cluster after you associate the secret. The documented requirements are specific: SCRAM-SHA-512 only; secret names must start with AmazonMSK_; each secret must be encrypted with a customer managed symmetric KMS key rather than the default Secrets Manager key; secrets must be in the same account and Region as the cluster; up to 10 secrets per BatchAssociateScramSecret call; and up to 1,000 users per cluster. Give each application its own secret so a credential can be rotated or revoked without touching other clients. See the AWS Secrets Manager Guide.

Mutual TLS with AWS Private CA

Client certificate authentication uses an AWS Private CA, which can sit in the same account as the cluster or a different one. MSK does not check certificate revocation lists, so AWS's guidance is to remove access for a compromised certificate with Kafka ACLs and to block its source with security groups. Keep certificate lifetimes short and automate reissue so revocation is rarely needed. See the AWS Certificate Manager and Private CA Guide.

Admin roles and privileged access

Privilege in MSK sits in both planes. Control-plane permissions change the cluster's security posture; data-plane permissions change topics, groups and cluster configuration from inside Kafka.

• Kafka administrators are separate from application roles: give topic and configuration changes to an admin role assumed through IAM Identity Center, not to producer or consumer roles. See the AWS IAM Identity Center Guide.
• SCRAM superusers: where SCRAM is used, the user that manages ACLs is effectively the cluster administrator. Keep it in its own secret, used only by the pipeline.
• Organisation guardrails: SCPs can deny UpdateConnectivity and UpdateSecurity outside the pipeline role. See the AWS Organizations and SCPs Guide.

Integrations, API tokens and service accounts

Producer and consumer workloads

Use IAM access control wherever clients run on AWS compute, so that each workload gets temporary credentials from its role (ECS task role, EKS pod identity or IRSA, Lambda execution role, EC2 instance profile) and nothing has to be stored. Java clients use the AWS MSK IAM authentication library with the AWS_MSK_IAM SASL mechanism. For workloads outside AWS, IAM Roles Anywhere can supply temporary credentials; where that is not possible, use SCRAM secrets retrieved at runtime rather than written into configuration files, which avoids the T1552.001 pattern. See the AWS IAM Roles Anywhere Security Guide.

MSK Connect execution roles

Every MSK Connect connector runs as a customer-created service execution role trusted by kafkaconnect.amazonaws.com. AWS's example trust policy adds aws:SourceAccount and aws:SourceArn conditions for confused deputy protection: keep both, with the connector ARN. On IAM-authenticated clusters, the role needs Connect and DescribeCluster on the cluster, read and write on its source or sink topics, CreateTopic, ReadData and WriteData on the internal __amazon_msk_connect_* topics, and DescribeGroup and AlterGroup on the __amazon_msk_connect_* and connect-* groups. AWS mentions broad managed policies such as AmazonS3FullAccess for testing only; for production, scope the connector's sink or source permissions to its own bucket, prefix or table. One role per connector keeps the blast radius to one pipeline.

Cross-account and service access

A cluster policy, managed with PutClusterPolicy, lets other accounts and services reach the cluster through multi-VPC private connectivity. Amazon Data Firehose, for example, can read from MSK in another account when the cluster has multi-VPC connectivity enabled and the cluster policy grants the Firehose service principal and the reading account actions such as kafka:GetBootstrapBrokers, kafka-cluster:Connect, kafka-cluster:DescribeTopic and kafka-cluster:ReadData. Name each account and limit actions to the topics it needs. Review cross-account grants with How to review AWS cross-account trust policies.

Logging and detection

MSK gives you three sources: CloudTrail for control-plane calls, broker logs for activity inside Kafka, and CloudWatch metrics. AWS notes that with IAM access control, MSK logs access events for auditing.

• CloudTrail: records kafka: API calls such as UpdateSecurity, UpdateConnectivity, BatchAssociateScramSecret and PutClusterPolicy. See the AWS CloudTrail Guide.
• Broker logs: deliver to CloudWatch Logs, S3 or Data Firehose, at INFO level by default. Create the destination first; for Firehose, the delivery stream needs the tag LogDeliveryEnabled set to true. AWS recommends S3 or Firehose rather than CloudWatch Logs if you raise the level to DEBUG or TRACE, because of broker performance impact.
• Secrets Manager and KMS events: GetSecretValue on AmazonMSK_ secrets and Decrypt calls on their key show which principals read SCRAM credentials outside the MSK service.

Metrics that show the controls are working

• Authentication coverage: percentage of clusters with unauthenticated access off.
• Plaintext: number of clusters with client-to-broker setting other than TLS only, trending to zero.
• Public exposure: number of clusters with public access on, each with an approved exception.
• Static credentials: number of SCRAM users per cluster and the age of their last rotation.
• Logging coverage: percentage of clusters with broker log delivery enabled.

Hardening baseline

The sequence below is NHI Mgmt Group guidance, not a vendor requirement; adjust it to your environment.

1. Choose the authentication model: IAM access control for AWS-hosted clients; SCRAM or mTLS for clients that need Kafka-native credentials. Turn off unauthenticated access.
2. Encryption in transit: client-to-broker set to TLS only, in-cluster encryption on. AWS notes that in-cluster encryption chosen at creation cannot be switched on later, so set it when the cluster is created. Apply a TLS security policy that allows TLS 1.3 on Provisioned clusters where clients support it. Brokers use public ACM certificates that MSK renews automatically every 13 months.
3. Encryption at rest: MSK always encrypts data at rest. Specify a customer managed KMS key at creation for sensitive clusters, so key policy and CloudTrail give you control and visibility; otherwise MSK uses an AWS managed key. See the AWS KMS Security Guide.
4. Authorisation: for IAM, one role per application scoped to topic and group ARNs. For SCRAM or mTLS, write ACLs per principal, then set allow.everyone.if.no.acl.found=false in the cluster configuration.
5. Network: place brokers in private subnets, allow only the listener ports you use in security groups, and reference client security groups instead of CIDR ranges. Use multi-VPC private connectivity for other accounts. See the Amazon VPC Security Guide.
6. Public access, only by exception: requires Kafka 2.6.0 or later, public subnets, unauthenticated access off, at least one of IAM, SCRAM or mTLS on, in-cluster encryption on and plaintext off. Enable it after creation with UpdateConnectivity, and restrict inbound rules to named addresses on ports 9194, 9196 or 9198.
7. Logging: enable broker log delivery and alert on the signals above.
8. Connectors: one execution role per MSK Connect connector with confused deputy conditions and scoped sink or source permissions.

Testing and rollback: roll out authentication changes by running two methods side by side (for example SCRAM and IAM), moving clients across, then disabling the old method with UpdateSecurity. Keep the previous cluster configuration revision so allow.everyone.if.no.acl.found can be reverted if a missing ACL stops a critical consumer. Small teams with a few AWS-hosted applications can standardise on IAM access control and skip ACL management; large estates with many external clients usually run IAM for internal workloads and mTLS for partners.

Common misconfigurations

• ACL default left open: SCRAM or mTLS enabled but allow.everyone.if.no.acl.found left at its permissive value, so any authenticated user can reach any topic without an ACL.
• Mixed TLS and plaintext listeners kept "for now": the TLS_PLAINTEXT setting stays on long after migration. Move to TLS only once client metrics show no plaintext connections.
• Wildcard IAM policies: application roles with kafka-cluster:* on topic/cluster-name/*, giving every producer delete and configuration rights.
• Shared SCRAM users: one user name for many services, so rotation breaks everything and logs cannot tell clients apart.
• Broad connector roles: MSK Connect roles carried into production with test-time managed policies.
• Wide security groups: broker ports open to whole VPC ranges or the internet when public access is on.

Framework mapping

Practitioner checklist

• Unauthenticated access is off on every cluster; each has IAM, SCRAM or mTLS on.
• Client-to-broker encryption is TLS only and in-cluster encryption is on.
• Sensitive clusters use a customer managed KMS key for encryption at rest.
• IAM client roles are scoped to named topic and group ARNs with no kafka-cluster:* for applications.
• SCRAM and mTLS clusters have ACLs per principal and allow.everyone.if.no.acl.found=false.
• Each application has its own SCRAM secret under a customer managed key, rotated on a schedule.
• Public access is off unless an approved exception exists, with inbound rules limited to named addresses.
• Each MSK Connect connector has its own execution role with aws:SourceAccount and aws:SourceArn conditions.
• Broker logs are delivered and CloudTrail alerts cover UpdateSecurity, UpdateConnectivity and PutClusterPolicy.

Standards and references

• AWS: IAM access control for Amazon MSK
• AWS: Semantics of IAM authorisation policy actions and resources
• AWS: Limitations when using SCRAM secrets
• AWS: Mutual TLS client authentication for Amazon MSK
• AWS: Amazon MSK encryption
• AWS: Turn on public access to an MSK Provisioned cluster
• AWS: Service execution role for MSK Connect
• NIST: SP 800-53 Rev. 5 Security and Privacy Controls

Related NHI Mgmt Group resources: Kafka Security Guide, Kafka Credentials Security Guide, Amazon Kinesis Security, AWS Secrets Manager Guide, Data Security Guide.