Amazon CloudWatch is where most AWS detection signals land, so whoever can read, mute or delete it controls what your security team sees. Log groups hold application output, VPC flow logs and CloudTrail copies; alarms decide who gets woken up; subscription filters feed your SIEM. Each of those is a customer-side setting with its own identities, keys and retention rules. This guide covers CloudWatch as a security asset in its own right: who can read and change it, how to encrypt and mask log data, retention and deletion protection, the CIS metric filter set, cross-account observability and how to keep the pipeline itself monitored. It is written for cloud security engineers, detection engineers and auditors reviewing AWS logging. For trail design and CloudTrail event coverage, see the AWS CloudTrail Guide.
This is an independent guide from NHI Mgmt Group, based on AWS's public documentation as of November 2026. It is not sponsored or reviewed by AWS. Product features change, so confirm settings against AWS's current documentation.
Key takeaways
- Logs never expire unless you say so. New log groups default to Never Expire, so set retention deliberately, account wide, and protect important groups from deletion (off by default).
- Masking starts at ingestion, not before. Data protection policies mask secrets and personal data from the moment the policy exists; earlier events stay as written, so turn the account-level policy on early.
- Muting an alarm is a log tampering technique. DisableAlarmActions, DeleteAlarms and subscription filter changes deserve the same alerting as StopLogging on a trail.
- Log class is a one-way choice. Essentials log groups cannot run metric filters, subscription filters or anomaly detection, and the class cannot be changed after creation.
- The pipeline needs its own health checks. Subscription deliveries that hit AccessDeniedException are paused and the events in that window are skipped, so watch forwarding metrics, not just alarms.
How identity and access work in CloudWatch
CloudWatch has two API surfaces with separate IAM namespaces: cloudwatch: (event source monitoring.amazonaws.com) for metrics, alarms and dashboards, and logs: (logs.amazonaws.com) for log groups, queries and filters. Access is granted through identity policies on roles, with resource scoping by log group ARN and tags. AWS notes that access to log streams is controlled at the log group level, and tags can control access to log groups and destinations, which makes tag-based access control practical for separating application teams from security logs.
Three permissions carry outsized weight:
- logs:Unmask: only principals with this permission can view data masked by a data protection policy. Grant it to a small break-glass role, never to wildcard
logs:*developer policies. - Read paths:
logs:GetLogEvents,logs:FilterLogEventsandlogs:StartQueryreturn raw content. For KMS-encrypted log groups the caller also needskms:Decrypton the key, which gives you a second, key-level gate. - Write paths:
logs:PutLogEventsandlogs:CreateLogStreamlet a principal inject events. A compromised workload with broad write rights can flood or poison a log group that detections rely on, so scope writers to their own groups.
| Action group | Example actions | Security impact | Who should hold it |
|---|---|---|---|
| Log read | logs:GetLogEvents, logs:FilterLogEvents, logs:StartQuery | Exposure of application data, tokens written to logs, flow and audit records | Owning team for its groups; SOC read role for security groups |
| Unmask | logs:Unmask | Reveals masked credentials and personal data | Break-glass role with approval and alerting |
| Log destruction | logs:DeleteLogGroup, logs:PutRetentionPolicy, logs:DeleteRetentionPolicy, logs:PutAccountPolicy | Evidence loss, shortened retention across the account | Platform logging admins only, guarded by SCP |
| Pipeline change | logs:PutSubscriptionFilter, logs:DeleteSubscriptionFilter, logs:DeleteMetricFilter | Silently stops SIEM forwarding or a CIS alarm feed | Logging pipeline role, change-controlled |
| Alarm control | cloudwatch:DisableAlarmActions, cloudwatch:DeleteAlarms, cloudwatch:SetAlarmState | Mutes or fakes the state of security alarms | Monitoring admins; deny on security-tagged alarms |
| Key association | logs:AssociateKmsKey, logs:DisassociateKmsKey | Changes the encryption gate for new data | Logging admins with KMS key owners |
See the AWS IAM Policies and Roles Guide for writing scoped policies and conditions.
Admin roles and privileged access
Treat the logging account as a security boundary. In a multi-account estate the central logging or monitoring account holds the copies that matter, and source accounts should not be able to delete them. Practical patterns:
- Service control policies: deny
logs:DeleteLogGroup,logs:DeleteSubscriptionFilter,logs:DisassociateKmsKeyandcloudwatch:DisableAlarmActionson security-owned resources (by name prefix or tag) for every principal except a named logging role. See the AWS Organizations and SCPs Guide. - Deletion protection: set
--deletion-protection-enabledon security log groups withaws logs put-log-group-deletion-protection. A delete attempt then fails with a ValidationException until protection is turned off, which adds a deliberate second step an attacker must take, and that you can alert on. - Account-level retention policies:
aws logs put-account-policy --policy-type RETENTION_POLICYapplies retention by log group name prefix, withenforcementModeofRETROACTIVEorNEW_ONLY. AWS warns that an account policy replaces a log group's own retention even where that is longer, so it can shorten retention; it also allows up to 20 retention policies per Region and takes effect 24 hours after creation. Use--dry-runfirst and restrictlogs:PutAccountPolicyto the logging admin role. - Human access: run administrators through federated roles from the AWS IAM Identity Center Guide rather than IAM users, so every change is tied to a person in CloudTrail.
Integrations, API tokens and service accounts
Most CloudWatch writers and readers are machines: agents on instances, Lambda execution roles, the CloudTrail delivery role, subscription roles and SIEM collectors. These are the non-human identities to inventory.
| Integration | Identity it uses | Hardening point |
|---|---|---|
| CloudWatch agent on EC2 | Instance role with the managed policy CloudWatchAgentServerPolicy | Attach through an instance profile; add logs:PutRetentionPolicy only if the agent must set retention |
| CloudWatch agent on premises | IAM user with access keys, as the agent documentation describes | AWS itself flags long-term keys as a risk: least privilege, secure storage, rotation, removal when unused. See the AWS Access Keys Security Guide |
| Subscription to Kinesis Data Streams or Firehose | Role assumed by logs.<region>.amazonaws.com | Trust policy conditions on aws:SourceArn and aws:SourceAccount (or aws:SourceOrgID) to prevent confused deputy use |
| CloudTrail to CloudWatch Logs | Delivery role CloudTrail assumes | Scope to the one log group; covered in the CloudTrail guide |
| Cross-account observability | Observability Access Manager sink in the monitoring account and links from source accounts | Share only the telemetry types needed; the source account owns and can remove its link |
| Private connectivity | Interface endpoints com.amazonaws.Region.logs and com.amazonaws.Region.stream-logs | Replace the default full-access endpoint policy, for example with a put-only policy allowing logs:CreateLogStream and logs:PutLogEvents |
Two further controls help with workload identities. CloudWatch Logs supports the aws:SourceVpc and aws:SourceVpce condition keys, which take effect for requests through a VPC endpoint, so a policy can insist that log writes from production arrive through the production endpoint. And a workload's instance role should never carry logs:*: if the host is compromised, the attacker inherits the right to delete the evidence of it. For finding over-privileged roles on hosts, see How to find EC2 instance profiles with admin rights.
Protecting log content: encryption and data protection
Encryption at rest
Log data is always encrypted at rest; by default CloudWatch Logs uses server-side 256-bit AES-GCM. Associating a customer managed KMS key adds a key policy you control and a key-level audit trail. Points from the AWS documentation that catch teams out:
- Per log group: the key is associated per log group, at creation (
create-log-group --kms-key-id) or later withassociate-kms-key. The console cannot associate a key with an existing group, so use the CLI or infrastructure as code. - Symmetric keys only, and the key policy must allow the Regional service principal
logs.<region>.amazonaws.com. Restrict it with thekms:EncryptionContext:aws:logs:arncondition to one log group ARN or the account. - Forward only: association affects newly ingested data; existing events keep their previous encryption. Changes can take up to five minutes.
- Key loss is data loss: if the key is disabled, CloudWatch Logs cannot read data encrypted with it, and if access is revoked or the key deleted, the data cannot be retrieved. Guard
kms:ScheduleKeyDeletionandkms:DisableKey(CIS filter CloudWatch.7 exists for exactly this).
AWS recommends a separate key per encrypted log group. See the AWS KMS Security Guide for key policy design.
Data protection policies (masking)
Applications leak secrets into logs: bearer tokens, connection strings, access keys. Data protection policies audit and mask matches using managed data identifiers in five categories (credentials such as private keys and AWS secret access keys, financial data, personal data, health data and device identifiers) plus custom identifiers you define.
- Scope: an account-level policy covers all existing and future log groups; a log group can also carry one log group policy. Where both exist, the identifiers from both apply. Policy size is limited to 30,720 characters.
- Coverage: masking applies at all egress points, including Logs Insights, metric filters and subscription filters, so your SIEM receives masked values too. Check that detections do not depend on a field you are now masking.
- Timing: detection and masking happen at ingestion; events ingested before the policy are not masked. If a secret has already been logged, rotate it.
- Signal: the free vended metric
LogEventsWithFindingsin theAWS/Logsnamespace counts events with findings. Alarm on it per log group: a rising count usually means a code change started logging credentials. See the Secrets Management Guide for the fix upstream.
Logging and detection
CloudWatch as the detection layer
AWS Security Hub's CloudWatch controls CloudWatch.1 to CloudWatch.14 encode the CIS metric filter and alarm set over CloudTrail logs delivered to CloudWatch Logs. Each requires the CIS filter pattern, a metric in the LogMetrics namespace and an alarm at threshold greater than or equal to 1. The control fails if the filter adds extra fields or terms, so keep the compliance filter exact and build richer detections alongside it.
| Control | Detects | Why it matters |
|---|---|---|
| CloudWatch.1 | Root user usage | Root should almost never act |
| CloudWatch.2 / .6 | Unauthorised API calls; console authentication failures | Credential probing and misused keys |
| CloudWatch.3 | Console sign-in without MFA | Weak human access paths |
| CloudWatch.4 | IAM policy changes | Privilege escalation and persistence |
| CloudWatch.5 | CloudTrail configuration changes | Log tampering |
| CloudWatch.7 | Disabling or scheduled deletion of customer managed keys | Can make encrypted logs and data unreadable |
| CloudWatch.8 / .9 | S3 bucket policy changes; AWS Config changes | Data exposure; posture monitoring blinded |
| CloudWatch.10 to .14 | Security group, NACL, gateway, route table and VPC changes | Network exposure |
| CloudWatch.15 / .17 | Alarm has an ALARM action; that action is enabled | An alarm with no live action alerts nobody |
| CloudWatch.16 | Log group retention at least minRetentionTime (default 365 days) | Evidence available for investigations |
Detecting attacks on CloudWatch itself
MITRE ATT&CK T1685.002 Disable or Modify Cloud Log covers adversaries turning off cloud logging integrations. In CloudWatch terms, alert on these CloudTrail management events:
- monitoring.amazonaws.com: DisableAlarmActions, SetAlarmState (an attacker can force an alarm back to OK), PutMetricAlarm changes to security alarms, StopMetricStreams and DeleteMetricStream.
- logs.amazonaws.com: DeleteLogGroup, PutRetentionPolicy, DeleteSubscriptionFilter, DeleteMetricFilter, DisassociateKmsKey, PutAccountPolicy, deletion protection changes and data protection policy deletion. Route these through EventBridge to a destination outside the account being watched.
Metric data-plane calls (GetMetricData, PutMetricData, GetMetricStatistics, ListMetrics, GetMetricWidgetImage) are CloudTrail data events, not logged by default and charged extra. Enable them only where you need to investigate metric injection or reconnaissance. Logs Insights read activity is worth reviewing for unusual principals querying security log groups.
Anomaly detection and pipeline health
- Log anomaly detection: detectors train on two weeks of a log group's events and flag new patterns, variations and token changes. AWS describes them as suited to application logs and not to long JSON records such as CloudTrail, because only the first 1,500 characters of each line are analysed. The default anomaly visibility time is 21 days, after which a recurring anomaly is treated as normal, and masked portions are not scanned.
- Subscription health: delivery is at least once, with retries for up to 24 hours on throttling and server errors. On non-retryable errors such as AccessDeniedException the filter is disabled for up to 10 minutes and logs in that window are skipped. Alarm on the subscription forwarding metrics, and design destinations that do not write back into the same log group, which can cause recursion.
- Cross-account view: CloudWatch cross-account observability lets a monitoring account search metrics, logs and traces from up to 100,000 source accounts; a source account can share with up to five monitoring accounts. Link the whole organisation so new accounts are onboarded automatically.
For writing and tuning the detections themselves, see the Cloud Detection Engineering Guide and the Amazon GuardDuty Guide for managed threat findings.
Hardening baseline
| Setting | Baseline value | How to enforce or check |
|---|---|---|
| Retention | Explicit on every group; 365 days or more for security logs | Account retention policy; Security Hub CloudWatch.16 |
| Deletion protection | On for security, audit and flow log groups | put-log-group-deletion-protection; SCP deny on delete |
| Encryption | Customer managed symmetric KMS key for security groups | describe-log-groups shows kmsKeyId; IaC policy check |
| Data protection | Account-level policy with credential identifiers at minimum | describe-account-policies; alarm on LogEventsWithFindings |
| Log class | Standard for any group that feeds alarms, filters or a SIEM | Set at creation; review new groups |
| Alarms | CIS set present, ALARM action configured and enabled | CloudWatch.1 to .15 and .17 |
| Endpoints | Interface endpoints with restrictive endpoint policies | Review endpoint policy is not the default full access |
| Unmask | Held by one break-glass role | IAM Access Analyzer and policy search for logs:Unmask and logs:* |
Rollout order
- Inventory: list log groups with retention, class, kmsKeyId and size per account and Region; flag Never Expire and Essentials groups that should be feeding detections.
- Protect evidence first: enable deletion protection and the SCP guardrails on security groups, then apply account retention policies in
--dry-runbefore enforcing. Roll back by deleting the account policy; log group settings resume. - Mask: deploy the account-level data protection policy in a non-production account, watch LogEventsWithFindings and check that detections and dashboards still work on masked output, then promote.
- Encrypt: create keys with the encryption context condition, associate them through the CLI or IaC, and test read access for the SOC role before and after.
- Detect: deploy the CIS filters and the tampering rules above, confirm each alarm action delivers to an on-call channel, and test by generating a harmless matching event.
- Centralise: link source accounts to the monitoring account through Organizations and verify the shared telemetry types.
Small and large estates
A single-account team can get most of the value from retention, deletion protection, the CIS alarm set and an account-level data protection policy. Larger organisations should add a dedicated monitoring account, SCPs that keep source accounts from deleting security groups, per-group KMS keys owned by the security team and tag-based access so application teams query only their own logs. For the cross-cloud view of retention and audit trails, see the Cloud Logging and Audit Trail Guide.
Common misconfigurations
- Never Expire everywhere, or a retroactive account policy that shortens it: both happen. One drives cost until someone deletes groups wholesale; the other quietly removes a year of evidence. Dry-run retention changes and keep security groups on explicit long retention.
- Security logs in the Essentials class: the group ingests happily but cannot run metric filters or subscriptions, and the class cannot be changed. The fix is a new Standard group and a cut-over.
- Alarms with no live action: an alarm without an ALARM action, or with actions disabled after maintenance, reports nothing. Security Hub CloudWatch.15 and .17 catch both.
- Broad logs:* on workload roles: a compromised host can then delete its own log group, change retention or remove the subscription feeding the SIEM.
- Masking turned on after the leak: earlier events remain unmasked. Rotate any credential found, and apply the account-level policy before production traffic.
- Disabled or deleted KMS key: encrypted log data becomes unreadable at the moment responders need it. Restrict key deletion and alarm on CloudWatch.7 events.
Framework mapping
| Framework | Reference | CloudWatch controls |
|---|---|---|
| NIST SP 800-53 Rev 5 | AU-2, AU-12 | What is logged and generated into log groups |
| NIST SP 800-53 Rev 5 | AU-6, SI-4 | Metric filters, alarms, anomaly detection, SIEM subscriptions |
| NIST SP 800-53 Rev 5 | AU-9 | Deletion protection, SCPs, KMS keys, logs:Unmask restriction |
| NIST SP 800-53 Rev 5 | AU-11 | Retention policies |
| NIST SP 800-53 Rev 5 | SC-28, SC-12 | KMS encryption of log groups and key management |
| NIST CSF 2.0 | PR.PS-04, DE.CM-01, DE.AE-02 | Log records available for monitoring; monitoring and analysis |
| CIS Controls v8.1 | Control 8 Audit Log Management | Retention, centralisation, review |
| ISO/IEC 27001:2022 Annex A | A.8.15 Logging, A.8.16 Monitoring activities | Log protection and monitoring |
| MITRE ATT&CK | T1685.002 Disable or Modify Cloud Log | Tampering detections on alarms, filters and log groups |
See the CIS Control 8 Audit Log Management Guide for the control-level requirements.
Practitioner checklist
- Set explicit retention on every log group, using account retention policies tested with
--dry-run. - Enable deletion protection on security, audit and flow log groups and back it with an SCP.
- Deploy an account-level data protection policy with credential identifiers and alarm on LogEventsWithFindings.
- Restrict
logs:Unmask,logs:DeleteLogGroup,logs:PutAccountPolicyandcloudwatch:DisableAlarmActionsto named admin roles. - Encrypt security log groups with symmetric customer managed keys scoped by
kms:EncryptionContext:aws:logs:arn. - Keep detection-feeding groups in the Standard log class.
- Deploy the CIS metric filters exactly, confirm alarm actions are configured and enabled, and add tampering rules for CloudWatch APIs.
- Add
aws:SourceArnandaws:SourceAccountconditions to subscription role trust policies and monitor forwarding metrics. - Replace default full-access endpoint policies on CloudWatch Logs interface endpoints.
- Link all accounts to a monitoring account through Organizations.
Standards and references
- AWS: Help protect sensitive log data with masking
- AWS: Encrypt log data in CloudWatch Logs using AWS KMS
- AWS: Security Hub controls for CloudWatch
- AWS: Working with log groups and log streams
- AWS: Real-time processing of log data with subscriptions
- AWS: Logging Amazon CloudWatch API calls with AWS CloudTrail
- AWS: CloudWatch cross-account observability
- MITRE ATT&CK: T1685.002 Disable or Modify Cloud Log
Related NHI Mgmt Group resources: AWS Security Guide · AWS CloudTrail Guide · Cloud Logging and Audit Trail Guide · Cloud Detection Engineering Guide