Secrets managers log every read, but most SOCs either do not collect those logs or drown in them, so a bulk dump of production credentials looks the same as a normal deployment. Applications read secrets thousands of times a day, which makes "secret accessed" useless as an alert; the value is in who read which secret, from where, how many and in what sequence. This guide lists the events that matter in AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, HashiCorp Vault and Kubernetes, shows how to switch on the logs that are off by default, and gives detection logic and tuning methods that keep alert volume low. It is for detection engineers, SOC analysts and platform teams who own secrets infrastructure. For network-layer detection in general, see the IDS and IPS Guide; for choosing and running a secrets manager, see the Cloud Secrets Managers Guide.
Key takeaways
- Some of the most important logs are off by default. Google Cloud Data Access audit logs, which record AccessSecretVersion, are disabled by default for most services, and Azure Key Vault logs need a diagnostic setting before AuditEvent records are kept.
- Detect on the reader, not the read. Each secret has a small, stable set of legitimate readers; an alert on "new identity reads this secret" is far quieter and more useful than any threshold on read counts.
- Enumeration plus breadth is the dump signature. Listing secrets and then reading many distinct secrets in a short window is what both supply chain worms and human operators do; applications almost never do it.
- Policy and permission changes outrank reads. A changed resource policy, IAM binding or Vault policy that widens access deserves a page; most reads do not.
- Protect the log pipeline too. Vault refuses requests when no audit device can write, which is a safety property and an outage risk; cloud audit logs need retention and tamper protection beyond the default views.
Why secrets access is hard to monitor
MITRE ATT&CK v20 tracks theft from secrets managers as T1555.006 Cloud Secrets Management Stores: an adversary with sufficient privileges, for example a high-privilege cloud account (T1078.004) or a compromised service with retrieval permission, requests secrets directly. The only mitigation listed is M1026 Privileged Account Management, limiting who can query the store and which secrets each reader can reach. ATT&CK's detection strategy DET0130 describes the signals: secrets accessed by unusual identities, from unexpected regions, outside business hours or at high volume, correlated with the surrounding authentication events.
Three properties make this difficult in practice:
- Volume: workloads that fetch a secret per request or per cold start generate a constant stream of successful reads.
- Identity reuse: one role or service principal often serves many pods, functions or pipelines, so "the identity is normal" says little.
- Legitimate bulk: deployment tools, backup jobs and secret synchronisation operators read many secrets at once by design.
The answer is to model each secret's expected readers and each reader's expected secrets, then alert on departures, rather than on activity itself.
The events that matter, by platform
| Platform | Secret read | Enumeration and metadata | Access and policy change | Logging prerequisite |
|---|---|---|---|---|
| AWS Secrets Manager (CloudTrail, event source secretsmanager.amazonaws.com) | GetSecretValue (also generated for BatchGetSecretValue calls) | ListSecrets, DescribeSecret, ListSecretVersionIds | PutResourcePolicy, DeleteResourcePolicy, UpdateSecret, ReplicateSecretToRegions, plus IAM policy changes | Console history covers 90 days; create a trail to S3 for retention and analysis |
| Azure Key Vault (category AuditEvent) | SecretGet | SecretListVersions and other Secret list operations | Access policy or role assignment changes; network rule changes | Enable logging to a storage account or Azure Monitor Logs; records arrive within 10 minutes |
| Google Cloud Secret Manager (Cloud Audit Logs) | AccessSecretVersion (DATA_READ) | ListSecrets, GetSecret, ListSecretVersions (ADMIN_READ) | SetIamPolicy, UpdateSecret, AddSecretVersion (Admin Activity) | Enable Data Access logs (DATA_READ and ADMIN_READ) for the service; they are off by default |
| HashiCorp Vault (audit device) | Read requests on KV paths; credential issuance on dynamic secrets engines | List operations on secret paths | Writes to sys/policy, auth method and role changes, new tokens with broad policies | Enable at least two audit devices (file, syslog or socket) |
| Kubernetes (API server audit) | get on secrets | list and watch on secrets | Role, ClusterRole and binding changes granting secrets access | Audit policy with secrets at Metadata level so values are not logged |
Platform notes
- AWS: rotation, replication and deletion outcomes (RotationStarted, RotationSucceeded, RotationFailed, RotationAbandoned, StartSecretVersionDelete, EndSecretVersionDelete) arrive with detail-type "AWS Service Event via CloudTrail" rather than as API calls, so EventBridge rules for them need that pattern. Logs before February 2024 sometimes carried the secret ARN under the key aRN; normalise both.
- Azure: the identity field holds the caller from the token (user, service principal or user plus appId) and callerIpAddress the client address, which are the two fields most detections pivot on.
- Google Cloud: Data Access logs can be large and are billable; enable DATA_READ for Secret Manager specifically rather than all services. Exempting principals is possible, but an exemption also hides that principal from investigations, so use it sparingly.
- Vault: audit entries hash most string values with HMAC-SHA256, so a token or secret value never appears in clear. To find a known token or accessor in the logs, compute its hash with the sys/audit-hash endpoint for that audit device. Setting hmac_accessor=false leaves token accessors readable, which helps correlation.
- Kubernetes: the upstream example audit policy logs secrets and configmaps at Metadata level; Request or RequestResponse levels for secrets would write secret material into the audit log.
See the HashiCorp Vault Guide and the Kubernetes Secrets Security Guide.
High-signal detections
| Detection | Logic | Why it is high signal | Main false positives |
|---|---|---|---|
| New reader for a secret | Identity reads a secret it has not read in the previous 30 days | Each secret has a stable reader set; theft almost always introduces a new reader or a new source | New deployments, new regions; suppress for change windows tied to a deployment record |
| Enumerate then read many | List or describe call followed within minutes by reads of more than N distinct secrets by one identity | Matches dump behaviour; applications know their secret names | Sync operators and backup jobs; allowlist by identity and source |
| Human reads production secret | Interactive user or console session reads a secret tagged production | Production secrets should be read by workloads; break-glass is rare | Approved break-glass; require a ticket reference |
| Workload identity read from unexpected network | Role, service principal or service account reads from an address outside its known VPC, subnet or egress ranges | A stolen workload credential is usually used from somewhere else | NAT changes; maintain ranges as code |
| Access widened | Resource policy, IAM binding, Key Vault access policy or Vault policy change that adds a principal or wildcard | Persistence and preparation step for later reads | Infrastructure as code pipelines; alert when the change did not come from the pipeline identity |
| Denied reads burst | Many AccessDenied or 403 results for one identity across many secrets | Attackers probe what a stolen credential can reach | Misconfigured new service; usually a single secret, not many |
| Canary secret read | Any read of a decoy secret that no workload uses | Near-zero false positives | Discovery tools; exclude by identity only after review |
| Rotation failures | RotationFailed or RotationAbandoned, or Vault lease revocation errors | Not an attack signal, but leaves stale credentials valid longer | Route to platform team, not the SOC queue |
Canary secrets are covered in depth in the Honeytokens and Canary Credentials Guide.
Building baselines and tuning alerts
- Tag secrets: environment, owner, data classification and expected reader identities. Detections become filters on tags rather than lists of names.
- Learn reader sets: for each secret, compute the set of (identity, source network) pairs over 30 days. Store it as a lookup table refreshed daily.
- Run in shadow mode: send detections to a log index, not the alert queue, for two to four weeks. Review every hit and either fix the baseline or keep the rule.
- Tie suppression to change records: suppress "new reader" alerts only when a deployment or change record names that identity and secret, and expire the suppression automatically.
- Separate workloads from humans: give automation distinct identities so human reads stand out. Shared roles defeat every reader-based rule.
- Promote to paging: only rules with a review-proven precision go to the on-call queue; the rest stay as hunting queries.
- Roll back cleanly: manage rules as code with version history, so a noisy change can be reverted without losing the tuning that came before.
Small teams should start with three rules: canary reads, access widened outside the pipeline, and enumerate-then-read-many. Large estates with thousands of secrets need the reader-set model, because static thresholds either miss low-and-slow theft or alert constantly.
Correlating secrets events with other telemetry
- Identity provider: join the reader to its most recent authentication: a role session's source, a service principal's sign-in, a federated token's issuer. A read by a session that began from an unfamiliar location is a stronger signal than either event alone.
- Use of the secret: after a suspicious read, look at the system the secret unlocks (database logins, API calls with that key). That tells you whether the secret was used and from where.
- Compute and CI context: map reads to the pod, function, runner or job that made them. In the Shai-Hulud first wave, malicious package code on developer machines and CI runners pulled AWS Secrets Manager and GCP Secret Manager contents with the credentials it found there, so the reads came from identities that were allowed to read.
- Network: egress from the reader shortly after bulk reads points to exfiltration; see the T1567 Exfiltration Over Web Service guide.
Triage: the first 30 minutes after an alert
- Confirm the reader: resolve the identity to a workload, pipeline or person, and check whether a change record or deployment explains the read. On Azure use the identity and callerIpAddress fields; on AWS the userIdentity and sourceIPAddress of the GetSecretValue event; on Vault, compute the token hash with sys/audit-hash if you hold the token.
- Build the read set: list every secret that identity read, and every denied attempt, from one hour before the alert to now. Enumeration calls tell you what the reader knew existed.
- Check the session origin: find the authentication event that created the session (role assumption, federated token exchange, Vault login) and its source network and client.
- Check downstream use: for each secret in the read set, look for logins or API calls with that credential from unexpected sources since the read.
- Decide: if the reader cannot be explained, revoke the session and tokens, block the source, and start rotation of the whole read set in priority order (production, highest privilege, externally reachable first).
- Preserve evidence: export the relevant audit records to write-once storage before any retention window or log rotation removes them.
Where identity fits
Secrets access logs are identity logs for non-human identities. Their quality depends on identity design: one identity per workload rather than per team, workload identity federation instead of long-lived keys so each session has a traceable origin, and scoped policies so each identity can read only its own secrets. When an alert fires, the response is identity work too: revoke the reader's session, rotate every secret it read in the window, and review how it got its permissions. See the Secrets Rotation Runbook Template and the Workload Identity Federation Guide.
Measuring whether it works
- Logging coverage: percentage of secret stores with read events collected centrally (target 100 per cent for production).
- Reader attribution: percentage of reads attributable to a single workload rather than a shared identity.
- Alert precision: share of paging secrets alerts that led to action, reviewed monthly.
- Time to rotate after a suspicious read: from alert to all secrets in the read set rotated.
- Canary coverage: number of high-value stores with at least one canary secret.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| MITRE ATT&CK v20 | T1555.006 Cloud Secrets Management Stores; T1552.007 Container API; T1078.004 Cloud Accounts; M1026 | Threat being detected and the listed mitigation |
| NIST SP 800-53 Rev. 5 | AU-2 Event Logging, AU-3 Content of Audit Records, AU-6 Audit Record Review, AU-9 Protection of Audit Information, AU-11 Audit Record Retention, AU-12 Audit Record Generation | Collecting, protecting and reviewing secrets logs |
| NIST SP 800-53 Rev. 5 | SI-4 System Monitoring; IA-5 Authenticator Management | Detection and the rotation that follows |
| NIST CSF 2.0 | PR.PS-04, DE.CM-03, DE.AE-03 | Logs available for monitoring; monitoring usage; correlation across sources |
Common failure patterns
- Data Access logs never enabled: on Google Cloud, AccessSecretVersion is a Data Access event, so without DATA_READ enabled there is no record of who read a secret.
- Alerting on every read: the queue fills in a day, the rule is disabled, and coverage silently drops to zero.
- Shared reader identities: one role for all services means a dump by a compromised pod looks exactly like normal traffic.
- Logging secret values: Kubernetes audit policies at Request or RequestResponse for secrets, or debug logging in applications, put the secrets into the SIEM.
- Single Vault audit device: when its disk fills or its syslog target fails, Vault stops serving requests.
- No follow-through to rotation: the alert is closed after the session is revoked, but the secrets that were read stay valid.
Lessons from incidents
In the AI agent retail card theft campaign, Gambit Security's recovered evidence showed an agent dumping AWS Secrets Manager on one compromised host, retrieving 46 secrets that led to the main e-commerce database and its encryption key. No source describes a detection that caught the dump. An enumerate-then-read-many rule on that account, and a new-reader rule on the database secret, are the two detections that match this behaviour.
Practitioner checklist
- Enable and centralise read logs for every production secret store: CloudTrail trail, Key Vault AuditEvent, Secret Manager DATA_READ, two Vault audit devices, Kubernetes audit at Metadata for secrets.
- Tag secrets with owner, environment, classification and expected readers.
- Give each workload its own identity so reader-based detections work.
- Deploy canary secrets in each high-value store and alert on any read.
- Build reader-set baselines and run new-reader and enumerate-then-read-many rules in shadow mode first.
- Alert on access-widening changes that did not come from the infrastructure pipeline.
- Correlate suspicious reads with the reader's authentication and the secret's downstream use.
- Route rotation failures to the platform team and track them to closure.
- After any confirmed suspicious read, rotate every secret in the read set, not only the one that triggered the alert.
Standards and references
- AWS: Secrets Manager CloudTrail log entries
- Microsoft Learn: Azure Key Vault logging
- Google Cloud: Secret Manager audit logging
- Google Cloud: Enable Data Access audit logs
- HashiCorp: Vault audit devices
- Kubernetes: Auditing
- MITRE ATT&CK: Cloud Secrets Management Stores (T1555.006)
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: Cloud Secrets Managers Guide · Honeytokens and Canary Credentials Guide · IDS and IPS Guide · Secrets Management Guide