A secret stored in AWS Secrets Manager is not a rotated secret: many accounts hold database passwords and API keys that have sat unchanged for years because rotation was never turned on, was turned on and keeps failing, or was replaced by someone pasting new values by hand. This runbook shows how to find all three cases with the data AWS already exposes: the ListSecrets fields, the three AWS Config managed rules for rotation, the matching Security Hub controls and the rotation events in CloudTrail. It then covers how to triage what you find, fix rotation safely and keep the problem from returning. It is written for cloud security and platform engineers working in one account or across an AWS Organization. For how rotation itself is designed and how the access model works, see the AWS Secrets Manager guide; this page is the discovery task.
This is an independent guide from NHI Mgmt Group, based on AWS's public documentation as of November 2026. It is not sponsored or reviewed by AWS. Product features change, so confirm settings against AWS's current documentation.
Key takeaways
- There are three kinds of unrotated secret. Rotation off (
RotationEnabledfalse), rotation on but failing (noRotationSucceededfor longer than the schedule), and rotation never completed (LastRotatedDatenull). Each needs a different fix. - ListSecrets gives you the whole picture for free. One paginated call per Region returns
RotationEnabled,LastRotatedDate,NextRotationDate,LastChangedDate,LastAccessedDateandOwningServicefor every secret. - AWS Config covers it continuously.
secretsmanager-rotation-enabled-check,secretsmanager-scheduled-rotation-success-checkandsecretsmanager-secret-periodic-rotationmap to Security Hub controls SecretsManager.1, .2 and .4. - Failing rotation is the dangerous case. It looks compliant in a "rotation enabled" report while the value stays the same, and a stuck
AWSPENDINGlabel blocks the next attempt. - Test before you schedule.
rotate-secret --no-rotate-immediatelyruns only the test step, so you can prove a fixed rotation function works before any application sees a new value.
When you need this
- Audit or compliance finding: Security Hub reports failed SecretsManager.1 or SecretsManager.4 findings, or an auditor asks for the age of every stored credential under NIST SP 800-53 IA-5.
- After a leak or a supplier breach: you need to know which secrets would still work for an attacker who copied them months ago. The Internet Archive breach shows how keys left unrotated after a first exposure became the route to a second compromise.
- Migration into Secrets Manager: teams often move a secret in, tick the project done and never configure rotation.
- Before tightening policy: enforcing a 90 day maximum age is easier once you know how many secrets fail it today.
Before you start
| Source | What it tells you | Cost and permission | Limit to know |
|---|---|---|---|
ListSecrets / DescribeSecret | Rotation flag, schedule, last and next rotation, last change, last access, owning service, rotation function ARN | secretsmanager:ListSecrets, secretsmanager:DescribeSecret; standard API pricing | Per Region; ListSecrets can lag about five minutes; secrets scheduled for deletion are excluded unless IncludePlannedDeletion is set |
Config rule secretsmanager-rotation-enabled-check | Secret not scheduled for rotation, or schedule longer than maximumAllowedRotationFrequency | AWS Config recorder and rule evaluations are charged | Configuration change trigger; re-evaluating within 4 hours has no effect |
Config rule secretsmanager-scheduled-rotation-success-check | The calculated rotation date passed without a successful rotation | As above | Results can lag up to 2 days; NOT_APPLICABLE for secrets without rotation |
Config rule secretsmanager-secret-periodic-rotation | Secret not rotated for more than maxDaysSinceRotation (default 90) | As above | Periodic trigger |
| CloudTrail rotation events | RotationStarted, RotationSucceeded, RotationFailed, RotationAbandoned | Event history free for 90 days; a trail for longer | Event history is per Region |
| Rotation function logs | Which of the four steps failed and why | CloudWatch Logs read access | Only for Lambda based rotation |
Run discovery from a read-only role. ListSecrets and DescribeSecret return metadata, not values, so the discovery role needs no GetSecretValue and no kms:Decrypt; keep it that way. Agree the age threshold first. Ninety days is the default for both the Config rule and SecretsManager.4, and Security Hub accepts 1 to 180 days. Pick a lower number for high-impact credentials such as database administrator passwords.
Step-by-step
Steps 1 to 5 are read-only. Step 6 changes configuration but is tested first. Step 7 rotates values and can affect applications.
- Export the inventory, Region by Region. The CLI paginates automatically:
Repeat for every enabled Region, and in every account through a read-only role. Keep the files: they are your baseline and your evidence.aws secretsmanager list-secrets --region eu-west-2 --include-planned-deletion \ --query "SecretList[].{Name:Name,Enabled:RotationEnabled,LastRotated:LastRotatedDate,Next:NextRotationDate,Changed:LastChangedDate,Accessed:LastAccessedDate,Owner:OwningService,Fn:RotationLambdaARN}" \ --output json > secrets-eu-west-2.json - Bucket every secret. Classify each row with the fields AWS documents:
A quick filter for bucket A in one Region:
Bucket Signal Meaning A. Never set up RotationEnabledfalse or absent;NextRotationDatenullRotation is not configured or was cancelled B. Never succeeded Rotation enabled, LastRotatedDatenullAWS sets this field only when rotation completes successfully C. Overdue Rotation enabled, LastRotatedDateolder than the schedule plus a marginRotation has been failing since that date D. Too slow Rotation enabled and working, but AutomaticallyAfterDaysor the schedule exceeds your maximumCompliant in AWS's eyes, not in yours E. Changed by hand LastChangedDaterecent butLastRotatedDateold or nullLikely manual PutSecretValueorUpdateSecret; confirm in CloudTrail before assumingaws secretsmanager list-secrets --query "SecretList[?RotationEnabled!=\`true\`].Name" --output text - Separate managed secrets. Secrets created by another AWS service carry the service ID in the name (
ServiceID!MySecret, for examplerds!) and inOwningService. You can list them with theowning-servicefilter key:
Some managed secrets use managed rotation with no Lambda function, and the managing service may restrict updates. Fix these through the owning service's settings, not by attaching your own rotation function.aws secretsmanager list-secrets --filters Key=owning-service,Values=rds - Turn on the Config rules for continuous findings. Deploy
secretsmanager-rotation-enabled-check(setmaximumAllowedRotationFrequencyto your maximum in days),secretsmanager-scheduled-rotation-success-checkandsecretsmanager-secret-periodic-rotation(setmaxDaysSinceRotation). If Security Hub is enabled, the same checks appear as SecretsManager.1, .2 and .4; set their parameters to the same values so the two views agree. In an organisation, use a delegated administrator and an aggregator so one team sees every account. See the AWS Config Guide and the AWS Security Hub Guide. - Find failing rotations in CloudTrail. For buckets B and C, confirm the failure and its date:
Then open the rotation function's CloudWatch log group and find which of the four steps (aws cloudtrail lookup-events --region eu-west-2 \ --lookup-attributes AttributeKey=EventName,AttributeValue=RotationFailedcreateSecret,setSecret,testSecret,finishSecret) stopped. Check the staging labels withaws secretsmanager describe-secret --secret-id <name>: anAWSPENDINGlabel on a different version fromAWSCURRENTmeans a rotation stopped part way, and AWS documents that a newRotateSecretcall will return an error until that label is removed. - Fix, then test without rotating. Correct the cause (see the table below), then run only the test step:
AWS states that withaws secretsmanager rotate-secret --secret-id <name> --no-rotate-immediatelyRotateImmediatelyfalse, Secrets Manager runs the function'stestSecretstep, creating anAWSPENDINGversion and then removing it. Watch forTestRotationSucceededorTestRotationFailedin CloudTrail. For bucket A, configure rotation with the same flag and acron()schedule, which AWS recommends overrate()when you change an existing schedule, because a previously scheduled rotation might still run. - Rotate. When the test passes and the application owner agrees a window, run
aws secretsmanager rotate-secret --secret-id <name>(the default rotates immediately) and confirmRotationSucceeded. Rotate high-impact secrets in a change window with the owning team watching their error rates.
Where identity fits
Every secret in this inventory is the credential of a non-human identity: a database account, a service user in a SaaS platform, an API client. Rotating the value in Secrets Manager is only half the job. For each secret in buckets A to C, record which system account it authenticates as, which IAM roles can call GetSecretValue on it, and whether a credential-free option exists (IAM database authentication, workload identity federation, a managed integration). A secret you can remove entirely is better than one you rotate well. The How to Map service account dependencies before rotation guide covers finding every consumer before you change a value, which is the step that prevents most rotation outages.
Why rotations fail
| Log symptom | Likely cause (per AWS) | Fix |
|---|---|---|
| Nothing after "Found credentials in environment variables" | The function cannot reach the Secrets Manager endpoint | Interface VPC endpoint or NAT route |
| Nothing after "createSecret" | Network ACLs, endpoint policy or security groups block HTTPS; timeout too short; database API unreachable | Fix network path, raise the timeout |
| "Access to KMS is not allowed" | The function lacks decrypt on the customer managed key | Update the key policy; see the AWS KMS Security Guide |
| "Key is missing from secret JSON" | Secret structure does not match what the function expects | Restore the documented JSON structure |
| "setSecret: Unable to log into database" | The stored credential is already wrong, or a network or TLS issue | Verify the current credential works; check security groups both ways |
| "Unable to import module 'lambda_function'" | Function runtime upgraded from Python 3.7 | Upgrade the function code to a supported runtime |
RotationFailed, pending version "not created by Lambda" | Cross-account or assumed-role rotation without RotationToken | Pass RotationToken to PutSecretValue |
Prioritise the work
Rank by what the secret unlocks, not by age alone. Put first any secret in bucket B or C that grants administrator or write access to a production database or a third-party API, then anything with a recent LastAccessedDate (in use, so worth an attacker's effort), then bucket A secrets for production. Secrets with no LastAccessedDate at all have never been retrieved in that Region: they may be candidates for deletion rather than rotation (Security Hub SecretsManager.3 flags unused secrets, default 90 days).
Verify it worked
- Per secret:
describe-secretshowsLastRotatedDateupdated,NextRotationDateset andAWSPENDINGgone. - Events: a
RotationSucceededevent for each fixed secret, and an EventBridge "Secret Label Updated" event withlabelUpdatedAWSCURRENT. - Findings: the Config rules and Security Hub controls move to compliant. Allow for the up to 2 day lag on the success check.
- Applications: no rise in authentication errors from consumers after rotation; clients using the AWS caching components pick up the new value on their next refresh.
Rollback and common errors
| Situation | Fix |
|---|---|
| Application fails after rotation | AWSPREVIOUS still holds the prior value; move AWSCURRENT back with update-secret-version-stage while you fix the consumer, and check whether the old credential is still valid on the target system |
RotateSecret errors with rotation already in progress | A stuck AWSPENDING label; confirm AWSCURRENT still works, remove the pending label, retry |
| Rotation loops between createSecret and setSecret | AWS points to concurrency throttling or a race; set reserved concurrency to at least 10 or leave it unset, and avoid manual PutSecretValue during rotation |
RotationAbandoned in CloudTrail | Someone wrote a new version during rotation; agree that rotation, not people, owns the value |
| Changing a schedule triggers an unexpected rotation | Use cron() with --no-rotate-immediately when changing an existing schedule |
Hardening afterwards
- Alert on failure, not just on age: an EventBridge rule on
RotationFailed(detail-type "AWS Service Event via CloudTrail") gives you hours, not the days the Config check can take. - Require rotation at creation: add the rotation configuration to the infrastructure as code module that creates secrets, so bucket A cannot grow.
- Stop manual edits: limit
PutSecretValueandUpdateSecretto the rotation function and break-glass roles, and alert on other callers. - Prefer managed rotation: where the owning service offers it, there is no rotation code to break.
- Measure age centrally: track rotation coverage alongside other credential stores. See How to Measure Secret Age and Rotation Coverage.
Small account or large organisation
A single account with a few dozen secrets can run step 1 weekly in a small script and skip AWS Config. An organisation with hundreds of accounts should deploy the three rules through a conformance pack or Security Hub central configuration, aggregate findings to one account and give each finding an owner from a mandatory tag (SecretsManager.5 checks tags). Do the rotation fixes team by team: the hard part is application readiness, not the API.
Metrics
| Metric | Target |
|---|---|
| Secrets with rotation enabled (excluding documented exceptions) | Rising towards 100% |
| Enabled secrets with no successful rotation inside schedule | Zero |
| Secrets older than the maximum age | Zero |
Mean time from RotationFailed to fix | Under one business day |
| Manual value changes outside rotation per month | Falling |
Common failure patterns
- Reporting "rotation enabled" as success: SecretsManager.1 passes for a secret whose function has failed every night for a year. Always pair it with the success check.
- Rotating the secret but not the credential: a custom function that writes a new value without changing it on the target system leaves the old password valid. The
testSecretstep must actually log in. - Single Region view: ListSecrets and CloudTrail event history are Regional; secrets in rarely used Regions are the ones nobody rotates.
- Clearing AWSPENDING without checking AWSCURRENT: AWS's own recovery steps start by confirming the current value works.
- Fixing managed secrets by hand: editing a
rds!secret outside the owning service can break that service.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | IA-5 Authenticator Management; IA-5(1) Password-based Authentication | Change authenticators on a defined schedule and after compromise |
| NIST SP 800-53 Rev 5 | AU-6 Audit Record Review | Review rotation failure events |
| NIST CSF 2.0 | PR.AA-01 | Credentials for services managed by the organisation |
| NIST CSF 2.0 | DE.CM-09 | Monitor for failed rotations and manual changes |
| MITRE ATT&CK | T1555.006 Cloud Secrets Management Stores | Rotation limits how long stolen values stay useful |
Practitioner checklist
- Export ListSecrets for every account and enabled Region, including planned deletions.
- Bucket each secret: never set up, never succeeded, overdue, too slow, changed by hand.
- Set managed secrets aside and handle them through the owning service.
- Deploy the three Config rules, aligned with SecretsManager.1, .2 and .4 parameters.
- Diagnose failures from RotationFailed events and the rotation function's logs.
- Test fixes with
--no-rotate-immediatelybefore any live rotation. - Alert on RotationFailed through EventBridge and restrict manual value changes.
- Track rotation coverage and age as standing metrics.
Standards and references
- AWS: ListSecrets API reference
- AWS: SecretListEntry data type
- AWS Config: secretsmanager-scheduled-rotation-success-check
- AWS Config: secretsmanager-secret-periodic-rotation
- AWS Security Hub: Controls for Secrets Manager
- AWS: Troubleshoot Secrets Manager rotation
- AWS: Secrets Manager CloudTrail entries
- AWS CLI Reference: secretsmanager rotate-secret
Related NHI Mgmt Group resources: AWS Secrets Manager · Credential Rotation at Scale · How to Rotate database credentials without downtime · Secret Rotation Outage Troubleshooting Guide