The January 2023 CircleCI incident forced every customer to assume that all secrets stored on the platform had been stolen, and many teams discovered they could not even list them. A CI/CD platform holds cloud keys, deploy keys, package publishing tokens and API tokens for every system it deploys to, so its configuration decides how bad a platform or account compromise becomes. This guide is for DevOps, platform and security teams using CircleCI cloud. It covers organisation access, contexts and their restrictions, OIDC tokens, API tokens and SSH keys, audit logs and detection, a hardening baseline, and the mass rotation playbook a provider breach demands. Pipeline compromise response in general is covered in the CI/CD Pipeline Compromise Playbook.
Key takeaways
- Replace stored cloud keys with OIDC. CircleCI issues
$CIRCLE_OIDC_TOKEN,$CIRCLE_OIDC_TOKEN_V2and$CIRCLE_OIDC_TOKEN_V3; V3 is recommended for new setups and lets cloud trust policies pin the project, repository and Git reference. - Restrict every context that holds a deploy secret. Security groups, project restrictions and expression restrictions combine with logical AND; without them, any organisation member's pipeline can use the context.
- Know your secret inventory before you need it. CircleCI told customers to rotate OAuth tokens, project API tokens, SSH keys and every stored secret after its 2023 incident. An inventory with owners turns that from weeks into hours.
- Audit logs need collecting. Organisation admins can request up to a year of audit logs in 30-day windows, three requests a day on paid plans; streaming to S3 needs the Scale plan.
- Masking is not protection. Secrets shorter than four characters, boolean-like values and anything printed under
set -xcan appear in job output, and CircleCI says masking can be bypassed.
How identity and access work in CircleCI
CircleCI organisations map to a version control organisation (GitHub, Bitbucket or GitLab) or exist as standalone organisations. Users sign in through their VCS identity or CircleCI accounts, and the organisation type affects which security features are available: for example, security group restrictions on contexts are available only for github type organisations integrated through the GitHub OAuth app.
Inside a pipeline, jobs reach secrets and other systems through:
- Project environment variables: set under Project Settings > Environment Variables, available to every job in that project.
- Contexts: organisation-level sets of environment variables, up to 100 per context, attached to jobs with the
contextkey in the workflow configuration and injected at runtime. - OIDC ID tokens: signed tokens issued per job by
https://oidc.circleci.com/org/<organization_id>that cloud providers exchange for short-lived credentials. - SSH keys: checkout keys and deploy keys for the VCS, plus additional project SSH keys. CircleCI holds the private key; the target system holds the public key.
- API tokens: personal and project tokens for the CircleCI API, used by scripts and other tools.
| OIDC token | What the sub claim identifies | Use |
|---|---|---|
$CIRCLE_OIDC_TOKEN | Organisation, project and user | Existing trust policies; pin project via the oidc.circleci.com/project-id claim |
$CIRCLE_OIDC_TOKEN_V2 | Adds repository (vcs-origin) and Git reference (vcs-ref) | Restrict roles to a repository and branch |
$CIRCLE_OIDC_TOKEN_V3 | Adds checkout source and reference, and config source and reference, separated by : | New setups; prevents a job running config from another source assuming the role |
The default audience is the organisation ID. On AWS, the STS condition key for oidc.circleci.com/project-id lets a trust policy accept only one project. Fork builds receive OIDC tokens only if Pass secrets to builds from forked pull requests is enabled, which should stay off. See the GitHub Actions OIDC Federation Hardening Guide for trust policy pitfalls that apply equally here.
Admin roles and privileged access
- Organisation admins: have read and write access to all projects and unrestricted access to all contexts. Only they can add security group, project or expression restrictions, request audit logs and change organisation settings. Keep admins few and named, and protect their VCS or SSO identities with phishing-resistant MFA.
- Context creation: organisation members can create contexts. A context security setting for
githubtype organisations requires all newly created contexts to be group-restricted, so a secret cannot sit in an unrestricted context by accident. - Restricted context editing: adding and deleting environment variables on a restricted context is limited to members of the context's groups.
- Approval jobs: an approval job before a job using a restricted context means a member of the context's security group must approve before deploy secrets are released.
- Team sync delay: CircleCI syncs GitHub teams and LDAP groups every few hours, so removing someone from a team does not instantly remove context access. For urgent removals, also remove the user from the organisation.
The CircleCI incident itself began with privileged access: malware on an engineer's laptop stole a valid, 2FA-backed SSO session, and the engineer could generate production access tokens. Your own admin accounts face the same risk. Bind admin sessions to managed devices and treat CircleCI admin as production access. See the CI/CD Pipeline Identity Security Guide.
Integrations, API tokens and service accounts
Context restrictions
| Restriction | Passes when | Best for |
|---|---|---|
| Security group | The user who triggered the pipeline is in at least one assigned group (default group is All members) | Limiting production deploy secrets to the release team |
| Project restriction | The job runs in one of the listed projects | Stopping other repositories in the organisation from using a context |
| Expression restriction | Every expression on pipeline values evaluates to true | Branch and source rules, for example pipeline.git.branch == "main" and not job.ssh.enabled and not (pipeline.config_source starts-with "api") |
All three combine with AND, and a failing check ends the workflow with Unauthorized. The example expression from CircleCI's documentation is a good default for production contexts: it blocks use from feature branches, from jobs rerun with SSH enabled, and from configuration supplied through the API.
API tokens and SSH keys
| Credential | Scope | Handling |
|---|---|---|
| Personal API token (web app) | Full read and write as the user, across the user's organisations | Set the shortest practical expiry date at creation; do not use for automation owned by a team |
| Personal token via OAuth dynamic client registration | Read, Write or Admin as chosen during authorisation | Prefer for tools that support it |
| Project API token | Status, Read Only or Admin (Read Only and Admin are API v1 only) | Status for badges; avoid Admin; record owner |
| Checkout and deploy keys | Repository read, or write for deploy keys with write access | Read-only where possible; rotate after incidents |
| Additional project SSH keys | Whatever the target server allows | Restrict on the target (forced commands, source IPs); rotate both halves |
CircleCI's secure secrets handling guidance also covers your workstations: avoid env and printenv, keep tokens out of shell history, put API tokens in a chmod 0600 header file read with curl --header @file, and use circleci context store-secret ... <<< "$MY_SECRET" so the value is not visible in process listings. Where jobs must reach private systems, the IP ranges feature (circleci_ip_ranges: true, Performance or Scale plans) lets you allow-list CircleCI egress addresses rather than opening services to the internet.
Logging and detection
All organisations can request audit logs. An organisation admin selects a date range under Request audit logs; the earliest start is one year back and each request covers a 30-day window. Free plans get one request a day and paid plans three; downloads are CSV and stay available for 30 days. Scale plan customers can stream audit logs to their own S3 bucket, authenticated with CircleCI OIDC. Duplicate events can occur, so de-duplicate on the id field.
| Behaviour | Audit events | Why it matters |
|---|---|---|
| Secret access and change | context.secrets.accessed, context.env_var.store, context.env_var.delete, project.env_var.create, project.env_var.copy | Who read or changed secrets, and copying of variables between projects |
| Restriction weakened | context.restriction.delete, context.group_remove, context.group_add | Removing a branch or group restriction exposes deploy secrets |
| New credentials | project.api_token.create, project.ssh_key.create, checkout-key.create, release_integration-token-create | Persistence through a credential that survives user removal |
| Settings and roles | organization.settings.update, project.settings.update, role_grant-update, project_group_role_grant.create | Changes to fork settings, permissions and access |
| Pipeline control | trigger-create, webhook-create, schedule-create, workflow-job-context-request | New ways to start pipelines that receive secrets |
| Supply chain | orb.publish.release, orb.version.promote | Changes to orbs your pipelines and others consume |
| Audit tampering signals | audit_log.requested, audit_log.download_url.generated, org_member.remove | Unexpected log pulls or removal of members during an incident |
Pair CircleCI events with the logs of the systems your pipelines reach: cloud trail events for roles assumed through OIDC, package registry publish logs, and VCS audit logs. CircleCI's own incident advice told customers to check GitHub audit logs for unexpected activity such as repo.download_zip; the GitHub Audit Log Detection Guide shows how.
Metrics
- Share of cloud deployments using OIDC rather than stored keys (target 100%).
- Share of contexts holding production secrets with both a group and an expression restriction.
- Number of personal API tokens without an expiry or older than 90 days.
- Secret inventory completeness: stored secrets with a named owner and a documented rotation procedure.
- Time to rotate all CircleCI-held secrets in a rehearsal.
Hardening baseline
- Inventory: list every context, project environment variable, SSH key and API token with its owner and the system it unlocks. Export it to your secrets register.
- OIDC first: move AWS, Google Cloud and Azure deployments to OIDC with
$CIRCLE_OIDC_TOKEN_V3and trust policies pinned to project, repository and branch; delete the stored keys. - Restrict contexts: enable the setting that forces new contexts to be group-restricted, add project and expression restrictions to production contexts, and add approval jobs for deployments.
- Forks: keep Pass secrets to builds from forked pull requests off, especially for public projects.
- Tokens and keys: set expiry on personal tokens, remove Admin project tokens, make deploy keys read-only where possible.
- Job hygiene: no
set -xaround secrets, no printing environment variables, and avoid SSH reruns on jobs with production contexts (the expressionnot job.ssh.enabledenforces this). - Network: use IP ranges or self-hosted runners for access to private systems, which CircleCI also recommended after its incident for privileged workloads.
- Logging: request audit logs on a schedule or stream them, and alert on the events above.
- Test and roll back: add restrictions to one context at a time in a non-production project first, watch for
Unauthorizedfailures, and keep a record of the previous restriction set so it can be restored quickly.
Mass rotation after a provider breach
CircleCI's January 2023 guidance is the template. It said to assume every secret stored during the affected period had been accessed and to rotate OAuth tokens, project API tokens, SSH keys and all environment variables and context secrets, then investigate activity from 16 December 2022 to the date of rotation. A rehearsed sequence:
- Freeze and scope: pause non-essential pipelines, export the secret inventory, and rank secrets by blast radius (cloud admin, package publishing, production databases first).
- Rotate at the source: issue new credentials in the target system, update the context or project variable, then revoke the old credential. Rotation that only updates CircleCI leaves the stolen value valid.
- Replace, do not re-store: where possible, switch the integration to OIDC during rotation instead of storing a new long-lived key.
- Rotate CircleCI-side credentials: personal and project API tokens, checkout and deploy keys, additional SSH keys, and VCS OAuth authorisations.
- Hunt: review cloud, VCS, registry and database logs for use of old credentials from the provider's stated exposure date onwards.
- Prove completion: track each secret to "revoked" with evidence, and report the percentage complete daily. The Secrets Rotation Runbook Template provides a structure.
Framework mapping
| Framework | Reference | CircleCI control |
|---|---|---|
| NIST CSF 2.0 | PR.AA-01, RS.MA-01, RS.AN-03, RC.RP-01 | Managed credentials, incident response execution, scoping what happened, recovery through rotation |
| NIST SP 800-53 Rev. 5 | IA-5, AC-6, SC-12, AU-6, IR-4, IR-5 | Authenticator management, least privilege, key management, audit review, incident handling and monitoring |
| MITRE ATT&CK v20 | T1552, T1552.001, T1528, T1539, T1078.004, T1098.001, T1195.002, T1199 | Stored and file-based credentials, stolen tokens and session cookies, cloud accounts and added credentials, supply chain and trusted relationship |
Common misconfigurations
- Unrestricted contexts with production keys: the default All members group lets any member's pipeline in any project use the secrets.
- Static cloud keys kept "for now": long-lived AWS keys in project variables were exactly what customers had to rotate in a hurry after the CircleCI breach, where exfiltrated data included environment variables, tokens and keys, and encryption keys taken from a running process.
- OIDC trust on organisation only: a trust policy that checks only the audience accepts tokens from every project in the organisation.
- Personal tokens owned by leavers: automation breaks when the person leaves, or worse, keeps running on a token nobody owns.
- Secrets passed to fork builds: enabling secrets for forked pull requests gives outside contributors a way to read them.
- Rotation only in CircleCI: updating the stored value without revoking the old credential at the source leaves the stolen copy working.
Practitioner checklist
- Keep a live inventory of contexts, variables, SSH keys and API tokens with owners.
- Move cloud access to OIDC with V3 tokens and pinned trust policies.
- Require group restrictions on new contexts and add expression restrictions to production contexts.
- Add approval jobs before deployments that use restricted contexts.
- Keep secrets for forked pull requests disabled.
- Set expiry dates on personal API tokens and remove Admin project tokens.
- Request or stream audit logs and alert on context, token and settings changes.
- Rehearse a full secret rotation and measure how long it takes.
- Revoke old credentials at the source system, not just in CircleCI.
- Protect admin accounts with phishing-resistant MFA on managed devices.
Standards and references
- CircleCI: CircleCI incident report for January 4, 2023 security incident
- CircleCI: Using contexts
- CircleCI: Using OpenID Connect tokens in jobs
- CircleCI: Audit logs
- CircleCI: Managing API tokens
- CircleCI: Secure secrets handling
- NIST: SP 800-53 Rev. 5
- MITRE ATT&CK: Unsecured Credentials
Related NHI Mgmt Group resources: CI/CD Pipeline Compromise Playbook · CircleCI Breach 2023 · CI/CD Security Guide · Secrets Rotation Runbook Template