Buildkite credentials: Identify, Revoke and Contain Leaked CI Tokens

A leaked Buildkite token can trigger builds, read job environments and build logs, or register a rogue agent that receives your jobs and their secrets, so the response depends entirely on which of Buildkite's many token types escaped. Buildkite issues API access tokens to people, agent tokens to clusters, short-lived job and session tokens to running agents, and separate tokens for Package Registries and GraphQL portals, each with its own prefix, scope, lifetime and revocation path. This guide is a leak response reference: how to tell the types apart, what each one reaches, where misuse shows up in Buildkite's audit log and API Access Audit, the exact revoke and rotate steps, and the controls that stop the next leak. It is written for platform, CI/CD and security operations teams running Buildkite Pipelines with self-hosted agents. For the wider platform hardening picture see the CircleCI, Buildkite and TeamCity Security Guide; this page goes deeper on credentials and incidents.

Key takeaways

• Read the prefix first. bkua_ is a user's API access token, bkct_ a cluster agent token, bkar_ an unclustered agent token and bkaj_ a job token; each has a different owner and revocation route.
• Revoking an agent token does not disconnect agents. Buildkite documents that revocation only blocks new registrations, so stop or replace any agent that may be attacker controlled.
• Organisation administrators can cut off a user's token without the user. "Remove Organization from Token" on the API Access Audit page is immediate and the owner cannot add the organisation back.
• GraphQL access is unscoped. A token with "Enable GraphQL API access" has full access to everything its owner can do, so treat any leaked token with that scope as an account compromise.
• The token is rarely the only thing exposed. A leaked agent or job token means the job environment and the secrets it fetched may be exposed too, so rotate downstream deployment credentials, starting with production.

How identity and access work in Buildkite

Buildkite separates the control plane (buildkite.com, the REST API at api.buildkite.com and GraphQL at graphql.buildkite.com) from the agents that run jobs on your infrastructure. People authenticate to the web interface with a password, two-factor authentication or SSO, and to the APIs with personal API access tokens. Agents authenticate with an agent token, which they exchange for internal session and job tokens. Buildkite publishes a token reference table, which the table below condenses.

OIDC tokens issued by buildkite-agent oidc request-token carry no Buildkite prefix and are presented to cloud providers, not to Buildkite; they last five minutes by default. A leaked OIDC token is a short window against whatever cloud role trusts it, which is the strongest argument for moving deployment credentials to federation (see the Workload Identity Federation Guide).

Admin roles and privileged access

• Organisation administrator: manages settings, SSO, members, teams and API security settings, and is the only role that sees the API Access Audit page and the audit log. Any leaked token belonging to an administrator inherits that reach, subject to its scopes.
• Cluster maintainer: can create, edit and revoke agent tokens for a cluster. Treat maintainers as privileged for the queues their cluster serves.
• Team maintainer and team member: control pipeline access within teams; a token can only reach pipelines its owner can reach.
• Break-glass account: Buildkite's incident guidance recommends a dedicated administrator account whose SSO requirement is optional, protected by 2FA, stored in a vault with multi-person approval, never used day to day and never given API access tokens, with an audit log alert on any event it performs.

Two nuances matter during an incident. API access tokens are not subject to SSO requirements, so disabling and re-enabling SSO (which forces every user to re-authorise) does not cut off a token. And organisations do not own user accounts: logging a user out of every Buildkite session requires sending the user IDs to Buildkite support.

Integrations, API tokens and service accounts

API access tokens belong to individual users, so "service" automation usually runs on a person's token or a shared bot user. Each token carries REST scopes chosen per feature with READ, WRITE and DELETE permissions. The scopes that matter most after a leak:

Lower-risk alternatives exist. OAuth Token Exchange lets a service swap a signed JWT for a short-lived bktx_ token; GraphQL portals expose only a fixed set of operations; bk auth login --scopes "read_only" limits CLI tokens; and job acquisition tokens keep the long-lived agent token inside the stack controller. The CI/CD Pipeline Identity Security Guide covers the general pattern.

Logging and detection

Secret scanning coverage

Buildkite is a GitHub secret scanning partner. GitHub's pattern data lists ten Buildkite secret types, all scanned in public repositories and in private repositories with GitHub Secret Protection, none with validity checks:

For bkua_ tokens found in public repositories, GitHub notifies Buildkite, which automatically revokes valid tokens and notifies the owner and organisations. For any other type, Buildkite asks you to contact support. Note the gap: the API access token is the one type without push protection, and the Gitleaks default configuration (checked 28 October 2026) has no Buildkite rule, so add a custom rule matching the documented prefixes (bk followed by ua, tx, ct, ar, aa, aj, jat, pt, pat or ps and an underscore). Buildkite also tells you not to wait for a scanning notice before acting. See the Git Secret Scanning Guide.

Where misuse shows up

• API Access Audit (organisation administrators): lists every token with access to the organisation, its scopes, creation time and last use. Filter by user, scope, IP or admin status, open a token to see its most recent request, and export CSV with Last Used At and Last Request IP. A leaked token with a recent last-use IP outside your egress ranges is your first misuse signal.
• Audit log (Enterprise): stored indefinitely, searchable in the interface for 12 months with type:, actor:, pipeline: and subject: terms (last 90 days, up to three terms), and queryable through REST and GraphQL beyond that. Stream it to your SIEM through Amazon EventBridge before you need it. An actor: term matches users only, so events by agents or API applications need a separate query.
• Agent views: unexpected agents on a queue, agents from unknown hosts or IPs, and jobs picked up by agents you did not start.
• Builds: builds triggered outside normal sources, builds on unusual branches, and new pipeline schedules.

What to alert on

Metrics that show it is working

• Mean time to contain: from leak alert to organisation access removed or agent token revoked and agents replaced.
• Tokens without expiry: share of agent tokens and API tokens with no expiry, trending to zero.
• Unused tokens: API tokens with no use in 30 days that still have organisation access.
• GraphQL-scoped tokens: count of tokens with unscoped GraphQL access, each with a named justification.
• Static cloud keys in jobs: pipelines still using long-lived cloud credentials instead of OIDC.

Leak response

Buildkite asks customers to email [email protected] for any security incident and [email protected] for an exposed secret. Start that in parallel with containment.

1. Classify the token by prefix using the reference table. For a bkua_ token, a GET to https://api.buildkite.com/v2/access-token using the exposed value returns its owner, scopes and expiry; run it from a controlled host and never paste the value into chat tools, tickets or AI assistants.
2. Contain an API access token: as an administrator, open the token on the API Access Audit page and select Remove Organization from Token; the owner is notified and cannot re-add your organisation. The owner (or anyone holding the value) can revoke the token entirely with a DELETE to the same access-token endpoint or from Personal Settings. If the owner is compromised, remove the user from the organisation and disable them in your identity provider.
3. Contain an agent token: revoke it on the cluster's Agent Tokens page, through REST or with the GraphQL clusterAgentTokenRevoke mutation (agentTokenRevoke for unclustered tokens). Then stop every agent registered with it and replace persistent agents from a known-good image, because revocation does not disconnect them. Issue a new token, ideally with an expires_at and Allowed IP Addresses, and roll it out through your secret store.
4. Job and session tokens: these die with the job or connection, so containment means stopping the job or agent and treating the job's environment as exposed.
5. Portal, registry and other tokens: replace a long-lived portal token from the portal's Security page, rotate portal secrets using the second slot, delete registry tokens, and contact support for anything you cannot revoke yourself.
6. Stop unwanted builds: use Disable Incoming GitHub Webhook Processing on affected pipelines, disable GitHub Actions compatibility triggers separately, and cancel scheduled or running builds. If a webhook URL leaked, ask support to reissue it, because the URL contains its secret.
7. Rotate downstream secrets: rotate everything the token, agent or job could read, starting with production deployment credentials, in Buildkite secrets or your external store. If a secret appeared unredacted in a job log, rotate it and tell Buildkite security.
8. Remove persistence: search the audit log across the incident window for the events in the alert table, and outside Buildkite review new deploy keys, webhooks, app installations, repository hooks, agent hooks, agent configuration, SSH keys and scheduled tasks on agent hosts.
9. Close out: recover only when every access path created during the window is removed and monitoring shows no new agent connections, token use or builds. Record the root cause in the Leaked Credential Response Playbook format.

The Codecov incident shows why step 7 is not optional: an altered uploader script sent CI job environment variables to an attacker, and those variables held cloud keys, signing keys and git credentials that were used against customers. See the Codecov breach analysis.

Hardening baseline

1. Inventory: export the API Access Audit CSV, list agent tokens per cluster, and assign an owner and purpose to each. Flag tokens with GraphQL access, read_job_env, no recent use, or no expiry.
2. Shorten lifetimes: keep API token expiry short (the default is seven days), create agent tokens through the API with expires_at, and prefer OAuth Token Exchange and job acquisition tokens for services and ephemeral agents.
3. Restrict networks: set Allowed IP Addresses on agent tokens and API tokens, and on Enterprise set the organisation API IP allowlist (it also applies to the Buildkite remote MCP server) and inactive token revocation.
4. Remove static cloud keys: replace them with OIDC, so a leaked job yields a five-minute token rather than a long-lived key.
5. Keep secrets out of Buildkite payloads: never put secrets in pipeline settings or the env block, and avoid interpolating them during pipeline upload, which sends them to Buildkite.
6. Prepare the response: write the runbook, create and quarterly test the break-glass account, stream the audit log to the SIEM, and capture triggering webhooks yourself if you need a durable record (Buildkite keeps them about a week for troubleshooting only).
7. Test before you revoke at scale: roll new agent tokens to one queue first, confirm agents reconnect, then revoke the old token; keep the old token until the new one is proven, because a revoked token cannot be restored.

Small teams without Enterprise lack the audit log, so the API Access Audit page, short token expiry and SSO/2FA enforcement carry most of the load. Larger organisations should separate clusters per trust level (production deploy agents never shared with pull request builds), require OAuth Token Exchange for services, and alert on audit events in the SIEM.

Framework mapping

Common misconfigurations

• Agent tokens that never expire: tokens created in the interface have no expiry and are often baked into AMIs or Helm values for years.
• One cluster for everything: a job token can read metadata, annotations and artifacts of other jobs in the same cluster, so pull request builds sharing a cluster with production deploys widen every leak.
• Automation on a person's token with GraphQL enabled: unscoped access tied to a human who may leave, and untouched by SSO changes.
• Revoke and walk away: revoking an agent token while the attacker's agent stays connected and keeps receiving jobs.
• Secrets in pipeline YAML: values in the env block are stored by Buildkite and appear in the job's uploaded pipeline.
• No audit trail when it matters: audit log streaming never set up, and webhook payloads assumed to be retained.

Practitioner checklist

• Keep a prefix-to-runbook table for all Buildkite token types, including support and security contacts.
• Add custom scanner rules for every bk*_ prefix and enable push protection wherever code lives.
• Export the API Access Audit CSV monthly and remove organisation access from unused or over-scoped tokens.
• Create agent tokens through the API with expires_at and Allowed IP Addresses, one per environment.
• Move ephemeral agents to job acquisition tokens and services to OAuth Token Exchange.
• Replace static cloud keys in jobs with OIDC federation.
• Separate clusters by trust level and never share production deploy agents with untrusted builds.
• Stream the audit log to the SIEM and alert on token creation, pipeline changes and new notification services.
• Maintain and quarterly test a break-glass administrator account.
• After any leak, rotate downstream secrets and rebuild affected agents from a known-good image.

Standards and references

• Buildkite: Token security
• Buildkite: Incident response
• Buildkite: Managing API access tokens
• Buildkite: Agent tokens
• Buildkite: Audit log
• Buildkite: Secrets risk considerations
• GitHub Docs: Supported secret scanning patterns
• NIST: SP 800-53 Rev 5, Security and Privacy Controls

Related NHI Mgmt Group resources: CircleCI, Buildkite and TeamCity Security Guide · CI/CD Pipeline Compromise Playbook · Leaked Credential Response Playbook · Developer Secret Scanning Triage