A CI/CD server holds more standing production access than almost any other system, and it drifts: plugins fall behind, stored credentials outlive the projects that needed them, build agents become long-lived pets and pipeline definitions change without a second pair of eyes. This health check is a recurring, measurable review of a live continuous integration and delivery server and its build agents, whether self-hosted or delivered as a service. It turns the questions a platform security lead or auditor would ask into thirteen metrics with formulas, thresholds and evidence, so each quarter can be compared with the last. It is written for build and release platform owners, DevSecOps leads, internal audit and the CISO's office.
This page measures; it does not explain how to harden pipelines. For control design, read the CI/CD Security Guide; for a step-by-step credential audit of a self-hosted CI server, use how to audit CI server credentials and secrets; return to them for every finding this review raises.
What this health check covers
The review covers the CI/CD server or service, its stored credentials, its build agents and runners, the pipeline definitions it executes and the production deployments it performs. Source control and artefact registries are reviewed in their own checks, linked below, and only appear here where the CI server depends on them.
- Who runs it: the build platform owner collects the figures; the DevSecOps or application security lead reviews them; the head of engineering and the CISO or delegate sign off red items.
- How it differs from the hardening guides: those explain what to configure. This check asks, with numbers, whether the configuration is still true across every project this quarter: how many credentials are long-lived, how many jobs run on persistent agents, how many production deployments carried verified provenance.
- Why the identity angle matters: most metrics below measure non-human identities: stored tokens, deploy keys, cloud credentials and the agents that use them. See the CI/CD Pipeline Identity Security Guide.
- Common search terms it answers: "CI/CD security audit checklist", "CI/CD security KPIs", "build server security review" and "how to measure pipeline credential hygiene".
Key takeaways
- Credentials are the headline risk. Count long-lived, broadly scoped and unused stored credentials every quarter; the OWASP Top 10 CI/CD Security Risks lists insufficient credential hygiene as CICD-SEC-6.
- Measure federation, not intention. The share of cloud deployment jobs using short-lived federated tokens shows real progress away from static keys.
- Persistent agents carry state between builds. Track the share of jobs run on ephemeral agents and any untrusted trigger reaching an agent with secrets.
- Deployments need evidence. Count production deployments with verified provenance and an approval gate, not the pipelines that could produce it.
- Thresholds are a starting point. The green, amber and red bands are NHI Mgmt Group recommendations; agree your own and keep them stable.
How to use this template
Copy the metrics table and the findings log into your own tracker, then run this CI/CD server health check at the cadence set out below. Collect each figure from the data sources described, score it against the green, amber and red thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review so you can compare results quarter on quarter: the trend matters as much as any single score.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| Server, plugin and agent update status | Monthly | Build platform owner | DevSecOps lead | Version inventory, outstanding security updates with age |
| Stored credential inventory and rotation | Monthly | Build platform owner | Secrets management or NHI owner lead | Credential export (metadata only, never values), rotation dates, owners |
| Full metrics review (this page) | Quarterly | Build platform owner | Head of engineering and CISO or delegate | Metrics table with figures, raw exports, findings log |
| Administrator and permission review | Quarterly | Build platform owner | Security risk or GRC | Admin role membership export, approved list, changes since last review |
| Independent sample test | Annually | Internal audit or second line risk | Audit committee | Sample of deployments traced from commit to production, credentials re-checked |
| Event-driven review | After a pipeline compromise advisory, platform migration or major upgrade | DevSecOps lead | CISO or delegate | Affected credentials, jobs and agents, rotation evidence |
The people who administer the CI server should not be the only ones signing off figures about its own admin roles and credentials; this follows the separation of duties principle in NIST SP 800-53 Rev. 5 AC-5. NSA and CISA guidance on defending CI/CD environments recommends regular auditing of CI/CD processes, and this quarterly scorecard is one way to evidence it.
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. The formulas are recommended methods too: record the numerator, denominator and data source once and reuse them every quarter. "Stored credential" means any secret held by the CI server or its credential store (password, token, private key, cloud access key, certificate); export metadata only, never values.
| ID | Metric | Formula | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| CI-01 | Platform update currency | CI server instances, plugins or extensions and agent software with no security update outstanding for more than 30 days ÷ all installed components | CI server version and plugin inventory, vendor or project security advisory feed | 100% | 95% to below 100% | Below 95%, or any component with a known exploited vulnerability |
| CI-02 | Administrators on the approved list | Holders of server-wide administration rights who are on the approved list and use a named account ÷ all holders of those rights | CI server role and permission export, identity provider group membership | 100% and five or fewer holders | 100% but above five holders | Any holder not on the approved list, or any shared admin account |
| CI-03 | Long-lived stored credentials | Stored credentials older than the rotation period in your secrets standard ÷ all stored credentials | CI credential store metadata export (creation or last update date) | Below 5% | 5% to 20% | Above 20% |
| CI-04 | Globally scoped credentials | Stored credentials available to every project or pipeline ÷ all stored credentials | CI credential store export with scope (global, folder, project, environment) | Below 5% | 5% to 15% | Above 15%, or any production deployment credential at global scope |
| CI-05 | Unused stored credentials | Stored credentials not referenced by any job run in the last 90 days ÷ all stored credentials | Credential store export joined to job run history or credential usage tracking | Below 5% | 5% to 15% | Above 15% |
| CI-06 | Federated cloud deployment | Cloud deployment jobs authenticating with short-lived federated tokens (for example OpenID Connect workload identity federation) ÷ all cloud deployment jobs | Pipeline definitions, cloud provider sign-in logs for CI identities | 90% or more | 60% to below 90% | Below 60% |
| CI-07 | Ephemeral build agents | Jobs run on an agent or runner created fresh for the job and destroyed afterwards ÷ all jobs | Job run history with agent identifiers, agent pool configuration | 90% or more | 60% to below 90% | Below 60% |
| CI-08 | Untrusted triggers on privileged agents | Count of jobs triggered by forks, external contributors or unreviewed branches that ran on agents with access to stored secrets or production networks | Job run history with trigger type, agent pool and credential binding | 0 | None (no amber band) | 1 or more |
| CI-09 | Pipeline definition changes reviewed | Changes to pipeline definitions and shared pipeline libraries merged with approval from someone other than the author ÷ all such changes | Source control merge records for pipeline files, CI server configuration audit log | 100% | 95% to below 100% | Below 95% |
| CI-10 | Production deployments with verified provenance | Production deployments of artefacts whose signed build provenance was verified before release ÷ all production deployments | Deployment records, provenance verification or admission policy logs | 95% or more | 70% to below 95% | Below 70% |
| CI-11 | Secrets exposed in build output | Build logs and artefacts with a confirmed secret detection ÷ build logs and artefacts scanned; report the count and time to rotation | Secrets scanner results over build logs and artefact stores | 0, all rotated within 24 hours | Any, all rotated within 72 hours | Any not rotated within 72 hours |
| CI-12 | Ownerless and dormant pipelines | Pipelines or jobs with no named owning team, or no successful run in 180 days, that still hold credential bindings ÷ all pipelines | CI job inventory, ownership metadata, job run history | Below 2% | 2% to 10% | Above 10% |
| CI-13 | Audit log delivery completeness | Hours in the quarter with CI server audit events received in the SIEM within 15 minutes ÷ hours in the quarter | SIEM ingestion metrics for the CI audit log source | 99.5% or more | 98% to below 99.5% | Below 98%, or any gap over 24 hours |
Report the absolute count beside every percentage. A single red privileged item (a production cloud key at global scope, a shared admin account, a fork-triggered job on a deployment agent) makes the line red whatever the percentage says.
How to collect the data
Export raw rows rather than dashboard percentages, keep the exports with the findings log and record the query or report used, so the next quarter is calculated the same way.
- Freeze the date: agree the measurement date and take every export within one working day of it.
- Inventory the platform: export the server version, the installed plugin or extension list with versions, and the agent pool list with agent software versions. Compare with the project's or vendor's security advisories for CI-01.
- Export the credential store metadata: credential identifier, type, scope, owner, created and last updated dates. Never export values. Join it to job run history to find credentials no job has used in 90 days (CI-05). See How to Audit CI/CD Secrets Scope.
- Check deployment identity: from pipeline definitions, list jobs that deploy to cloud accounts and classify each as federated or static credential; confirm with cloud provider sign-in logs, which show whether the CI identity used a federated token or an access key. See the Workload Identity Federation Guide.
- Classify agents and triggers: from job run history, tag each run with agent type (ephemeral or persistent), trigger type and the credential bindings it received. CI-07 and CI-08 come from the same extract.
- Trace deployments: take the production deployment records for the quarter and check each against provenance verification or admission logs for CI-10. Sample 25 in the annual review and trace each from commit to running workload.
- Scan build output: run or collect secrets scanner results over build logs and stored artefacts for the quarter. See How to find secrets in CI logs and build artefacts.
- Sample the reds: for every red metric, check five to ten records directly in the CI server to confirm the figure is real.
Red flags that need action now
- Any CI server or plugin with an outstanding update for a vulnerability listed in the CISA Known Exploited Vulnerabilities catalogue.
- A shared or generic administrator account, or an administrator not on the approved list.
- A production cloud access key stored at global scope, or any static cloud key older than one year.
- Any job triggered from a fork or external contributor that ran on an agent holding production secrets, the pattern behind poisoned pipeline execution (OWASP CICD-SEC-4). See MITRE ATT&CK T1677 Poisoned Pipeline Execution.
- A confirmed secret in a build log that has not been rotated.
- Pipeline definition or shared library changes merged without a second approver.
- Audit logging disabled, or a gap of more than 24 hours in CI audit events reaching the SIEM.
- An unexplained new agent registered to a production agent pool. Treat it as a possible compromise and follow the CI/CD Pipeline Compromise Playbook.
Findings log
Keep one running log across quarters so repeat findings are visible. Escalate a finding that stays red for two consecutive quarters to the CISO, and one that repeats three times to the risk register.
| ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-CICD-01 | [n] stored credentials older than rotation period, including [n] production | CI-03 | [Red / Amber] | [Build platform owner] | [Date] | [Open / In progress / Closed] |
| HC-CICD-02 | [n] cloud deployment jobs still using static access keys | CI-06 | [Severity] | [Platform engineering lead] | [Date] | [Status] |
| HC-CICD-03 | [n] fork-triggered runs on agent pool [name] with secret access | CI-08 | [Severity] | [DevSecOps lead] | [Date] | [Status] |
| HC-CICD-04 | [n] production deployments without verified provenance | CI-10 | [Severity] | [Release manager] | [Date] | [Status] |
| HC-CICD-05 | [Finding] | [Metric ID] | [Severity] | [Owner] | [Date] | [Status] |
Remediation map
| Finding type | Where to fix it |
|---|---|
| Outdated server, plugins or agents; excess administrators | CI/CD Security Guide: runner and build server isolation |
| Long-lived, global or unused stored credentials | How to audit CI server credentials and secrets; Secrets Management Guide |
| Static cloud keys in deployment jobs | Workload Identity Federation Guide; Dynamic Secrets and Ephemeral Credentials Guide |
| Persistent agents, untrusted triggers on privileged agents | Self-hosted CI/CD Runners and Build Agents |
| Unreviewed pipeline definition changes | OWASP CI/CD Top 10 CICD-SEC-4 Poisoned Pipeline Execution Guide |
| Deployments without verified provenance | SLSA and Build Provenance Guide; Sigstore Guide |
| Secrets in build logs and artefacts | How to find secrets in CI logs and build artefacts |
| Ownerless pipelines holding credentials | CI/CD Pipeline Identity Security Guide |
Related guides
- CI/CD Security Guide: the control design this check measures.
- CI/CD server maturity model: where the platform sits on a five-level scale.
- CI/CD Pipeline Compromise Playbook: response when a red flag turns out to be an attack.
- Source control platform health check and Artefact and container registry health check: the companion checks either side of the pipeline.
- Software Supply Chain Maturity Model: the wider programme view.
References
- OWASP: Top 10 CI/CD Security Risks
- NIST: SP 800-204D Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
- NSA and CISA: Defending Continuous Integration/Continuous Delivery (CI/CD) Environments
- OpenSSF SLSA: Build track basics (v1.2)
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls (AC-5 Separation of Duties, IA-5 Authenticator Management, CM-3 Configuration Change Control, SR-4 Provenance)
- NIST: Cybersecurity Framework 2.0 (PR.AA-05, PR.PS-01, PR.PS-06)
- CIS: Critical Security Controls v8.1 (Control 4 Secure Configuration, Control 6 Access Control Management, Control 16 Application Software Security)
- CISA: Known Exploited Vulnerabilities Catalog
Related NHI Mgmt Group resources: CI/CD Security Guide, CI/CD Pipeline Identity Security Guide, Self-hosted CI/CD Runners and Build Agents, Secrets Management Guide.