CI/CD Security Tooling Buyer's Guide: Criteria

Most CI/CD security spend goes on scanners that run inside the pipeline, while the pipeline itself, its workflow files, runners, tokens and third-party actions, is left unmonitored and is now a primary target. Attacks such as poisoned pipeline execution, compromised third-party actions and stolen publishing tokens succeed because build systems hold production credentials and run code from many sources with little oversight. CI/CD security tooling covers the controls that protect the pipeline: posture management for source control and CI configuration, static analysis of workflow files, runtime monitoring of build runners, secrets and token governance, and audit log detection. This vendor-neutral buyer's guide is for platform engineering, AppSec and security operations teams writing requirements or running an RFP. It does not repeat dependency, SBOM and provenance buying criteria, which are covered in the Software Supply Chain Security Buyer's Guide.

Key takeaways

Buy against the OWASP Top 10 CI/CD Security Risks. The list (CICD-SEC-1 to CICD-SEC-10) is a practical requirements checklist: map each risk to a capability you need and to the product feature that covers it, and the gaps become obvious.
Static workflow analysis and runtime runner monitoring catch different attacks. Static analysis finds script injection, risky triggers and unpinned actions before merge; runtime monitoring sees a trusted dependency or action behaving maliciously during a build. Compromised actions that were pinned by tag pass static checks.
Posture must cover the code host and the CI system together. Branch protection, review rules, token defaults, runner registration and secret scoping are settings, not code, and drift quietly. Continuous checks with an owner per finding matter more than a one-off audit.
Credential reduction is the highest-value outcome. Tools should help you replace stored cloud keys with OIDC federation, cut default token permissions to read and find secrets exposed in logs, because stolen pipeline tokens drive most recent incidents.
The tool becomes a privileged identity in your build system. Posture products read every repository and setting, and runtime agents run on every runner. Insist on least-privilege, short-lived access, and plan for the tool itself being compromised.

Define requirements

Platforms in scope: code hosts (cloud and self-managed), CI systems (hosted workflows, self-managed servers such as Jenkins, cloud-native build services), artifact registries and deployment tools. Coverage beyond one code host is the first filter for most organisations.
Runner estate: hosted runners, self-hosted virtual machines, Kubernetes-based ephemeral runners, and whether any self-hosted runner serves a public repository. GitHub says self-hosted runners "should almost never be used for public repositories". See Self-hosted CI/CD Runners and Build Agents.
Credentials held by pipelines: cloud deployment keys, registry and package publishing tokens, signing keys and third-party API keys. List them; this list is your blast radius.
Enforcement points: pull request checks on workflow changes, organisation-level policies (allowed actions, default token permissions), runner network egress policy, and deployment approvals in protected environments.
Detection needs: whether code host and CI audit logs reach your SIEM today, and who in the SOC owns pipeline alerts. See the CI/CD Pipeline Compromise Playbook.
Existing coverage: secrets scanning, SCA and ASPM products often include some pipeline posture checks. Decide whether you need replacement or only the missing layers. See the ASPM Buyer's Guide.

Size changes the answer. A small team on one hosted code host and its built-in CI can get far with native settings (read-only default token, allowed actions list, environment protection rules), a free workflow linter in pull requests and OIDC for cloud access. A large organisation with several code hosts, self-hosted runners, Jenkins estates and regulated release processes needs central posture across platforms, runtime monitoring on runners, policy as code for pipeline rules and SOC-ready audit log detection.

Capability areas

OWASP CI/CD risk	Capability to look for	How to test it
CICD-SEC-1 Insufficient Flow Control Mechanisms	Posture checks for branch protection, required reviews, CODEOWNERS on workflow files and protected deployment environments	Disable a required review in a trial repository and time the alert
CICD-SEC-2 Inadequate Identity and Access Management	Inventory of human and machine identities with access to code and pipelines, stale and over-privileged accounts, outside collaborators	Compare the tool's inventory with your own export
CICD-SEC-3 Dependency Chain Abuse	Usually covered by supply chain and SCA tools; check integration rather than duplication	See the Software Supply Chain Security Buyer's Guide
CICD-SEC-4 Poisoned Pipeline Execution	Static analysis of workflow files for script injection from untrusted input, privileged triggers that check out untrusted code, and secrets exposed to fork builds	Seed a workflow with an injectable expression and a risky trigger
CICD-SEC-5 Insufficient PBAC	Checks for default token permissions, per-job permissions, secret and environment scoping, and runner group restrictions	Seed a workflow with write-all permissions
CICD-SEC-6 Insufficient Credential Hygiene	Secrets in workflow files and logs, long-lived cloud keys that could be replaced with OIDC, unrotated tokens	Plant a test token in a log line and a stored cloud key
CICD-SEC-7 Insecure System Configuration	Configuration checks for CI servers and runners: versions, plug-ins, executors on the controller, runner persistence	Point it at a deliberately weak test CI server
CICD-SEC-8 Ungoverned Usage of 3rd Party Services	Inventory of installed code host apps, OAuth grants, webhooks and third-party actions, with pinning status	Install a test app with broad scopes
CICD-SEC-9 Improper Artifact Integrity Validation	Detection of unsigned artifacts and tag-pinned actions; provenance is usually a separate tool	Use a tag-pinned action and an unsigned artifact
CICD-SEC-10 Insufficient Logging and Visibility	Collection of code host and CI audit logs, runner telemetry, detection rules and SIEM export	Generate admin events and check they arrive and alert

Workflow static analysis: findings for template or script injection, excessive token permissions, credential persistence after checkout, privileged triggers, unpinned or impostor action references, and cache poisoning paths, with inline pull request comments. GitHub's own guidance is to pin actions "to a commit SHA", set the default token to read access only, and pass untrusted values through an intermediate environment variable rather than building inline scripts from them.
Runner runtime monitoring: an agent or instrumented runner image that records process, file and network activity per job, baselines expected egress, and alerts on or blocks unexpected destinations or tampering with source files during a build.
Pipeline posture management: continuous checks of organisation and repository settings across code hosts and CI systems, mapped to a benchmark, with drift alerts and an owner per finding.
Policy as code: rules such as "no workflow may use a third-party action not pinned to a SHA" or "deploy jobs must use an approved environment" expressed in a versioned policy language and enforced as a required check.
Detection content: rules on audit logs for new admin grants, disabled branch protection, new self-hosted runner registration, new deploy keys or apps, workflow changes on default branches without review, and secrets accessed by unusual workflows.

Evaluation criteria

Weight these to your requirements. An illustrative split is 25% posture coverage across your platforms, 20% workflow analysis quality, 20% runtime monitoring, 15% detection and response integration, 20% platform security, integration and cost.

Criterion	What good looks like	Weak signs
Platform coverage	All your code hosts and CI systems, including self-managed servers, with published lists of checks per platform	One code host only, with others "on the roadmap"
Workflow analysis accuracy	Distinguishes injectable from safe expressions, understands trigger context and fork behaviour, low false positive rate on your real workflows	Flags every use of an expression; no trigger awareness
Runtime visibility	Per-job process and network telemetry, egress allow-listing with audit mode, works on hosted and self-hosted runners with low build overhead	Only log scraping after the job; large build slowdowns
Credential reduction support	Inventories stored secrets, flags those replaceable with OIDC, detects secrets in logs and artifacts, reports default token permissions	Secret detection only in committed code
Third-party governance	Inventory of actions, apps, plug-ins and webhooks with pinning status, permissions and owner; alerts on new installs	No view of apps and OAuth grants
Fix guidance and automation	Specific fixes (exact permission block, SHA for a pinned action), optional fix pull requests reviewed by humans	Generic advice with links to documentation
Detection integration	Normalised audit log events and alerts to SIEM, mapping to MITRE ATT&CK T1677 Poisoned Pipeline Execution and related techniques, response guidance	Dashboard-only alerts
Platform security and exit	Read-only app integrations by default, separate scoped write integration, short-lived tokens, SSO, audit log of the tool's own changes, data export	Organisation owner token required; no record of what the tool changed

Where identity fits

Pipelines are among the most privileged non-human identities in most organisations: build jobs assume cloud roles, publish packages and sign releases. Recent incidents show the pattern. In the Codecov breach a leaked cloud storage key let attackers alter a CI script that then harvested customers' secrets; in the tj-actions/changed-files compromise a stolen bot token was used to poison a widely used action that printed secrets into build logs; and in the CircleCI breach a stolen session exposed customers' stored CI secrets. Ask each vendor to show how its product would have detected or reduced the impact of these specific cases.

The NSA and CISA guidance on defending CI/CD environments recommends minimising long-term credentials in favour of federated identity and temporary cloud credentials, and never passing secrets in plaintext. Tooling should measure progress against that: count of stored long-lived secrets, share of deployments using OIDC, and default token permissions. See the CI/CD Pipeline Identity Security Guide and the GitHub Actions OIDC Federation Hardening Guide.

Remember that the security tool can be the attack path. Runtime agents run inside every build and posture products hold organisation-wide read tokens; security scanners have themselves been compromised in recent supply chain campaigns, as the Trivy supply chain attack and the Checkmarx KICS supply chain attack in 2026 showed. Pin the tool's own actions and images to digests, give its integration the narrowest scopes, and include it in your CI/CD Pipeline Compromise Playbook.

Framework mapping

Framework	Reference	Relevance
OWASP	Top 10 CI/CD Security Risks, CICD-SEC-1 to CICD-SEC-10	Requirements checklist for capability coverage
MITRE ATT&CK v20	T1677 Poisoned Pipeline Execution; T1195.002 Compromise Software Supply Chain	Detection content and threat modelling
NIST CSF 2.0	PR.PS-01, PR.PS-06, GV.SC-05	Configuration management, secure development practices, supply chain requirements in contracts
NIST SP 800-53 Rev. 5	SA-10, SA-15, CM-3, SR-3	Developer configuration management, development tools, change control, supply chain controls
NIST SP 800-204D	Software supply chain security in DevSecOps CI/CD pipelines	Reference architecture for securing build, test, package and deploy stages

Questions to ask vendors

Which code hosts and CI systems do you cover, including self-managed versions, and how many checks run on each? Show the list for our platforms.
Show your workflow analysis on our repositories: how do you separate an injectable expression from a safe one, and how do you handle privileged triggers and fork pull requests?
How does runtime monitoring work on hosted and self-hosted runners, what is the build time overhead, and can egress policy run in audit mode before blocking?
Would your product have detected the tj-actions/changed-files compromise during a build that used a tag-pinned reference? What would the alert have shown?
How do you help us replace stored cloud keys with OIDC federation, and how do you report remaining long-lived secrets?
Which permissions does each integration require, which are write permissions, and can they use short-lived app tokens? What happens if your service is compromised?
Which audit log events do you collect, how quickly do they reach our SIEM, and which detection rules ship by default?
How are your own actions, agents and images published, signed and versioned so we can pin them?
What does the licence count (contributors, repositories, runners, build minutes) and how does cost scale?

Red flags

"CI/CD security" that is only SAST and SCA scanners run in the pipeline, with no checks on the pipeline itself.
Integrations that need organisation owner rights or a long-lived personal token.
Runtime agents that cannot be pinned to a digest or that update themselves without notice.
Posture scores without the underlying checks, or checks that cannot be tuned per repository.
No coverage for self-hosted runners or self-managed CI servers when you run them.
Alerts that cannot be exported to your SIEM or that lack the raw audit event.

Proof of concept

Scope it: one organisation or group per code host, ten to twenty representative repositories including a release repository, one self-hosted runner pool and one self-managed CI server if you run them. Agree weights and pass criteria first.
Connect with least privilege: install through dedicated read-only app identities limited to the trial scope, record every permission requested and keep any write integration separate.
Seed workflow weaknesses: an injectable expression from a pull request title, a privileged trigger that checks out fork code, a write-all token, a tag-pinned third-party action and a stored cloud key. Measure what each product flags and the quality of the fix.
Test runtime monitoring: in an isolated test repository, run a build step that calls an unexpected domain and modifies a source file mid-build. Start egress policy in audit mode, then block, and measure build overhead.
Test posture drift and detection: disable branch protection, register a new self-hosted runner and add a deploy key. Confirm alerts reach the SIEM and time them.
Test credential reduction: ask the product to list stored secrets that could move to OIDC and secrets visible in recent logs, and check the result against your own inventory.
Test exit and roll back: export findings and history, remove runtime agents from runner images, switch policies back to warn, revoke all integration tokens and confirm access ends.

Keep measuring after go-live: share of workflows with read-only default token permissions, share of third-party actions pinned to a SHA, number of stored long-lived cloud credentials (falling), workflow findings merged per month, runners covered by runtime monitoring, and median time to detect a protection setting change.

Standards and references

How NHI Mgmt Group can help

We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.

CI/CD Security Tooling Buyer’s Guide: How to Evaluate Pipeline Security Tools