Application security testing tools are often bought on a demo against the vendor's own sample application, then struggle on real code with false positives, slow pull request scans and partly understood frameworks. Platforms now bundle static analysis (SAST), dynamic testing (DAST), interactive testing (IAST), software composition analysis (SCA), secrets, infrastructure as code (IaC) and API testing, but engine depth varies widely. This vendor-neutral guide is for AppSec leads, platform engineers and architects writing requirements or running a bake-off: what to define, what to score, what to ask, and a proof of concept on your own code. The tools should help you meet the code review and testing practices (PW.7 and PW.8) in NIST SP 800-218, the Secure Software Development Framework.
Key takeaways
- Test on your code, not theirs. Accuracy claims mean little until you have counted true and false positives on your own repositories and a test set with known answers.
- No single engine covers everything. NIST's static analysis study states that "SATE's purpose is NOT to evaluate nor choose the "best" tools" because "Tools have different strengths". Buy coverage across techniques, then judge each engine on its own.
- Developer experience decides adoption. Findings must arrive in the pull request and IDE, with a clear fix, within your CI time budget, or developers will route around the tool.
- Treat AI autofix as a suggestion, not a fix. Vendor documentation warns generated fixes can miss the flaw, change behaviour or add non-existent dependencies.
- The scanner holds your source code and repository tokens. Score its deployment model, data handling and its own access to your source control as hard as its detection rate.
Define requirements
Agree these before the first vendor call; they become your RFP scope and proof of concept criteria.
- Languages and frameworks: every language and major version you ship, high-risk frameworks (for example Spring, ASP.NET Core, Django, Express), mobile code and monorepos. Note code that does not build cleanly: OWASP warns that "Many SAST tools have difficulty analyzing code that can't be compiled".
- Testing types: most organisations start with SAST, SCA and secrets scanning in pull requests, then add authenticated DAST and API testing for internet-facing services.
- Source control and CI: your SCM platforms, CI systems, pull request volume and the pull request scan time developers will tolerate.
- Reporting obligations: the OWASP Top 10:2025, the 2025 CWE Top 25, the OWASP API Security Top 10 2023 or sector rules. See the PCI DSS 4.0 Guide.
- Data constraints: whether source code may leave your network, where it is processed, and how long code snippets and findings are retained.
- Adjacent controls: dependency policy and build integrity are separate purchases. See the Open Source Security Guide and the SLSA and Build Provenance Guide.
Size changes the answer. A small team on one SCM platform is usually best served by one platform with native pull request integration. A large enterprise often runs specialist engines and aggregates their output, so open formats and deduplication matter more than any one dashboard.
Capability areas
| Area | What to look for | Known blind spots |
|---|---|---|
| SAST (source, bytecode, binaries) | Data flow across files and frameworks, incremental scans of changed code, custom rules | Configuration, authorisation logic and issues across service boundaries |
| DAST (running application) | Authenticated scanning with multiple roles, JavaScript-heavy front ends, passive and safe modes | Code location of the flaw, pages it cannot reach, most business logic |
| IAST (in-process agent during tests) | Runtime support, performance overhead, confirmed data flows | Code paths your tests never exercise |
| SCA (manifests, lockfiles, images) | Transitive dependencies, reachability, malicious package detection, licence policy, SBOM export | Your own code; vendored code without a manifest |
| Secrets scanning (commits, history, logs) | Push-time blocking, validity checks against the issuer, custom patterns | Secrets outside scanned locations; formats with no pattern |
| IaC scanning (Terraform, Kubernetes, Dockerfiles) | Policies mapped to your cloud baselines, custom policy as code, drift detection | Changes made outside the templates |
| API testing (REST, GraphQL, gRPC) | OpenAPI or GraphQL schema import, undocumented endpoint discovery, authorisation tests with two users | Abuse of legitimate business flows |
Business logic is where automated testing is weakest. OWASP's API list includes API1:2023 Broken Object Level Authorization and API6:2023 Unrestricted Access to Sensitive Business Flows, both of which depend on what a given user should be allowed to do. Tools can only test them with at least two accounts in different roles and expected outcomes; subtler logic still needs manual testing.
Evaluation criteria
Weight these to your requirements. An illustrative split is 30% accuracy and coverage on your code, 25% developer experience and CI fit, 15% deployment and data handling, 15% policy and reporting, 15% cost and licensing; adjust it rather than copy it.
| Criterion | What good looks like | Weak signs |
|---|---|---|
| Language and framework coverage | Named support per language version and framework, with framework-aware sources and sinks | "Supports Java" with no framework detail; one engine strong, the rest pattern matching |
| Accuracy | True and false positive rates you can reproduce on the OWASP Benchmark and on your own seeded code | Benchmark scores you cannot reproduce; no way to mark and learn from false positives |
| Developer experience | IDE plugins, pull request comments on changed lines, fix examples, suppression with reason and expiry | Findings only in a separate portal |
| AI fix suggestions | Reviewable pull request changes, re-scanned before merge, dependency changes flagged, can be disabled per repository | Auto-merged fixes; no statement on what code goes to which model |
| CI integration and speed | Native CI steps, differential scans, predictable run time | Full scans only; queueing on shared vendor capacity |
| Output and interoperability | SARIF 2.1.0 export with stable rule IDs and fingerprints, full API access, webhooks to ticketing | Proprietary export only; duplicate findings on every run |
| Policy gates | Gate on new findings by severity, CWE, reachability or secret validity; logged override; audit mode first | All-or-nothing blocking, or gates that only warn |
| Reporting | Findings mapped to OWASP Top 10:2025 and CWE IDs; fix times and exceptions by team | Mapping you cannot inspect; vanity counts with no age or ownership |
| Deployment and data handling | SaaS or self-hosted; scanning inside your network with only results leaving; stated retention and regions | Full source upload as the only option; unclear sub-processors |
| Licensing | Clear unit (contributing developer, application, scan or lines of code) and how it is counted; no penalty for scanning more repositories | Per-scan pricing that discourages scanning every pull request |
On accuracy, the OWASP Benchmark is a Java and Python suite for SAST, DAST and IAST whose authors state that "all the vulnerabilities deliberately included in and scored by the Benchmark are actually exploitable". Run every candidate yourself on the same release (v1.2) and treat it as one input: tools can be tuned to a well-known test set. For DAST, add a known-vulnerable target such as OWASP Juice Shop, which its project describes as "probably the most modern and sophisticated insecure web application".
On AI autofix, the documentation for one widely used autofix feature states that "The system may suggest fixes that fail to remediate the underlying vulnerability or introduce new vulnerabilities", and that it may propose packages that do not exist. Require every generated fix to pass the same scans and review as human code.
On output, SARIF 2.1.0 is the common exchange format; check for stable rule IDs and fingerprints. GitHub code scanning, for example, uses partialFingerprints "to detect when two results are logically identical" and accepts only SARIF 2.1.0.
Where identity fits
A SaaS scanner needs read access to every repository it scans, write access to pull requests to comment, and a token in each CI pipeline that calls it: privileged non-human identities. Prefer a source control app integration over a personal access token: GitHub's guidance notes that with an app the installer "can decide what repositories the app can access", that short-lived tokens limit damage if leaked, and that apps "remain installed even when the person who initially installed the app leaves the organization". Keep the scanner's CI credential in a secrets manager or use OIDC federation, scope it per project and give each integration an owner. The findings database also holds the secrets it found, so mask them. Security tooling in CI is a target in its own right, as the LiteLLM PyPI breach showed when a poisoned security scanner leaked a project's publishing token, and AI assistants that write fixes hold tokens too: see the Amazon Q Developer extension compromise. See the CI/CD Pipeline Identity Security Guide and the AI Coding Agents Security Guide.
Questions to ask vendors
- For each of our languages and frameworks, which engine analyses it, at what depth (data flow or pattern), and which framework versions are tested?
- Will you run on our sample repositories and the OWASP Benchmark in the trial and give us raw results, not a summary?
- How long does a differential scan of a typical pull request take, and what happens to pipeline time when your service is busy?
- For AI fix suggestions: which model, where does it run, what code is sent, is it used for training, and are fixes re-scanned?
- Show DAST handling SSO or MFA login, multiple roles, single-page applications and API schemas on our staging application.
- What permissions does your SCM integration request, does it use short-lived app tokens, and can we limit it to selected repositories?
- Where is our source code processed and stored, for how long, and can scanning run inside our network with only results sent out?
- What exactly does the licence count, and what happens to cost when we add repositories, contributors or scan frequency?
Red flags
- Accuracy figures based only on the vendor's own test applications, or a refusal to let you benchmark.
- AI-generated fixes that can be merged without review or a re-scan.
- DAST that cannot authenticate, so it only tests the login page.
- SCM integrations that need organisation owner rights or a long-lived personal token.
- No SARIF or API export, which locks findings history into one product.
Proof of concept
- Scope it: three to five repositories in your main languages, one web application with staging, and one API with a schema. Agree weights and pass criteria first.
- Seed known answers: add a handful of known flaws (for example SQL injection, path traversal, a missing authorisation check and a hard-coded test credential) on a branch, and run each tool against the OWASP Benchmark and a known-vulnerable application.
- Connect with least privilege: install each tool through a dedicated app or service identity limited to the trial repositories, record every permission requested, and never use an administrator's personal token.
- Measure accuracy: have an engineer triage a fixed sample of findings per tool and record true positives, false positives and missed seeded flaws.
- Measure developer fit: run real pull requests for two weeks with gates in audit mode. Record scan time and developer feedback; have a security engineer review AI fix suggestions.
- Test DAST safely: scan staging, not production. The OWASP ZAP project warns that active scanning is a real attack and "actual damage can be done to a site's functionality, data, etc." Use two test accounts with different roles.
- Test output and exit: export SARIF into your ticketing or aggregation tool, then uninstall each trial integration, revoke its tokens and get written confirmation that uploaded code is deleted.
Keep measuring after go-live: false positive rate on triaged samples, median pull request scan time, share of repositories scanned on every pull request, time to remediate critical findings, and expired suppressions.
How NHI Mgmt Group can help
We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: Open Source Security Guide · SLSA and Build Provenance Guide · AI Coding Agents Security Guide · Vulnerability Management Guide · CI/CD Pipeline Identity Security Guide