Corgea vs Snyk on a vulnerable repo: what the benchmark shows

At a glance

What this is: Corgea benchmarked against a deliberately vulnerable repository and reported stronger overall vulnerability detection than Snyk, including 42 of 47 confirmed issues found.

Why it matters: For IAM and security teams, the result is a reminder that code-scanning effectiveness depends on coverage, context, and false-positive discipline, not just familiar pattern matching.

By the numbers:

• Corgea found 42 of 47 source-confirmed issues in the benchmark repository.
• Snyk found 26 of 47 source-confirmed issues in the benchmark repository.
• Corgea reached 89.36% recall in the benchmark review.
• Corgea reported 85.71% F1 in the benchmark review.

Context

The core problem in code scanning is not whether a tool can spot obvious sink patterns. It is whether it can retain application context across routes, frameworks, configuration, and deployment files strongly enough to surface the issues that actually change risk.

This comparison uses a deliberately vulnerable repository, so the practical question is how well each scanner converts source evidence into actionable findings. For security teams, that affects remediation backlog quality, not just detection volume.

Key questions

Q: What breaks when a scanner finds vulnerabilities but does not fix them?

A: Teams accumulate findings faster than they can close them, so the scanner becomes a reporting layer instead of a control. The result is remediation backlog, developer fatigue, and a weaker security posture because exposure stays live after discovery. In practice, organisations need fix-oriented workflows for high-confidence issues, not just more alerts.

Q: Why do recall and precision both matter in SAST evaluation?

A: Recall shows how many real issues the scanner finds, while precision shows how much of its output is trustworthy. High recall without precision creates review fatigue, and high precision without recall leaves exploitable code undiscovered. Teams should judge scanners on the combined operational cost of missed issues and noisy findings.

Q: What do security teams get wrong about secure-code benchmark scores?

A: They often assume a higher benchmark score means safe deployment. In practice, a score only shows that the model avoided certain known vulnerability patterns in a test set. It does not prove the code is safe in a real system, where secrets, IAM, dependency risk, and runtime configuration can still create compromise paths.

Q: How should teams compare two SAST tools before adoption?

A: Compare them on the same fixed repository, the same confirmed issue set, and the same classification rules for true positives, false positives, and false negatives. Then assess whether the tool exposes issues across code, configuration, and deployment assets, not only whether it catches familiar vulnerability patterns.

Technical breakdown

Why contextual code scanning outperforms simple taint-flow matching

Traditional SAST often excels when data moves cleanly from a request source to a known sink, such as SQL queries, shell execution, or unsafe URL fetches. It performs less well when the weakness depends on missing authorization, framework conventions, configuration files, or multiple code paths. Contextual analysis attempts to preserve those relationships so the scanner can recognise that a route, parser, or deployment setting is unsafe even when no classic sink pattern is present.

Practical implication: evaluate scanners on multi-file and framework-aware findings, not only on obvious input-to-sink examples.

Why recall and precision both matter in vulnerable code review

Recall measures how many confirmed issues a tool finds, while precision measures how many reported findings are actually valid. A scanner with high recall but poor precision floods teams with noise, while a precise but shallow scanner leaves exploitable code unreported. In a benchmark like this, the operational value comes from the balance: fewer missed issues, but also fewer false positives that waste reviewer time.

Practical implication: compare scanners using confirmed true positives, false positives, and missed issues together instead of treating any single metric as decisive.

Configuration weaknesses are often the blind spot

Application security tools commonly focus on code paths and miss insecure defaults, deployment settings, or repository-adjacent files that still create exposure. In practice, that is where hardcoded credentials, unsafe XML parser flags, disabled certificate verification, and infrastructure misconfigurations often live. A repository benchmark that includes configuration weaknesses tests whether the scanner understands the full attack surface rather than only the application logic.

Practical implication: include charts, manifests, templates, and test assets in scanner evaluation because many real findings live outside source handlers.

NHI Mgmt Group analysis

Coverage is the deciding variable in scanner selection when a repository contains mixed vulnerability classes. A tool that sees only familiar taint-flow patterns will miss authorization gaps, parser abuse, and deployment-level exposures that materially increase risk. In this benchmark, the better outcome came from finding more confirmed issues across more parts of the repository, not from optimising for a single vulnerability family. Practitioners should treat breadth of detection as a governance requirement, not a convenience feature.

Context-aware analysis is now a differentiator in secure code review, not an optional enhancement. The article shows that the highest-value misses were not exotic exploits but weaknesses that required understanding framework behaviour, route semantics, and surrounding configuration. That makes the comparison relevant to application security programmes that rely on SAST as a front-line control. Teams should expect their scanner to reason about code paths, not just match signatures.

False positive discipline is a workflow control, not just a vendor metric. If a scanner repeatedly classifies weak or unsupported lines as vulnerabilities, reviewers spend time validating noise instead of fixing real defects. That undermines developer trust and delays remediation. The practical lesson is to benchmark tools on the same repository and same issue set before standardising them across an engineering programme.

Application security and identity security meet at the authorization boundary. Several of the missed findings were not pure code-smell problems but exposure points where access control was absent or misapplied. For IAM and PAM teams, that matters because code paths that expose user records, tokens, or privileged actions become identity enforcement failures as soon as the application executes them. Practitioners should align code scanning with authorization review, not leave it isolated in the AppSec queue.

What this signals

Context-aware SAST is becoming the practical dividing line in application security. Tools that understand routing, authorization, and adjacent configuration are better suited to mixed-language repositories than scanners that depend mainly on taint-flow signatures. For practitioners, that means evaluation should focus on whether the tool can surface the findings that reviewers would otherwise never see.

Authorization exposure in code quickly becomes identity exposure in production. When an application returns user records, modifies state without checks, or exposes privileged endpoints, the problem is no longer just code quality. IAM and application security teams need to review the same defect through both the access-control lens and the source-code lens.

Benchmarking a scanner against a fixed repository creates a cleaner procurement signal than marketing claims. It reveals whether the product can actually reduce backlog noise and find the defects most likely to persist in engineering workflows. For security leaders, that is the more useful measure of value than a feature checklist.

For practitioners

• Benchmark scanners on the same confirmed issue set Use a fixed, source-confirmed repository benchmark and compare true positives, false positives, and false negatives on identical scope before standardising a scanner.
• Include configuration and deployment files in evaluation Test whether the tool inspects manifests, templates, and environment files for secrets, parser flags, and exposure settings, not only application source.
• Review authorization gaps alongside code findings Route findings that expose user data, token data, or state-changing endpoints through an authorization review so missing access controls are not treated as ordinary code defects.
• Measure reviewer time spent on unsupported findings Track how many findings are dismissed as duplicates, misclassified, or unsupported, then treat that noise rate as a procurement and tuning metric.

Key takeaways

• Corgea’s benchmark showed that scanner performance is defined by more than recognizable injection patterns, because missing authorization and configuration weaknesses still matter operationally.
• The most relevant evidence here is the gap in confirmed findings, with Corgea finding 42 and Snyk finding 26 in the same 47-issue benchmark set.
• For practitioners, the right procurement test is a fixed repository comparison that measures coverage, precision, false negatives, and reviewer noise together.

Key terms

• Static Application Security Testing: Static Application Security Testing is a method for finding security flaws by examining code, binaries, or configuration without executing the application. It is strongest when used early in development, where teams can fix issues before deployment and prevent avoidable defects from reaching production.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• False negative: A missed detection where a real threat is not recognised as malicious or important. In AI SOC environments, false negatives often arise when the system under-collects context, stops at the first plausible answer, or over-optimises for speed.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle topics that underpin access control decisions. It helps security practitioners connect identity governance to the operational controls their programmes depend on.