TL;DR: AI SAST tools uncovered many vulnerabilities in ConcreteCMS, but their overlap was slight and each still needed human validation for exploitability, production relevance, and false positives, according to Tenzai. Production context decides which findings matter, so one scan cannot stand in for coverage.
Editorial analysis by NHI Mgmt Group, based on content published by Tenzai: “Can AI find them all? - A case study on ConcreteCMS”.
By the numbers:
- Tenzai reported 25 unique CVEs and an additional 7 collisions from its ConcreteCMS research.
Key questions
Q: Why do different AI security tools find different bugs in the same codebase?
A: They use different search strategies, context windows, and vulnerability hypotheses, so they do not inspect code in the same way.
Q: What do teams get wrong about AI-assisted SAST?
A: Many teams assume AI automatically improves detection, when in practice it usually improves triage and remediation.
Q: How should security teams validate AI-assisted offensive findings before treating them as real risk?
A: Teams should require a reproducible attack path, not just a scanner result or model-generated claim.
Practitioner guidance
- Build a production validation gate for AI findings Require every high-severity AI-generated finding to be checked against a deployed or production-like instance before it enters remediation tracking.
- Separate lead generation from confirmed vulnerability status Use one workflow for candidate issues and a second workflow for findings that have a correct root cause, a reproducible path, and verified reachability.
- Audit authorisation boundaries across similar endpoints Compare parallel routes for object checks, function-level checks, CSRF enforcement, and unsafe rendering, especially in admin and bulk-action flows.
Bottom line: AI security tools can expose many defects in one repository, but the overlap between tools is limited and no single scan is enough to claim coverage.
Glossary terms: AI SAST, Exploit Validation, Authorization Boundary, Production Context
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
AI vulnerability discovery is becoming a triage problem, not just a detection problem. The article shows that different tools found different classes of defects in the same codebase, with only partial overlap. That means the security question is no longer whether a model can find bugs, but whether a team can separate exploitable findings from noise fast enough to matter. The practitioner conclusion is that validation quality is now part of the security control, not a downstream admin task.
A few things that frame the scale:
- 19% of organisations give AI systems dramatically more access than human employees, nearly one in five granting unrestricted privilege, according to the 2026 Infrastructure Identity Survey.
- 70% of organisations grant AI systems more access than they would give a human employee performing the exact same job, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: What is the difference between a code defect and a security issue that matters in production?
A: A code defect is a flaw in the source or logic. A security issue that matters in production is a flaw that can be reached, triggered, and used in the deployed environment to change access, integrity, confidentiality, or availability. The distinction depends on configuration, exposure, and whether the intended control actually works in practice.
👉 Read our full editorial: AI SAST tools miss exploitability context in ConcreteCMS audits