ConcreteCMS and AI SAST: can tools really find every bug?

TL;DR: AI SAST tools can uncover many vulnerabilities in the same codebase, but their results overlap only slightly and still require human validation for exploitability, production relevance, and false positives, according to Tenzai’s ConcreteCMS case study. The practical lesson is that finding bugs is not the same as proving security coverage.

Editorial analysis by NHI Mgmt Group, based on content published by Tenzai: “Can AI find them all? - A case study on ConcreteCMS”.

By the numbers:

• Tenzai reported 25 unique CVEs and an additional 7 collisions from its ConcreteCMS research.

Key questions

Q: How should security teams validate AI-assisted offensive findings before treating them as real risk?

A: Teams should require a reproducible attack path, not just a scanner result or model-generated claim.

Q: Why do different AI security tools find different bugs in the same codebase?

A: They use different search strategies, context windows, and vulnerability hypotheses, so they do not inspect code in the same way.

Q: What do teams get wrong about AI-assisted SAST?

A: Many teams assume AI automatically improves detection, when in practice it usually improves triage and remediation.

Practitioner guidance

• Build a production validation gate for AI findings Require every high-severity AI-generated finding to be checked against a deployed or production-like instance before it enters remediation tracking.
• Separate lead generation from confirmed vulnerability status Use one workflow for candidate issues and a second workflow for findings that have a correct root cause, a reproducible path, and verified reachability.
• Audit authorisation boundaries across similar endpoints Compare parallel routes for object checks, function-level checks, CSRF enforcement, and unsafe rendering, especially in admin and bulk-action flows.

Bottom line: AI security tools can expose many defects in one repository, but the overlap between tools is limited and no single scan is enough to claim coverage.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →