Notifications
Clear all
Topic starter
20/08/2026 5:58 pm
TL;DR: Replit’s research shows AI-only code scanners missed dependency CVEs, secrets hidden in JSX literals, and produced inconsistent results across identical runs, underscoring that security scanning must be deterministic and auditable, according to Pixee’s analysis of the findings. The real issue is not AI in remediation, but replacing structural detection with statistical inference, which leaves AppSec blind to known risk classes.
NHIMG editorial — based on content published by Pixee: Why AI Can't Audit Its Own Code: Replit's Research on Deterministic Security
Questions worth separating out
Q: What breaks when AI security is limited to AppSec scanning?
A: Static scanning misses emergent behaviour, so the organisation can approve code that later behaves outside policy in production.
Q: Why do secrets and machine credentials need deterministic detection in code pipelines?
A: Because exposure depends on the value and context, not on whether a model recognises a familiar naming pattern.
Q: What do security teams get wrong about AI-powered scanners?
A: They often treat probabilistic scanners as replacements for deterministic controls, when they are better thought of as a second layer.
Practitioner guidance
- Implement deterministic baseline scanning Use SAST, SCA, and secret detection as the authoritative layer for code review and keep AI out of first-pass detection decisions.
- Separate remediation from detection Allow AI to assist with fix generation, explanation, and triage only after a deterministic scanner has identified the vulnerability class.
- Test secret scanners against syntax variants Create regression cases that move the same secret between literals, variables, comments, and configuration files so you can verify detection is structural, not prompt-driven.
What's in the full article
Pixee's full analysis covers the operational detail this post intentionally leaves for the source:
- Replit’s test methodology for comparing AI-only scanners against deterministic SAST, SCA, and secret detection
- Code examples showing how small syntax changes altered AI detection outcomes across repeated runs
- The specific dependency and secret patterns that deterministic tools caught consistently but AI-only scanners missed
- Pixee’s discussion of where AI belongs in the remediation workflow after baseline detection has already happened
👉 Read Pixee’s analysis of why AI code security scanners need deterministic baselines →
AI code security scanners are missing basics. What should teams do?
Explore further
20/08/2026 6:37 pm
AI-assisted code review is not a substitute for deterministic security controls. Replit’s findings make clear that probabilistic interpretation cannot carry the assurance burden of application security. When the same input produces different outputs, the control is not auditable in the way security teams need. For AppSec leaders, the correct model is deterministic baseline detection with AI reserved for remediation support.
A question worth separating out:
Q: How should organisations use AI in IAM without weakening governance?
A: Use AI for pattern detection, summarisation, and query generation, but keep access approval, role creation, and policy enforcement under human accountability. AI should accelerate evidence gathering, not replace the judgement needed to decide whether access is appropriate. If the input data is poor, treat the model output as advisory only.
👉 Read our full editorial: AI code security scanners need deterministic baselines, not prompts
