What breaks when vulnerability verification relies only on a model saying a code path is exploitable?

Why This Matters for Security Teams

Model-only verification is risky because an LLM can describe a plausible exploit path without proving that the path exists in the running system. For security teams, that creates false confidence, noisy triage, and wasted remediation effort. The problem is not just accuracy, but operational reachability: a finding must survive code tracing, input provenance checks, and endpoint or runtime validation before it should be treated as actionable.

That distinction shows up across NHI-heavy environments where credentials, tokens, and service accounts are already overexposed. NHI Mgmt Group notes that 97% of NHIs carry excessive privileges in its Ultimate Guide to NHIs, which means a convincing narrative from a model is not the same thing as a real exploitation path. Current guidance from CISA cyber threat advisories and CIS Controls v8 consistently favors evidence-based validation, not assertion-based analysis.

In practice, many security teams encounter the weakness only after analysts have already escalated an exploit that cannot be reproduced, rather than through intentional verification design.

How It Works in Practice

Effective verification starts by treating the model as a hypothesis generator, not an oracle. An agent or analyst can ask the model where a code path might be exploitable, but the answer must be checked against source, runtime, and environment evidence. That means tracing the relevant function, identifying the exact input source, confirming whether the data can actually reach the sink, and validating whether the target endpoint, build artifact, or deployed service still matches the code under review.

This is especially important when review spans agents, pipelines, or tooling that can chain actions. The OWASP NHI Top 10 is relevant here because autonomous systems can produce persuasive but ungrounded claims, and Top 10 NHI Issues highlights the broader governance gap that appears when machine identities are not tied to observable execution. A practical workflow usually includes:

• Code-level tracing to confirm control flow and data flow.
• Provenance checks to verify whether the input is attacker-controlled.
• Runtime validation to confirm the service, endpoint, or container is actually deployed.
• Exploitability testing in a safe environment to prove the path end to end.
• Evidence capture so the finding can be reproduced and audited later.

Where teams need external corroboration, ENISA Threat Landscape is useful for understanding how attack narratives map to real-world tactics, while NHI Mgmt Group research helps anchor the identity and secrets dimension. This guidance tends to break down in large monorepos, rapidly changing CI/CD environments, and pre-production scans because the code, build, and runtime state can diverge before verification finishes.

Common Variations and Edge Cases

Tighter verification often increases analyst time and test complexity, so organisations have to balance speed against proof quality. That tradeoff becomes sharper when the code path exists only in one deployment tier, behind feature flags, or inside a branch-specific build that no longer matches production.

Current guidance suggests treating model output differently depending on the evidence already available. If a model identifies a likely flaw but cannot tie it to reachable input, live dependencies, and an active endpoint, the result should remain a lead, not a confirmed vulnerability. In agentic or automated review pipelines, this matters even more because a model can chain together steps that are logically coherent but operationally impossible. That is why human review, runtime telemetry, and exploit reproduction still matter even when the model sounds certain.

For identity-heavy systems, the risk is compounded by exposed secrets and service accounts. The 52 NHI Breaches Analysis shows how often identity compromise becomes the real attack path, not the code flaw itself. The same caution applies when assessing token exposure, credential reuse, or lateral movement potential in a JetBrains GitHub plugin token exposure-style scenario. The safest rule is simple: if the exploit cannot be demonstrated against the actual system state, it should not be treated as verified.

Related resources from NHI Mgmt Group

• What breaks when customer onboarding relies too heavily on no-code verification workflows?
• What breaks when a BYOC model still relies on standing vendor access?
• What breaks when deepfake detection relies on periodic model updates?
• What breaks when mobile identity verification relies only on root detection?