OpenAnt shows why LLM vulnerability discovery needs runtime verification

At a glance

What this is: Knostic's OpenAnt research argues that LLM-assisted vulnerability discovery needs runtime verification because static analysis and fuzzing alone miss too much at repository scale.

Why it matters: It matters because teams evaluating AI-assisted code review need to understand that verification, not just model reasoning, determines whether findings are trustworthy enough to operationalise across large codebases.

By the numbers:

• OpenAnt reduces the attack surface by up to 97% through reachability-filtered analysis units.

Context

LLM-assisted vulnerability discovery is increasingly attractive because models can reason about code semantics, but that same strength creates governance problems at repository scale. The challenge is not only finding bugs, but deciding when an AI-generated candidate is trustworthy enough to act on, especially when false positives, compute cost, and environment setup can overwhelm review workflows.

This topic sits at the boundary of application security, AI-assisted engineering, and supply-chain governance. Where coding assistants, extensions, and agent-like tooling interact with source code and build artefacts, the security question becomes how to verify exploitability without turning the review process into another source of noise or uncontrolled runtime access.

Key questions

Q: How should security teams handle AI-generated vulnerability findings in the release pipeline?

A: Security teams should treat AI-generated findings as inputs, not decisions. Every finding should pass deterministic validation against source context, runtime evidence, and policy before it influences release gating or remediation. That reduces false confidence and keeps automation from pushing low-quality or environment-agnostic results into production workflows.

Q: Why do reachability filters matter in LLM-based code review?

A: They reduce the search space to code paths that an attacker could plausibly hit, which lowers compute cost and cuts down irrelevant findings. Without reachability, AI review tends to over-focus on syntactic matches and unrelated dead code. That makes the workflow slower, more expensive, and harder to trust.

Q: What are the signs that AI-assisted vulnerability hunting is becoming noisy?

A: The clearest signs are a surge in low-confidence findings, duplicated issue reports, and a growing backlog of items that no one can verify quickly. When reviewers spend more time sorting output than understanding root causes, the workflow has outgrown the team’s triage capacity and needs tighter scope control.

Q: How do teams keep sandboxed validation from becoming a security risk itself?

A: Use isolated, disposable environments with no persistence between runs, tightly scoped inputs, and clear teardown after each test. Validation systems should exist only to prove exploitability, not to accumulate state or expand access. The control objective is containment, reproducibility, and short-lived execution.

Technical breakdown

Why static analysis and fuzzing miss repository-scale risk

Static analysis is good at pattern recognition, but it struggles when exploitability depends on code paths, environment state, or chains of conditions spread across a large repository. Fuzzing can expose runtime bugs, but it is expensive, infrastructure-heavy, and narrow in the classes of defects it reaches. OpenAnt addresses the gap by combining symbolic reachability filtering with LLM reasoning, which narrows the search space before verification begins. That matters because the bottleneck is not just detection volume, but whether a finding can survive a stronger test of exploitability.

Practical implication: teams should treat LLM findings as hypotheses until they have survived a verification step that reflects real execution conditions.

How reachability filtering changes the economics of review

Reachability filtering decomposes a codebase into analysis units that are actually exposed to a plausible attack path, rather than treating every line as equally relevant. This reduces wasted model calls and lowers the cost of reasoning over a large repository, while also shrinking the attack surface that the verifier must inspect. In practice, this shifts AI-assisted review from broad scanning to target selection. The security value comes from forcing the tool to spend effort only where execution paths make a vulnerability plausible, not where a syntactic pattern merely looks suspicious.

Practical implication: use reachability as a gating control before running expensive validation or human review on AI-generated findings.

Why sandboxed dynamic validation is the trust anchor

Dynamic validation is the point where a candidate issue is tested in an exploit environment rather than inferred from code alone. By generating sandboxed containers and discarding them after use, OpenAnt tries to prove whether an attacker could actually exercise the weakness under realistic constraints. That is a different security claim from static discovery. For practitioners, this is the bridge between a promising signal and an actionable defect, and it is also where runtime governance matters most because the validation environment itself must be controlled and disposable.

Practical implication: require disposable, isolated validation environments for AI-assisted exploit testing so the verification process does not become a new risk surface.

Threat narrative

Attacker objective: The objective is to identify code paths that are actually exploitable so defenders can prioritise real vulnerabilities instead of noise.

1. Entry begins with repository-scale code analysis where static review or fuzzing alone produces incomplete or noisy coverage of exploitable paths.
2. Credential or exploitability assumptions are tested by adversarial verification, which simulates attacker constraints to separate plausible weaknesses from false positives.
3. Impact is the identification of previously unknown vulnerabilities with lower review noise, enabling stronger remediation decisions before code reaches production.

Breaches seen in the wild

• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Runtime verification is becoming the deciding control in AI-assisted vulnerability discovery. LLMs can reason semantically about code, but semantic reasoning alone does not prove exploitability. Once repository scale enters the picture, verification quality becomes the factor that determines whether AI output is operationally useful or just another stream of false positives. For practitioners, that means the governance question is no longer whether to use LLMs, but where to place the proof boundary.

Verification trust gap: AI-assisted review creates a new assurance problem because the model can be right about code intent and still wrong about exploitability. OpenAnt's combination of reachability filtering, adversarial testing, and sandboxed validation shows that discovery pipelines need a stronger trust model than traditional static tools. The practitioner takeaway is that evidence quality must be designed into the workflow, not bolted on after triage.

Cost now governs detection as much as accuracy. Large-scale LLM review fails when the number of candidates, the runtime environment, or the context window overwhelms the process. That is a governance issue, not just an engineering one, because expensive review paths shape what gets tested and what gets ignored. Teams should treat compute cost as part of the control design for AI-assisted code security.

Open-source AI vulnerability tooling will increasingly converge with software supply-chain governance. When projects like OpenAnt evaluate repositories, extensions, and supporting components, the real question is whether the surrounding toolchain can be trusted to produce stable and reproducible signals. That puts code review, verification environments, and dependency integrity into the same governance conversation. Practitioners should expect AI-assisted discovery to pull supply-chain controls closer to application security workflows.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Read next: Ultimate Guide to NHIs — Key Challenges and Risks

What this signals

Verification trust gap: AI-assisted vulnerability discovery now fails or succeeds on the strength of its runtime proof, not on the sophistication of its first-pass reasoning. As organisations push LLMs into code review and security engineering, the control question shifts from finding possible issues to proving which issues are actually exploitable.

That change also affects how teams design governance around coding assistants and agentic tools. Discovery pipelines need bounded execution, deterministic validation environments, and review gates that separate semantic suspicion from operational evidence; otherwise, AI output becomes an expensive source of ambiguity rather than risk reduction.

For practitioners

• Define a verification threshold for AI-assisted findings Require every LLM-generated vulnerability candidate to pass an explicit exploitability check before it enters triage or remediation workflows.
• Gate repository-scale analysis with reachability Filter code into reachability-based analysis units so expensive reasoning and sandboxing only run on paths that an attacker could plausibly reach.
• Use disposable sandbox environments for validation Run dynamic exploit tests in isolated containers and destroy the environment after validation to prevent residual state or accidental reuse.
• Track false positives as a governance metric Measure how many AI-generated findings fail verification, then use that rate to tune where the workflow should rely on automation versus human review.

Key takeaways

• LLM-assisted bug hunting becomes materially useful only when runtime verification separates plausible defects from false positives.
• Repository-scale discovery changes the economics of code review by forcing teams to narrow analysis before they spend compute and human attention.
• Sandboxed validation and reachability filtering turn AI findings into evidence, which is the threshold practitioners need before remediation.

Key terms

• Runtime Verification: Runtime verification is the practice of checking what an identity is doing while it is active, rather than relying only on provisioning-time controls. For autonomous agents, it means monitoring prompts, tool use, outputs, and policy violations as actions unfold so harmful behavior can be contained early.
• Reachability filtering: A method for shrinking the analysis surface to code paths that can actually be invoked from an external or attacker-controlled entry point. It helps teams avoid spending time on dead code and unreachable findings, which improves triage quality and makes downstream validation more meaningful.
• Sandboxed dynamic validation: A runtime testing method that executes exploit hypotheses in isolated, disposable environments. It lets teams observe real behaviour without exposing production systems, which is essential when proving whether a vulnerability is operationally exploitable rather than just theoretically possible.
• AI-Assisted Vulnerability Scanning: The use of an AI system to inspect code, configurations, or repositories for security weaknesses faster than conventional manual review. It improves discovery speed, but the control value depends on how well findings are triaged, validated, and turned into remediation actions.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security and identity practitioners connect emerging AI workflows to the controls their programmes depend on.