OpenAnt shows why LLM vulnerability discovery needs runtime verification

At a glance

What this is: OpenAnt is a multi-stage LLM-assisted vulnerability discovery pipeline that combines static analysis, adversarial verification, and sandboxed validation to cut false positives and surface previously unknown bugs.

Why it matters: It matters because security and IAM teams increasingly depend on AI-assisted code review and supply-chain assurance, where identity, secrets, and tool access issues can sit alongside conventional software flaws.

By the numbers:

• Decomposing codebases into reachability-filtered analysis units reduces the attack surface by up to 97%.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

👉 Read Knostic's paper on OpenAnt and LLM vulnerability discovery

Context

LLM vulnerability discovery has a basic governance problem: traditional static analysis can drown teams in false positives, while fuzzing is expensive and narrow in what it can explore. OpenAnt responds to that gap by treating repository-scale analysis as a workflow problem, not just a model problem, which is where the security value actually sits.

The identity angle is indirect but real. As AI coding assistants, MCP-connected tools, extensions, and other software supply-chain components become part of the delivery path, teams need assurance not only that code is vulnerable but that the surrounding access and execution context is understood. That makes OpenAnt relevant to both application security and NHI governance where tool credentials and execution boundaries intersect.

Key questions

Q: How should security teams reduce false positives in LLM-assisted vulnerability discovery?

A: Use a validation pipeline that ranks findings by reachability, tests exploitability under realistic attacker constraints, and confirms behaviour in a sandbox before escalating. That combination reduces noise because it separates suspicious code patterns from issues that can actually be exercised in context. The goal is not more alerts. It is fewer, better-defended findings that reviewers can trust and act on.

Q: Why do large codebases make AI-assisted security review harder?

A: Large repositories create context loss. Once analysis spans many packages, dependencies, and build paths, a model can identify patterns without proving whether the path is reachable or the exploit is real. That drives false positives, validation debt, and triage fatigue. Security teams need workflows that preserve execution context, not just tools that generate more candidate issues.

Q: What do teams get wrong about static analysis for LLM security?

A: They often treat static analysis output as a risk verdict rather than a hypothesis. In LLM-assisted review, a pattern may look dangerous but still fail exploitability checks because the relevant path is unreachable, constrained, or dependent on runtime conditions. Teams should use static analysis to narrow scope, then require adversarial verification and runtime testing before committing remediation effort.

Q: How can organisations govern AI tools that review or modify code?

A: Treat the tools as part of the software supply chain and the identity plane. Review what they can access, which repositories they can reach, and what credentials they use to operate. If an AI coding assistant or connected extension can touch secrets, build pipelines, or privileged services, it needs scoped access, explicit oversight, and separate validation boundaries.

Technical breakdown

Reachability-filtered analysis units reduce code review noise

OpenAnt decomposes codebases into smaller analysis units and then filters them by reachability, meaning it only spends deeper effort where code paths are actually exposed to a plausible attacker path. That is a practical response to the scale problem in LLM-assisted review. Static analysis alone often flags issues without enough context to prove exploitability, which creates triage fatigue and weakens trust in findings. By narrowing the candidate set first, the pipeline improves signal quality before semantic reasoning is applied.

Practical implication: security teams should prioritise workflows that rank findings by exploit reachability before human review or remediation begins.

Adversarial verification tests exploitability under attacker constraints

The second stage does not ask whether a pattern looks suspicious in the abstract. It asks whether the issue can be turned into an exploit under realistic attacker constraints, which is a different and more useful question. This is the bridge between code smell detection and actual vulnerability discovery. For LLM-assisted analysis, that matters because model reasoning can overfit to textual similarities while missing execution reality. Adversarial verification forces the system to prove that a candidate flaw survives scrutiny from the standpoint of an attacker with bounded access and limited context.

Practical implication: teams should treat exploitability testing as a validation layer, not a follow-on activity after triage.

Sandboxed dynamic validation closes the loop on LLM reasoning

OpenAnt generates exploit environments in disposable sandboxed containers and discards them after use. That gives the system a way to test whether a hypothesised weakness behaves as expected when code actually runs, rather than when a model merely reasons about it. This is important for complex repositories because LLM output can be directionally right yet operationally wrong. The validation stage also helps reduce false positives by separating theory from runtime evidence, which is critical when securing open source dependencies and AI-assisted development pipelines.

Practical implication: organisations should isolate exploit validation from production systems and require runtime confirmation before escalating findings.

NHI Mgmt Group analysis

OpenAnt reflects the growing need for verification-first security in AI-assisted code review. LLM reasoning can improve vulnerability discovery, but it does not remove the need to prove exploitability in context. The article shows that scale, cost, and context management are the real constraints, not model intelligence alone. For practitioners, the lesson is to design review pipelines around validated risk, not model confidence.

Repository-scale analysis creates a context-management problem that traditional tools do not solve. Once analysis moves from a single file to a whole codebase, the question becomes which paths matter, which dependencies are reachable, and which findings can be defended under attacker pressure. That is a governance issue as much as a technical one because teams need defensible triage, not just more output. The practical takeaway is to align review scope with reachable attack surface.

AI-assisted vulnerability discovery will increasingly overlap with identity and tool access governance. As repositories include agents, extensions, MCP servers, and other connected tooling, security review must account for what those components can reach and execute. That is where NHI thinking becomes relevant: machine credentials, scoped tokens, and delegated tool access are part of the same exposure chain. Practitioners should govern code and identity together, not as separate control planes.

“Verification debt” is the new failure mode in LLM security workflows. Teams can accumulate findings faster than they can prove or dismiss them, which creates backlog risk and weakens trust in the process. OpenAnt’s multi-stage design is a response to that debt because it adds exploitation proof and runtime validation before the queue grows unmanageable. Practitioners should measure whether their review process can verify, not just detect.

Open-source AI security tooling now needs to be judged by operational boundaries, not by novelty. A research pipeline is only valuable if it can keep costs bounded, context intact, and results reproducible across real codebases. The article’s evaluation across OpenSSL, WordPress, and Flowise shows that the market is moving toward practical assurance rather than model theatre. For practitioners, the standard is whether the workflow produces decisions you can act on.

From our research:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, according to AI LLM hijack breach.
• From our research: Only 44% of organisations have implemented any policies to manage their AI agents, despite 92% agreeing that governing AI agents is critical to enterprise security, according to The 2026 Infrastructure Identity Survey.
• For adjacent guidance: Explore NHI Lifecycle Management Guide for provisioning, rotation, and offboarding patterns that reduce standing exposure in toolchains and automation.

What this signals

Verification-first workflows will become the baseline for AI-assisted security review. As codebases absorb more AI-generated and AI-reviewed changes, teams will need to prove that findings are exploitable rather than merely plausible. That shifts operational maturity toward validation depth, bounded testing, and clearer decision criteria for remediation.

Repository security and identity governance are converging around delegated tool access. AI coding assistants, extensions, and connected services can all act through credentials, tokens, or service accounts, which means review pipelines must track what those identities can touch. The control question is no longer just whether code is safe, but whether the surrounding machine identities are scoped tightly enough to keep analysis and execution separate.

OpenAnt’s biggest signal is that scalable security tooling now needs a measurable discard rate. If a workflow cannot show how many findings survive verification, the organisation cannot tell whether it is gaining assurance or simply generating noise. That is the operational version of verification debt, and it is where governance teams should focus next.

For practitioners

• Prioritise reachable attack paths first Filter vulnerability findings by code reachability before assigning triage work, so reviewers spend time on paths an attacker can actually hit. This is especially important in large repositories where false positives rapidly overwhelm backlog capacity.
• Add exploitability verification to code review Require a second-stage check that tests whether a candidate issue can be driven to impact under bounded attacker constraints. Treat that validation as part of the review workflow, not as an optional follow-up.
• Isolate dynamic testing in disposable sandboxes Run exploit generation and runtime validation in sandboxed containers that are destroyed after use, so proof-of-concept work cannot leak into development or production environments.
• Map AI tools and repository access together Inventory which agents, extensions, and connected services can reach source code, secrets, and build pipelines. Where those tools use scoped tokens or delegated access, review them as part of the same control surface.
• Measure false-positive pressure as a governance signal Track the ratio of findings that survive validation versus those discarded after verification. If the discard rate is high, your detection pipeline may be producing output faster than the organisation can trust it.

Key takeaways

• AI-assisted vulnerability discovery is only useful when findings are validated against reachable, exploitable paths.
• OpenAnt’s 97% attack-surface reduction claim shows why context management is now a core security control, not a convenience feature.
• Practitioners should combine static filtering, adversarial verification, and sandboxed runtime checks before they trust LLM-generated findings.

Key terms

• Reachability filtering: A method for shrinking the analysis surface to code paths that can actually be invoked from an external or attacker-controlled entry point. It helps teams avoid spending time on dead code and unreachable findings, which improves triage quality and makes downstream validation more meaningful.
• Adversarial verification: A validation step that asks whether a suspected weakness can survive realistic attacker constraints, not just whether it looks suspicious in source code. It bridges the gap between static detection and exploitability by testing the finding as an attacker would, within bounded conditions.
• Verification debt: The backlog risk created when a security process generates more findings than it can confidently prove or dismiss. In AI-assisted review, this debt grows quickly if validation is weak, causing triage fatigue, lower trust in alerts, and slower remediation decisions.
• Sandboxed dynamic validation: A runtime testing method that executes exploit hypotheses in isolated, disposable environments. It lets teams observe real behaviour without exposing production systems, which is essential when proving whether a vulnerability is operationally exploitable rather than just theoretically possible.

What's in the full article

Knostic's full arXiv paper covers the operational detail this post intentionally leaves for the source:

• The multi-stage pipeline design, including how reachability filtering reduces the analysis set before LLM reasoning begins
• The evaluation methodology across OpenSSL, WordPress, and Flowise, including what the system found in each codebase
• The sandboxed exploit-generation workflow, with enough detail to understand how validation environments are created and discarded
• The paper's performance and false-positive comparisons, which are the implementation details teams need before adapting the approach

👉 Knostic's full paper covers the pipeline design, evaluation results, and validation workflow in detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners connect identity controls to automation, tool access, and emerging AI-driven workflows.