LLM vulnerability hunting in codebases: what security teams should expect

TL;DR: Teleport says 13 engineers spent three months “pressure washing” a large codebase with frontier LLMs and found that simple, human-guided prompts outperformed complicated agentic harnesses, turning model output into a higher-bandwidth vulnerability discovery workflow. The practical shift is toward expert-led triage and targeted review, not noisy automation.

Editorial analysis by NHI Mgmt Group, based on content published by Teleport: “We Had 13 Engineers Spend Three Months Finding Vulnerabilities with LLMs”.

Key questions

Q: How should security teams use LLMs to find vulnerabilities in large codebases?

A: Use LLMs as a focused review accelerator, not as an autonomous scanner.

Q: Why do simple prompts often outperform complex agentic harnesses in bug hunting?

A: Simple prompts preserve focus, reduce orchestration noise, and keep the reviewer close to the code and the exploit hypothesis.

Q: What are the signs that AI-assisted vulnerability hunting is becoming noisy?

A: The clearest signs are a surge in low-confidence findings, duplicated issue reports, and a growing backlog of items that no one can verify quickly.

Practitioner guidance

• Constrain AI hunts to one component at a time Start with a specific file, module, or subsystem so reviewers can assess exploitability in context instead of drowning in broad candidate output.
• Use expert-led prompts before adding orchestration Begin with simple prompts that ask for a critical vulnerability in a defined scope, then add workflow layers only if they improve signal quality.
• Build triage capacity before expanding AI coverage Assign human reviewers who can validate severity, duplicate reports, and exploit paths, because discovery volume rises faster than decision throughput.

Bottom line: LLM-assisted bug hunting works best when humans tightly scope the task and retain final judgement over exploitability.

What's in the full article

Teleport's full blog post covers the operational detail this post intentionally leaves for the source:

• The team setup and quarter-long workflow used to pressure wash the codebase with frontier LLMs
• The comparison between multi-stage agentic harnesses and simple human-guided prompts
• Examples of how reviewers iterated on findings to find similar vulnerabilities in other code areas
• The security release impact, including the bug-fix outcomes from the quarter's review work

👉 Read Teleport’s analysis of how LLMs are changing vulnerability discovery in large codebases →

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →