Topic starter
02/08/2026 9:46 am
TL;DR: Open-source AI security projects fall into three harness families, LLM-led exploit generation, LLM-skill-boosting, and SAST plus LLM hybrids, with different tradeoffs in validation, execution, and patch generation, according to Semgrep. The key shift is that vulnerability discovery is becoming more agentic, but trustworthy findings still depend on adversarial verification and tight scope control.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “Comparing Open-Source AI Code Security Harnesses”.
Key questions
Q: What fails when an LLM security harness skips adversarial validation?
A: False positives become much harder to separate from real vulnerabilities, especially when the model is reasoning fluently over code but no second pass tries to disprove the finding.
A: They start with deterministic analysis that narrows the search space before the model weighs in.
Q: How should security teams govern AI-assisted vulnerability research tools?
A: Teams should treat AI-assisted vulnerability research tools as privileged systems with explicit scope, sandboxing, and review boundaries.
Practitioner guidance
- Define the research lane before choosing the harness Separate exploit-oriented workflows, researcher-skill workflows, and SAST triage workflows so each has its own scope, approval boundary, and success criteria.
- Require adversarial validation for AI-generated findings Make a second independent pass mandatory before a finding is accepted, especially when the first pass came from an LLM reasoning over code context.
- Keep execution privileges narrowly bounded If a harness can compile or run code, isolate it in a sandbox and restrict egress so a research agent cannot create uncontrolled blast radius.
Bottom line: LLM security harnesses are splitting into exploit-generation, skill-boosting, and SAST-plus-LLM patterns, and each category makes different tradeoffs between creativity, reproducibility, and governance.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:38 pm
These harnesses are really governance patterns, not just tooling patterns. The article shows three ways to distribute trust between deterministic analysis, model reasoning, and code execution. That matters because the main security question is no longer whether an LLM can help find bugs, but where proof, validation, and scope control live in the workflow. Practitioners should treat each harness as a different control design for vulnerability research, not as interchangeable AI wrappers.
A few things that frame the scale:
- Software supply chain attacks were projected to cost organisations $60 billion in 2025.
A question worth separating out:
Q: What is the difference between exploit generation and LLM-skill-boosting in vulnerability research?
A: Exploit generation pushes the model toward a crash or proof-of-exploit outcome, while skill-boosting gives the model structured methods for finding, analysing, and verifying issues. One is outcome-driven and narrow, the other is process-driven and composable, so they fit different research goals and risk tolerances.
👉 Read our full editorial: LLM security harnesses are splitting into three distinct categories