Notifications
Clear all
Topic starter
04/09/2026 6:46 pm
TL;DR: Open-source offensive security agents have grown from fewer than five before GPT-4 to roughly seventy by March 2026, but the real divide is not model quality, it is whether the system can execute, retain state, validate findings, and enforce scope, according to FireCompass. The practical shift is toward harness design, where governance and blast-radius controls matter as much as reasoning.
NHIMG editorial — based on content published by FireCompass: LLMjacking: How Attackers Hijack AI Using Compromised NHIs
By the numbers:
- GPT-4 shipped, ve open-source offensive security agents existed before GPT-4 shipped, but by March 2026 Hadrian’s census counted roughly seventy.
- VulnBot’s published results reported 69.05 percent subtask completion and 30.3 percent overall completion on AutoPenBench.
Questions worth separating out
Q: What breaks when AI pentesting scope is not enforced technically?
A: Tests can drift outside intended environments, touch production paths, or produce results that are impossible to trust.
Q: Why do autonomous security agents need governance controls beyond the model?
A: Because the model only decides, while the surrounding system executes, stores state, and records outcomes.
Q: How should security teams evaluate agentic pentest tools?
A: Evaluate the full workflow, not the model alone.
Practitioner guidance
- Define agent scope as a policy object Document which assets, commands, and environments an offensive agent may touch, and enforce that allow list before any tool invocation.
- Separate execution, validation and memory tests Run each candidate tool in a lab and verify that it can execute commands, retain state across long sessions, and prove findings on the live system without duplicating or losing context.
- Require tamper-evident session logging Capture tool calls, outputs, prompts, and operator overrides in logs that support after-action review.
What's in the full article
FireCompass's full analysis covers the operational detail this post intentionally leaves for the source:
- Per-tool breakdowns of execution, memory, and validation behaviour across the named agentic pentest categories.
- Benchmark context and methodology notes that explain why sub-task completion and end-to-end completion should not be confused.
- Detailed guidance on which tools are best suited to lab, staging, or self-hosted deployment patterns.
- Discussion of where the orchestration layer becomes a control problem rather than a tooling convenience.
👉 Read FireCompass's analysis of open-source agentic pentest tools and harness design →
Agentic pentest tools: what they actually cover in a test?
Explore further
04/09/2026 6:46 pm
Open-source agentic pentesting has crossed from experiment to governance problem. The category now contains systems that can reason over tool output, retain state, and act across multiple steps, which makes them materially different from scanners or LLM wrappers. That creates a control question for IAM and security teams: who authorises the agent, what credentials can it use, and how is its scope bounded. The practical conclusion is that agentic tooling should be governed like a privileged workload, not treated like a convenience script.
A question worth separating out:
Q: What should teams do before allowing an agent to run offensive tools?
A: Start with a lab-only deployment, define an allow list of targets and commands, and require human approval for any high-risk step. Then verify that session logs are complete and that the agent cannot exceed the approved blast radius. If those controls are missing, the tool is not ready for sensitive environments.
👉 Read our full editorial: Open-source agentic pentest tools are really harnesses, not scanners
