Human-in-the-loop security testing is the control AI autonomy still lacks

At a glance

What this is: This is an independent analysis of Terra Security’s argument that fully autonomous AI pentesting is not yet safe enough to replace human judgment, especially for intrusive actions and business-logic testing.

Why it matters: It matters because security teams adopting agentic AI in testing, red teaming, or SOC workflows need clear accountability and control boundaries before they let software act on live environments.

By the numbers:

• A single incident can spiral into the billions, as seen during the 2025 Jaguar Land Rover breach, which cost the company upwards of £2 billion.
• A Fortune 100 Security Director reported that Terra doubled the web app attack surface tested without doubling costs or internal resources.

Context

Human-in-the-loop security testing sits at the point where AI speed meets operational safety. In practice, the issue is not whether agents can generate volume, but whether they can distinguish a harmless probe from an action that could disrupt a production workflow or distort a finding.

That distinction becomes especially important when offensive testing moves from scanners to agentic systems that can take multi-step actions across applications, APIs, cloud infrastructure, and network layers. In security programmes that are also experimenting with agentic AI elsewhere, the control question is the same: who governs the action boundary, and who is accountable when software crosses it?

Terra Security’s article argues that pure autonomy is still atypical for safe enterprise testing because the judgment burden has not gone away, even if the execution burden can be shared with AI.

Key questions

Q: What breaks when AI pentesting agents are allowed to act without approval gates?

A: The main failure is blast-radius expansion. An agent that can discover a viable exploit path may also execute it, turning validation into unintended disruption or data exposure. Without approval gates, the organisation loses the boundary between finding risk and creating it, which is especially dangerous in production environments where testing and impact can converge quickly.

Q: Why do autonomous testing tools still need human oversight?

A: Autonomous testing tools still need human oversight because auditors and governance teams need validated evidence, not just generated findings. Human review confirms exploitability, checks safety, and ensures the report can support accountability. Without that control, penetration testing may become faster but less credible, especially in regulated environments where assurance artefacts must be defensible.

Q: What are the signs that an AI pentesting system is producing unreliable findings?

A: Common warning signs include findings that lack concrete exploitation evidence, reports based only on weak surface indicators, and inconsistent results when the same test is repeated. A high false positive rate is another signal. If the system often infers a vulnerability from suspicious behavior without proving impact, its conclusions need tighter verification.

Q: Who should be accountable when AI-assisted IT actions affect production systems?

A: Accountability should sit with the team that owns the workflow, not with the AI tool itself. The human sponsor, the platform owner, and the security function all need defined responsibility for approval, scope, and review. If no one can name the accountable owner, the access model is too weak for production use.

Technical breakdown

Why fully autonomous pentesting creates a control problem

Fully autonomous testing changes the control problem from observation to action. A scanner can enumerate and classify; an agentic tester can probe, follow chains, and trigger side effects. That raises the risk of false positives, noisy results, and unintended disruption because the system optimises for execution, not business context. The hard limit is not technical reach, but judgement under uncertainty. In live environments, the tester must know when a payload is only a test and when it could alter real data or availability. That is why autonomy without human approval becomes an operational risk, not just a tooling choice.

Practical implication: constrain intrusive actions behind human approval in any environment where live transactions or customer-facing workflows can be affected.

Human judgment is the missing layer in business-logic testing

Business-logic flaws are different from routine misconfigurations or known CVEs. They sit in workflow design, edge cases, and intent, so finding them requires understanding what the application is meant to do, not just what it technically allows. AI can accelerate reconnaissance, replay standard checks, and expand coverage, but it still struggles to tell a meaningful workflow abuse path from harmless noise. That is why a human-in-the-loop model keeps the tester responsible for interpreting whether an issue is exploitable in context and whether the evidence is strong enough to escalate.

Practical implication: use human reviewers to validate workflow abuse paths before you treat agent-generated findings as exploitable findings.

Continuous testing needs a named control owner

The article’s strongest governance point is that continuous testing is only trustworthy when a specific human owns the engagement. The model Terra describes is not AI replacing people, but AI working under a credentialed security professional who approves intrusive activity and interprets results. That matters because continuous change in web apps, APIs, cloud infrastructure, and network exposure means testing is now an always-on control, not a periodic project. Without a named owner, continuous offensive testing becomes a liability chain with no clear accountability point.

Practical implication: assign explicit human ownership for continuous offensive testing and define which actions remain AI-assisted versus human-approved.

NHI Mgmt Group analysis

Human-in-the-loop is a governance control, not a usability preference. The article correctly reframes AI pentesting as an accountability problem before it is a speed problem. In agentic security testing, the central risk is not simply missed findings, but a tool taking an intrusive action without understanding production context. That makes named human oversight the control boundary that keeps agentic execution inside organisational tolerance.

Pure autonomy does not solve security validation debt. It shifts the burden from analyst effort to judgement quality, and those are not the same thing. Agentic testing can increase coverage, but coverage without contextual interpretation produces noisy assurance at best and unsafe action at worst. Practitioners should treat the model as a way to reallocate labour, not remove accountability.

Business-logic testing is where agentic AI still depends on people. Misconfigurations and known vulnerabilities can be automated at scale, but workflow abuse requires reasoning about intent, process, and consequence. That is the exact place where human judgment remains the differentiator between a signal and a false alarm.

Continuous offensive testing will converge with broader AI governance expectations. As organisations delegate more runtime decisions to AI, they will need explicit approval thresholds, audit trails, and action ownership. This is the same governance pattern emerging across agentic AI security more broadly: automation can assist, but it cannot be the final authority when live systems are at stake.

Attack-surface validation debt: continuous testing is only credible when the organisation can prove who approved each intrusive action, what was tested, and how the result was validated. That is the operational standard security teams will need as AI-driven testing becomes normal.

From our research library:

• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• Gartner predicts that by the end of 2026, 40% of enterprise apps will feature task-specific AI agents.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Attack-surface validation debt: organisations that let agents test continuously still need a human approval layer, because the real governance burden shifts to deciding which actions are safe to execute in production. The model only works when that approval boundary is explicit and auditable.

As AI testing expands across web apps, APIs, cloud infrastructure, and network surfaces, the control question becomes less about raw coverage and more about trustworthy escalation. Continuous validation can reduce blind spots, but only if the programme can prove who owned each intrusive step and why it was allowed.

Security leaders should expect human-in-the-loop testing to become the reference model for broader agentic security operations: AI can accelerate execution, but accountable people must remain the final authority over live-environment risk.

For practitioners

• Define an intrusion approval boundary Separate reconnaissance, benign validation, and intrusive exploitation into explicit approval tiers so AI agents never self-authorise actions that could affect production data or availability.
• Assign named human owners for continuous testing Make a credentialed security professional accountable for every engagement, including who can override the agent, stop the test, and certify a critical finding.
• Validate business-logic findings manually Require a human reviewer to confirm workflow abuse paths, replay the evidence, and decide whether an AI-generated issue is truly exploitable in context.
• Track when AI-assisted testing crosses into live risk Log every agent action that touches customer-facing flows, privileged functions, or state-changing operations so you can audit what was attempted and why.

Key takeaways

• Fully autonomous pentesting can increase speed and coverage, but it also creates false positives, production risk, and accountability gaps when agents act without human judgment.
• The key weakness is not scanning depth alone. It is the absence of business context when software has to decide whether an action is safe, meaningful, or intrusive.
• Human-in-the-loop testing preserves AI scale while keeping approval, validation, and escalation under named human control, which is the practical standard for live environments.

Key terms

• Human-in-the-loop Security: Human-in-the-loop security is a control model where people review, approve, or override important system actions before they take effect. It adds human judgment to automated workflows for access, data use, model outputs, and incident response, especially when AI, NHI, or privileged systems make decisions that could create security, compliance, or safety risk.
• Agentic Testing: Security testing in which software agents can choose actions, sequence steps, and adapt during a validation workflow. In practice, it combines automation with governance, because the agent is not just running a script. It is operating with enough decision-making to require scope limits, auditability, and oversight.
• Business Logic Flaw: A business logic flaw is a weakness in how an application handles intended behaviour, such as permissions, workflow order, or transaction state. These flaws often bypass signature-based checks because the problem is not a malformed input, but a legitimate action used in the wrong sequence or context.
• Approval Boundaries: The policy limits that define which access requests can be approved automatically and which require human review. Strong approval boundaries prevent workflow tools from turning convenience into excessive entitlements or uncontrolled app adoption.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps practitioners build the control thinking needed to govern autonomous systems and their credentials.