Continuous AI pentesting exposes the real build-vs-buy trade-off

At a glance

What this is: This is Synack's analysis of Dow's AI pentesting build-versus-buy decision, with the key finding that continuous coverage and a purpose-built harness matter more than simply standing up a model.

Why it matters: It matters because AI-assisted security testing still depends on human validation, operational guardrails, and access to changing assets, all of which intersect with IAM, NHI governance, and broader security operations.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities.

👉 Read Synack's analysis of Dow's build-vs-buy decision for AI pentesting

Context

AI pentesting sits at the intersection of automation, assurance, and governance. The practical problem is not whether a model can generate attack ideas, but whether a testing programme can keep pace with a changing attack surface, separate exploitable findings from noise, and preserve human judgment where business risk is determined. That is why the primary question is operational control, not model novelty.

For identity and access teams, the relevance is in the control plane around the agent: which assets it may touch, what actions it may take, and who validates the result before downstream remediation begins. That makes this more than a tooling choice. It is a governance choice about how machine assistance is allowed to participate in security validation, and Dow's starting point is not unusual for teams trying to scale coverage without losing confidence.

Key questions

Q: How should security teams decide whether to build or buy AI pentesting capabilities?

A: Teams should compare the full operating cost, not just the first prototype. Building means ongoing model tuning, orchestration, token management, guardrails, and validation work every time the underlying model changes. Buying makes more sense when the goal is continuous coverage and the internal team needs to stay focused on assets, risk decisions, and remediation ownership.

Q: Why do application security tools still need human validation?

A: Because a scanner can identify unusual behaviour, but it cannot reliably decide whether the behaviour is intended, tolerated, or exploitable in your organisation. Human validation is what separates a noisy finding from a real control failure. That matters most where authorisation, business rules, and identity context intersect.

Q: What do security teams get wrong about agentic pentesting swarms?

A: They often assume more agents automatically means better coverage. In reality, multi-agent systems need tighter orchestration, scope enforcement, and handoff controls, or they drift into noisy or unsafe behaviour. The question is not how many agents you can run, but whether each task, permission, and stop condition is governed.

Q: How can organisations govern AI-assisted testing without losing speed?

A: Use policy to separate candidate generation from decision-making. Let the agent gather evidence and propose findings, then require a human or approved workflow to confirm exploitability, deduplicate results, and route remediation. That preserves speed while keeping reporting, triage, and accountability under control.

Technical breakdown

Why continuous pentesting beats point-in-time testing

Point-in-time penetration testing measures a system at a moment, but modern attack surfaces move continuously. New subdomains, cloud storage endpoints, exposed services, and workflow changes appear faster than quarterly or annual tests can chase them. Continuous pentesting uses a persistent workflow that keeps testing aligned to actual exposure rather than audit cadence. The technical value is not just volume, but timing, because the finding only matters if it lands while the asset is still reachable and relevant. In practice, this turns testing into an always-on control loop rather than a periodic event.

Practical implication: feed new assets into testing automatically so coverage tracks exposure in near real time.

Why the harness matters more than the model

An AI pentesting model is only as useful as the harness that constrains it. The harness defines rules of engagement, prioritisation, tool use, and decision logic for when a result should be treated as exploitable or ignored. Raw model output tends to overgenerate, miss business context, and confuse lab success with enterprise reality. A purpose-built harness adds the judgment layer that experienced pentesters normally supply, including what to test next and when to stop. Without that layer, the system becomes an expensive prompt engine rather than a testing programme.

Practical implication: design the orchestration layer first, then evaluate models inside that control framework.

Why human validation still governs exploitability

Human validation remains the decision point because a detected weakness is not the same as a credible finding. Pentesting output has to be checked against existing compensating controls, application behaviour, and business tolerance. AI can expand surface coverage, but it cannot reliably know whether a potential issue is already mitigated elsewhere or whether chaining conditions actually exist in production. That validation step is especially important in complex enterprise environments where false positives can consume remediation time and erode trust in the programme.

Practical implication: require human sign-off before findings move into remediation queues or executive reporting.

NHI Mgmt Group analysis

Continuous validation is becoming the real security control, not a periodic test. When attack surfaces expand faster than formal review cycles, point-in-time pentesting leaves blind spots that governance cannot explain away. Continuous testing aligns better with modern cloud and application change rates, but only if the workflow is tied to asset discovery and prioritisation. The practitioner conclusion is that coverage cadence now matters as much as scanner depth.

The harness is the control plane for agentic security testing. In AI pentesting, model quality is secondary to orchestration, policy enforcement, and decision thresholds. That mirrors a broader NHI pattern: the identity of the agent is not the only question, the permissions and guardrails around it determine whether it is safe to let it act. The practitioner conclusion is to govern the wrapper before the model.

Human validation remains the boundary between automation and assurance. AI can accelerate reconnaissance and candidate finding, but it cannot on its own decide exploitability in context. That is the same governance failure pattern seen in many NHI and agentic AI programmes, where speed is mistaken for certainty. The practitioner conclusion is to preserve expert review for findings that will drive remediation or reporting.

AI pentesting is moving toward specialised swarms, which will make policy enforcement more important. A swarm can distribute tasks by skill and stage, but it also increases the number of execution paths that need constraints. The named concept here is agent orchestration drift: the tendency for multi-agent systems to wander outside their intended testing scope as task complexity rises. The practitioner conclusion is to treat every agent handoff as a control point.

What this signals

AI-assisted pentesting will increasingly be judged on control quality rather than model novelty. Agent orchestration drift: as teams add specialised agents and more handoffs, the risk is not just false positives but scope leakage across testing steps, which makes policy enforcement and auditability central to programme design. For identity-heavy environments, that means the agent's permissions and stop conditions deserve the same scrutiny as any other privileged workflow.

The practical signal for security leaders is that validation capacity becomes a bottleneck, not a nice-to-have. When AI can produce findings faster than humans can assess them, triage design, deduplication, and escalation rules matter more than raw throughput. Teams that already manage privileged access and secrets lifecycle controls will be better placed to govern the testing agent as a bounded system rather than a free-running tool.

For practitioners

• Automate asset handoff into pentest queues Connect discovery output for cloud storage, forgotten subdomains, and other newly exposed assets to the testing workflow so coverage follows the live attack surface rather than a calendar.
• Require a harness-defined rules of engagement layer Encode action limits, scope boundaries, and decision thresholds in the orchestration layer before any model is allowed to interact with targets.
• Keep human validation before remediation routing Review candidate findings for exploitability, duplication, and compensating controls before they move into downstream fix tracking or executive reporting.
• Test against production-like complexity, not lab-only paths Benchmark AI pentesting workflows on real login flows, inconsistent backend versions, and ordinary enterprise edge cases, because capture-the-flag success does not predict live performance.

Key takeaways

• AI pentesting only works as a governed system, not as a raw model, because the harness determines scope, safety, and decision quality.
• Human validation remains the step that separates plausible output from exploitable risk, especially in complex production environments.
• Continuous coverage, automated asset handoff, and policy-bound orchestration are the controls that make AI-assisted testing operationally credible.

Key terms

• Continuous Pentesting: A security validation model that checks exploitability repeatedly as systems change, rather than at a single scheduled point. It is designed for environments where releases, integrations, and attack surfaces move quickly, so evidence remains aligned to the current application state instead of a past snapshot.
• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Human Validation: A review step where a qualified person confirms whether an AI-generated finding is truly exploitable and relevant. It prevents false positives from entering remediation queues and keeps business context inside the decision process.
• Agent Orchestration Drift: The tendency for a multi-agent system to move beyond its intended scope as task complexity rises. It usually appears when handoffs, permissions, or stop conditions are weak, causing noise, unsafe actions, or loss of testing focus.

What's in the full article

Synack's full article covers the operational detail this post intentionally leaves for the source:

• How Dow operationalised continuous pentesting against newly exposed assets and changing web-facing environments
• How Synack structured Sara AI Pentesting as a harnessed swarm rather than a single general-purpose agent
• How token usage, engineering effort, and maintenance overhead shaped the build-versus-buy decision
• How human validation is inserted into the workflow before findings reach downstream teams

👉 The full Synack post covers the Dow use case, agent harness design, and the cost logic behind buying versus building.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It gives practitioners a structured way to connect machine access, privilege, and lifecycle control to broader security programmes.