Frontier LLM pentesting still depends on strategy over raw power

At a glance

What this is: This is an analysis of updated pentesting-agent benchmark results showing that splitting strategizer and executor roles improves frontier LLM performance and cost efficiency.

Why it matters: It matters because teams building AI-assisted security workflows need to govern delegation, orchestration, and task separation, not just choose a stronger model and assume better outcomes.

By the numbers:

• Claude Opus 5 combines a very strong precision of about 95% with the highest recall in the benchmark.

👉 Read Ethiack's analysis of frontier LLM pentesting strategy and execution

Context

Frontier LLM pentesting is a control problem as much as a model-selection problem. When planning, reconnaissance, exploitation, and validation are split across different components, the quality of orchestration can matter more than raw model capability. That is especially relevant for AI security programmes, where agent behaviour, tool use, and privilege boundaries must be governed rather than assumed safe.

Ethiack’s updated EthiBench results are an example of an operational benchmark, not a theoretical claim. The article shows that some models perform well as strategizers while others are stronger executors, which is the same pattern security teams are likely to see in AI agents that interact with tools, credentials, and security workflows. The starting position is typical of early agentic AI programmes: promising capability, uneven control.

For teams tracking the identity side of agentic systems, the governance issue is not only what the model can do, but what identity it uses while doing it. The split between planning and execution mirrors the need to separate decision authority from action authority in AI workflows.

Key questions

Q: How should security teams govern AI systems that split planning and execution across models?

A: Treat the planner and the executor as separate control points. Give each role a scoped identity, record every delegated action, and verify that one component cannot expand another component’s access without an auditable approval path. That keeps orchestration from becoming implicit privilege escalation.

Q: Why do AI workflows make traditional IAM controls less effective?

A: Traditional IAM controls assume slower change, clear ownership, and periodic review. AI workflows break those assumptions because they can move data quickly, act inside trusted platforms, and rely on service accounts or agents that outlive the task they were created for. That creates standing exposure that manual governance usually misses.

Q: What breaks when AI agents are evaluated as a single model instead of a workflow?

A: Single-model testing hides weak orchestration. A model may look poor when it is asked to plan and execute at once, even though it performs well in one of those roles. Workflow evaluation exposes whether the real problem is reasoning, tool use, or delegation design.

Q: How can teams tell whether AI role separation is improving security or just reducing cost?

A: Look for evidence that the system preserves traceability, approval boundaries, and revocation paths while performance improves. If role separation lowers cost but removes auditability or widens access scope, it is a business optimisation, not a security control.

Technical breakdown

Strategizer and executor roles in AI pentesting

The article distinguishes between a Strategizer, which plans and orchestrates actions, and an Executor, which carries out reconnaissance and exploitation. In agentic systems, this is an important architectural split because the system’s success depends on both reasoning and action quality. A strong strategizer can improve task sequencing, while a strong executor can improve tool use and result generation. The benchmark shows that a model may excel in one role and struggle in the other, which means a single-model evaluation can hide important control differences. For security teams, this is a reminder that AI workflows should be treated as multi-step systems with distinct trust boundaries.

Practical implication: Separate decision authority from action authority, then validate each role independently before connecting them in production workflows.

Why model pairing changes benchmark outcomes

When Claude Opus 5 is fixed as the Strategizer, performance gaps between executors shrink, which suggests the weakest link was often the planning layer rather than the action layer. That is a classic systems effect: the quality of upstream orchestration shapes downstream effectiveness. In practice, pairing a stronger planner with a cheaper executor can raise overall efficiency without requiring the same model to do every task. This matters for AI governance because it creates an opportunity to reduce cost while preserving capability, but only if the interface between roles is tightly controlled and observable.

Practical implication: Use role-specific evaluation to decide where to spend model budget and where to constrain delegation.

Cost efficiency does not remove governance risk

The article shows that some paired configurations approach top performance at roughly half the cost of the best solo setup. That is operationally attractive, but it also increases the importance of governance over orchestration, because cost savings can encourage broader deployment of semi-autonomous workflows. In regulated or sensitive environments, a cheaper agentic stack still needs defined privilege boundaries, logging, and reviewable action trails. The technical lesson is that cost optimisation and control design must advance together, or teams will scale behaviour they cannot reliably explain later.

Practical implication: Tie cost-effective model pairing to explicit logging, approval boundaries, and rollback controls for every action chain.

NHI Mgmt Group analysis

Role separation is becoming the defining control pattern for agentic AI security. The article’s core finding is that the best outcomes come from assigning planning and execution to different models or components. That is not just an efficiency choice. It creates a governance model in which decision-making and action-taking can be evaluated separately, which is closer to how IAM and PAM already treat authority. For practitioners, the key question is whether the system can explain who decided and who acted.

AI pentesting benchmarks are now surfacing an identity problem, not just a model-quality problem. Once an AI system can plan, choose tools, and execute steps, the identity of the system becomes part of the security boundary. That is where agentic AI intersects with NHI governance: the agent needs scoped identity, bounded permissions, and auditable delegation. The same logic applies to any AI workflow that can access credentials, launch tools, or alter security state. Practitioners should treat the agent as a governed identity, not a convenient automation layer.

Cost reduction can accelerate governance debt if role boundaries are not enforced. The benchmark shows that mixed-model setups can deliver near-top performance at lower cost, which will encourage scaling. The problem is that scaling governed capability is different from scaling unmanaged action. In the absence of explicit approvals, traceability, and revocation paths, cheaper agentic workflows can expand the blast radius of mistakes. The practitioner conclusion is simple: cost optimisation should be evaluated only after control design is in place.

Agentic AI testing needs a named concept for the planning gap: strategizer dependence. The article makes clear that execution quality often improves once strategy is stabilised, which means many apparent model weaknesses are really orchestration weaknesses. Strategizer dependence matters because teams may misread poor outcomes as a model failure when the true issue is delegated coordination. For security programmes, that means evaluation must include who plans, who acts, and how those roles are governed end to end.

What this signals

Strategizer dependence is the practical signal this benchmark leaves behind. Teams will increasingly discover that poor agent outcomes are caused less by model quality than by orchestration quality, which means evaluation frameworks need to measure delegation, traceability, and bounded action, not just benchmark scores.

The governance implication is straightforward: as AI agents become more common, the control plane matters more than the model choice. Practitioners should align agent testing with NIST AI Risk Management Framework thinking, then map delegated actions to identity and privilege boundaries that can be reviewed and revoked.

For practitioners

• Separate planning from execution in AI workflows Define distinct strategizer and executor roles for any AI system that can touch tools, credentials, or security data, then evaluate each role independently before combining them.
• Scope agent identity to the task chain Assign each AI workflow a bounded identity with only the permissions needed for one task chain, and revoke those permissions when the chain ends.
• Instrument every delegated action Log the model, role, input context, tool calls, and outcome for each delegated step so you can reconstruct how the system reached a security decision.
• Test cost savings against control loss Use paired-model experiments to compare cost and accuracy, but only approve deployments where logging, review, and rollback remain intact across the full workflow.

Key takeaways

• The article shows that AI pentesting performance improves when planning and execution are governed as separate roles.
• The main security lesson is that orchestration quality can matter more than model capability when agents are allowed to act in multi-step workflows.
• Practitioners should treat agent identity, delegation scope, and auditability as core controls, not optional guardrails.

Key terms

• Strategizer: A strategizer is the component in an AI workflow that decides what should happen next, including task ordering, tool selection, and branching logic. In agentic security systems, it is the planning layer that determines whether later actions are sensible, auditable, and within scope.
• Executor: An executor is the component that carries out the planned actions of an AI workflow, such as running tools, querying systems, or validating results. It turns intent into operational behaviour, so its permissions, logging, and guardrails are a major part of the control design.
• Agent Identity: An agent identity is the set of attributes, credentials and permissions assigned to an autonomous software entity. It is treated as a non-human identity because it can authenticate, act on systems and accumulate access over time, which creates governance, audit and lifecycle obligations similar to other production identities.
• Delegation Boundary: A delegation boundary is the line that separates what a person, service account, or agent may do directly from what it may do on behalf of someone else. It matters because runtime access should follow the delegated relationship, not expand into broad inherited privilege.

What's in the full article

Ethiack's full article covers the operational detail this post intentionally leaves for the source:

• Run-by-run benchmark output for all 7 frontier LLMs across average and cumulative scoring
• The stratifier plus executor pairing results that show where performance changes most sharply
• Relative cost comparisons that explain why some pairings approach top results at lower spend
• Updated EthiBench ground truth context that underpins the newer evaluation set

👉 Ethiack's full post covers the benchmark methodology, role split results, and cost-effectiveness comparisons in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It helps security practitioners design access boundaries and accountability for systems that act independently.