Agent harness choice materially changes autonomous red-teaming outcomes

At a glance

What this is: Lasso Security’s comparison shows that the agent harness, not just the model, materially changes autonomous red-teaming results, with large swings in success and multiple outcome flips across the same missions.

Why it matters: IAM, NHI, and AI security teams need to govern the whole agent runtime, because harness choice can alter tool orchestration, prompt packaging, and control effectiveness even when the model and task stay fixed.

By the numbers:

• 43 outright flips across 100 model-and-mission pairings showed that one harness broke into the target at least once while the other never did, according to Lasso Security.
• Objective success rates ranged from 1% to 34% across the harness and model combinations, according to Lasso Security.

Context

Autonomous red-teaming is the use of an AI agent to pursue a goal through multiple turns, tools, and retries rather than following a fixed script. In this study, the question is not whether one model is stronger than another, but whether the harness that wraps the model changes what the agent can actually do.

That matters for identity and access governance because the harness is part of the runtime control plane for an agent. It shapes prompt assembly, tool exposure, loop length, and termination, which means the operational security question is not only what the model knows, but what the surrounding execution scaffold allows it to reach and retain.

Key questions

Q: How should security teams evaluate autonomous agents when the harness changes outcomes?

A: Security teams should evaluate the model and harness together, using the exact runtime stack that will run in production. A standalone model score can hide major differences in tool exposure, prompt packaging, and turn management. The right test is the deployed pair against the real task, with outcomes measured on the full execution path.

Q: Why does self-scoring create false confidence in autonomous red-teaming?

A: Self-scoring fails because the agent’s own judgment is not an independent measure of success. It can misread partial leaks, near misses, or ambiguous behavioural outcomes as wins, which inflates capability and masks control weaknesses. A blind external judge or deterministic validation keeps the outcome separate from the attacker’s narrative.

Q: What are the signs that an agent harness is affecting security outcomes?

A: Look for large swings in success rate across the same model, toolset, and mission, especially when one scaffold repeatedly reaches the target and another does not. Filters that trigger only under one request shape, or tool calls that disappear when middleware changes, are strong signals that the harness is shaping the result.

Q: Should organisations treat agent harness design as an authorisation decision?

A: Yes. The harness determines what the agent can reach, how long it can continue, and how tool calls are packaged across turns, so it directly affects effective privilege. That makes harness design part of authorisation governance, especially when the agent can act as a non-human identity with persistent runtime state.

Technical breakdown

Why the harness changes agent behaviour

An agent harness is the runtime that assembles prompts, dispatches tool calls, manages state, and decides when the loop ends. That makes it more than plumbing. In practice, the harness can reorder instructions, add middleware, change how tool schemas are presented, and influence how often the model re-enters the decision loop. For autonomous red-teaming, those differences matter because the attacker’s persistence, escalation path, and willingness to pivot are shaped by the scaffold as much as by the model weights. When two harnesses package the same model and tools differently, they can produce materially different attack trajectories even before the first tool call lands.

Practical implication: Evaluate the harness as part of the agent control surface, not as an implementation detail.

Why self-scoring agents distort red-team results

Autonomous attackers can overstate their own success if they grade themselves. The article shows why independent judging is necessary: a self-verdict may treat near-misses, partial leaks, or behavioural failures as wins even when they are not. In red-teaming, that creates false confidence and hides the real effect of harness choices. An external judge, blind to the harness and the attacker’s own report, gives a more defensible measure of whether the agent actually achieved the objective. This is especially important when the target outcome is fuzzy, such as essence leaks or behavioural manipulation rather than a simple string match.

Practical implication: Separate attack generation from outcome adjudication so success metrics are not contaminated by the agent’s own judgment.

Why tool orchestration matters more than a single model choice

The study deliberately fixed the model, prompt, tools, targets, and judge, then changed only the harness. Even with one tool dominating the run, the two scaffolds still diverged in how they packaged context and controlled the loop. That is the key technical lesson: tool orchestration is not just about what tools exist, but how they are exposed, ordered, and re-invoked across turns. For agentic systems, this becomes an authorisation problem as much as a model problem, because the execution environment can expand or suppress what the agent can attempt. Once multiple tools or sub-agents are involved, the difference is likely to widen, not shrink.

Practical implication: Test the full model-plus-harness stack under the exact toolset the agent will use in production.

Threat narrative

Attacker objective: The attacker objective is to complete the multi-step mission and reach the target through the agent’s own tool chain, despite runtime controls and guardrails.

1. Entry occurs through the agent harness, which packages the prompt and tools into the exact request the model sees on each turn.
2. Credential or tool access is then mediated by the scaffold, which can expose, suppress, or reshape the capabilities the agent can reach.
3. Escalation happens when the harness loop and middleware allow the agent to persist, pivot, or re-enter tool use in ways that change attack success.
4. Impact is a materially different red-teaming outcome, including attack completion, blocked requests, or inconsistent success rates across identical missions.

Breaches seen in the wild

• Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.
• Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Harness governance is now part of agent governance. The article shows that an agent’s runtime scaffold can materially alter the outcome of the same model, prompt, and tools. That means security teams cannot treat harness selection as engineering convenience; it is a control decision that changes how access, tool invocation, and termination are enforced. For AI security and NHI governance, the practical conclusion is straightforward: the execution layer must be evaluated with the same discipline as the model and the permissions it can exercise.

Autonomous red-teaming exposes a control-plane truth that model benchmarks miss. Benchmarks that fix a single harness silently bake in runtime behaviour that may dominate the result. The article’s 43 flips across 100 pairings show that harness variance is not noise in adversarial work. The named concept here is harness-dependent attack variance, meaning the same agent capability can look materially stronger or weaker depending on the runtime scaffold. Practitioners should treat that variance as a governance signal, not a lab curiosity.

Agent authorisation has to move closer to execution time. The harness decides what the agent can see, when it can retry, and how tool calls are framed turn by turn. That is where authorisation pressure appears in agentic systems: not only at onboarding, but in every loop iteration. This intersects directly with NHI governance because AI agents behave like non-human identities when they are allowed to plan, call tools, and persist state. The practitioner takeaway is to govern the agent runtime as an identity-bearing control surface, not as a passive wrapper.

Independent adjudication should be mandatory for autonomous attacks. The article’s warning about self-scoring agents is broader than red-teaming. Any autonomous system that evaluates its own outcome can inflate success and hide failure modes, especially when the output is ambiguous. That is a governance problem for AI assurance, incident simulation, and agentic security testing alike. Teams should assume that internal scoring will overstate capability unless an external, blind judge or deterministic test condition is in place.

Agentic security will converge on runtime standardisation, not just model selection. The market conversation still overweights model choice, but this study shows that the scaffold can be equally decisive. As agent deployment matures, the differentiator will be how well organisations control the runtime contract around models, tools, and termination. For security programmes, that means the next layer of assurance sits in harness design, harness testing, and harness inventory.

From our research library:

• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• 19% of organisations give AI systems dramatically more access than human employees, nearly one in five granting unrestricted privilege, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Authorisation Guide

What this signals

Harness-dependent attack variance: the same model can appear materially more or less capable depending on the runtime scaffold, so agent governance has to move beyond model-only evaluation. For practitioners, the issue is not simply whether an agent can act, but whether the surrounding execution contract reshapes what it is allowed to reach, retry, or persist.

Autonomous red-teaming also shows why identity security and AI security are converging. When an agent is allowed to plan, call tools, and keep state across turns, it behaves like a non-human identity with dynamic privilege. That makes authorisation, termination, and observability the controls that matter most, especially as 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.

For practitioners

• Benchmark the full agent stack Test the exact model, harness, toolset, and target combination you plan to run in production, rather than scoring the model alone.
• Separate generation from judging Use an external judge or deterministic checks for autonomous red-teaming so the attacker cannot grade its own success.
• Inventory harness-specific control differences Document how each harness assembles prompts, exposes tools, manages state, and terminates loops so authorisation assumptions are explicit.
• Test request-shape sensitivity Probe whether a harmless-looking change in scaffold text or middleware alters filter outcomes, tool reach, or attack persistence.

Key takeaways

• The study shows that harness choice can materially change autonomous red-teaming results, so the runtime scaffold belongs in the security control surface.
• Outcome swings were large enough to produce 43 flips across 100 pairings, which is evidence of harness-dependent variance rather than random noise.
• Teams should benchmark the exact model, harness, tools, and judge together, because agent capability is only meaningful when the full execution stack is tested.

Key terms

• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Autonomous Red-Teaming: Autonomous red-teaming uses an AI agent to pursue a multi-step attack or abuse objective with limited human steering. The system adapts across turns, making it useful for testing how models, tools, and runtime controls behave under adversarial pressure.
• Harness-Dependent Attack Variance: Harness-dependent attack variance is the change in outcome caused by the agent runtime rather than the underlying model alone. It appears when the same model, prompt, and tools produce materially different success rates or attack paths under different scaffolds.

Deepen your knowledge

The NHI Foundation Level course covers NHI governance, agentic AI identity, and machine identity security through the industry's only accredited NHI security programme. It helps practitioners turn runtime access questions into clear governance decisions across AI and identity programmes.