Harness-dependent attack variance is the change in outcome caused by the agent runtime rather than the underlying model alone. It appears when the same model, prompt, and tools produce materially different success rates or attack paths under different scaffolds.
Harness-Dependent Attack Variance in the Agent Runtime
harness-dependent attack variance describes a difference in attack outcome that comes from the runtime scaffold around a model, not from the model weights alone. The same prompt and tools can succeed, stall, or diverge into different attack paths depending on orchestration, memory handling, retries, permissions, and tool wiring.
This matters because evaluation and production often treat “the model” as the unit under test when the real security behaviour is produced by the full agent stack. A harness can mask a weakness, amplify it, or create a failure mode that never appears in a simpler test setup.
Why the Harness Changes Security Behaviour
The harness controls how an agent is invoked, what context it receives, how often it retries, what state it preserves, and which tools are available at each step. Those choices can alter whether an attack reaches a sensitive action, whether a guardrail is actually present at execution time, and whether the agent can recover from a blocked path.
In practice, that means two environments with the same model can produce materially different security outcomes. One runtime may expose a direct tool path, while another inserts review, token refresh, permission checks, or memory resets that break the attack chain before impact.
Common Sources of Variance
Variance often comes from orchestration details rather than the prompt itself. Differences in tool schemas, system-message ordering, state persistence, memory summarisation, hidden retries, and policy wrappers can all change the shape of the attack surface.
Runtime variance is especially visible when the agent can call external systems, because small implementation changes can alter authorization boundaries, secret exposure opportunities, or the sequence in which actions are executed. For readers mapping the broader attack surface, MITRE ATLAS adversarial AI threat matrix is useful for thinking about agentic failure modes, while CISA cyber threat advisories provide the operational backdrop for real-world abuse patterns.
How to Interpret the Term in Testing and Review
This term is best used when comparing behaviour across runtimes, harnesses, or deployment scaffolds rather than across model families. It helps explain why a control may appear effective in one evaluation path but fail under a different orchestration layer, where the effective attack route is longer, shorter, or simply different.
For defenders, the key question is not only whether the model can be induced to misbehave, but whether the surrounding runtime makes that misbehaviour reachable, repeatable, and exploitable. That is why harness-dependent results should be treated as a property of the system, not as a pure model verdict.
Risk and Threat Considerations
Harness-dependent attack variance creates a false sense of confidence when teams test only one scaffold and assume the result generalises. A weak harness may expose tool misuse or context leakage that a stronger harness prevents, while a different orchestration path may re-enable the same attack later in production.
Failure mechanism: attackers exploit differences in runtime state, retries, memory handling, tool routing, or permission checks to find the path where the agent is most permissive or least observable.
Impact: the same underlying system can show inconsistent security posture across environments, leading to missed exploitability, incomplete remediation, and unexpected access to tools, data, or actions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI08 — Cascading Failures | Harness variance changes agent outcomes across runtime paths and execution chains |
| Recommendation — Test multiple harness paths to expose cascading failures in agent execution and containment. | ||
| NIST SP 800-53 Rev 5 | SA-11 — Developer Testing and Evaluation | Harness-dependent variance requires evaluation of security behaviour across representative runtime conditions |
| Recommendation — Validate the agent under representative runtimes and test conditions before release. | ||
| NIST CSF 2.0 | ID.RA-03 — Threat and Vulnerability Identification | Different scaffolds can reveal different exploit paths and effective vulnerabilities |
| Recommendation — Assess how the runtime scaffold changes the system's threat and vulnerability profile. | ||
Practitioner Guidance
Why practitioners should care: treat the harness as part of the security boundary, because it can be the deciding factor in whether an attack path exists at all. Compare outcomes across multiple realistic runtimes, not just one clean benchmark setup.
What to watch for: divergent success rates, different tool sequences, hidden state effects, and guardrails that only work under one execution pattern are all signals that the system’s true exposure depends on the scaffold. The practical lesson is to validate the full agent runtime, not just the base model response.