Why do sandboxed agent evaluations need per-run isolation rather than a shared working directory?

Why This Matters for Security Teams

Per-run isolation is not just a cleanliness preference. In agent evaluations, the working directory often becomes part of the attack surface, because the agent can read, write, rename, delete, and sometimes chain files into later steps. When a shared workspace is reused, one run can leave behind artifacts that distort the next run’s outcome, mask unsafe behavior, or create false confidence in a model that only appears stable because it is inheriting prior state.

This matters even more for agentic systems that interact with tools, code, and secrets. A shared directory can turn a failed attempt into an accidental dependency, where the next evaluation succeeds because of leftover files rather than correct reasoning. That undermines reproducibility, which is central to security testing, governance, and model comparison. Guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both support stronger controls around evaluation integrity, traceability, and operational reliability.

In practice, many security teams discover evaluation contamination only after a “successful” agent run cannot be reproduced without the hidden leftovers that made it work.

How It Works in Practice

Per-run isolation means each evaluation gets its own filesystem boundary, temporary workspace, and cleanup lifecycle. The goal is to ensure that every run starts from the same known baseline and that any files produced during execution belong to that run alone. For agents with tool use, that baseline often includes a fresh directory tree, unique identifiers, fixed test inputs, and explicit teardown after scoring.

Operationally, this can be implemented with ephemeral containers, VM snapshots, unique temp paths, or namespace-backed sandboxes. The specific mechanism matters less than the guarantee: no shared mutable state across runs. For agent testing, the workspace should be reset before every attempt, and write permissions should be limited to the minimum required for the task. If the agent needs to inspect inputs but not persist outputs, the directory can be mounted read-only with a separate scratch area for generated artifacts.

• Use a fresh workspace per run, not per test suite.
• Hash or snapshot inputs so the baseline is auditable.
• Store outputs in run-scoped folders for attribution.
• Delete or archive artifacts before the next evaluation begins.
• Log filesystem actions so post-run review can separate agent behavior from environment state.

This approach also supports stronger analysis of tool misuse, prompt injection fallout, and unexpected file access patterns, which are increasingly relevant in agent security work. The MITRE ATLAS adversarial AI threat matrix is useful here because it helps map how attackers or malformed prompts can influence agent behavior through the surrounding environment. These controls tend to break down when evaluation runners reuse persistent network mounts or host-level caches because hidden state survives beyond the intended run boundary.

Common Variations and Edge Cases

Tighter isolation often increases runtime overhead and storage cost, requiring organisations to balance reproducibility against throughput. That tradeoff is real, especially when thousands of agent evaluations must be executed quickly or when models need access to large datasets that are expensive to clone.

Current guidance suggests there is no universal standard for how much persistence is acceptable, but the safest practice is to keep anything mutable outside the shared execution path. For example, shared model weights, package caches, or immutable fixtures may be acceptable if they cannot be altered by the agent and cannot influence scoring artifacts. The key distinction is between read-only shared dependencies and writable state that can leak across runs.

Edge cases appear when evaluations intentionally test memory, long-horizon workflows, or multi-step tasks that span sessions. In those cases, the environment should still remain run-scoped, with any intended carryover explicitly serialized and versioned rather than silently inherited from the previous workspace. For broader threat modeling of agent behavior, the CSA MAESTRO agentic AI threat modeling framework is helpful for separating intended state from accidental residue. Where isolation is weakened to support debugging, the evaluation no longer measures the agent alone, but the agent plus whatever the last run left behind.

Related resources from NHI Mgmt Group

• How do you know if sandboxed agent investigations are actually working?
• Container Isolation Per Agent
• When does AI agent access become an IAM risk rather than an automation benefit?
• When does an AI agent become an NHI risk rather than a usability feature?