RL and SFT improve agent training on held-out tasks

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Patronus AI's blog post: Reinforcement learning and supervised fine-tuning improved held-out agent task performance, but reproducible evaluation and controlled task design were the real governance lesson.

Why it matters: IAM, NHI and AI security teams should care because agent capability gains are only trustworthy when training data, evaluation runs and execution environments are tightly governed.

Context

Agent training quality is often overstated when teams look only at aggregate success rates. In practice, reproducibility depends on separating genuine model improvement from task leakage, environment failure and benchmark drift, especially when AI systems are being trained to use tools, browse interfaces or operate across software environments.

Patronus AI's work sits in the governance gap between model improvement and operational trust. The core issue is not whether a model can score higher once, but whether the result can be reproduced across runs, checkpoints and environments without hidden contamination or brittle evaluation design.

Key questions

Q: How do teams prevent benchmark leakage in agent training?

A: Teams prevent benchmark leakage by keeping training data, prompts, test tasks, and grading rubrics separate, then restricting agent access to hidden evaluation sets. They also version datasets, track provenance, and block reuse of benchmark outputs in fine-tuning, retrieval, or prompt libraries. If an agent can read the answer key or memorise scored examples, the benchmark no longer measures generalisation.

Q: Why do supervised fine-tuning and reinforcement learning both need controlled evaluation for agents?

A: Both supervised fine-tuning and reinforcement learning need controlled evaluation because agents can optimise for the scoring setup instead of the real task. A fixed eval harness, held-out tasks, and unchanged grading rules let teams compare runs fairly, detect reward hacking, and see whether the model learned usable behaviour or only adapted to the test process.

Q: How should teams isolate sandboxed agent execution from grading logic?

A: Teams should run the agent in a separate execution environment from the grader, with only the outputs passed across a narrow interface. The sandboxed task runner should prevent the agent from reading labels, hidden tests, or scoring code, and the grading logic should be isolated from model prompts, tools, and memory so the agent cannot influence its own score.

Q: What makes an agent leaderboard trustworthy?

A: An agent leaderboard is trustworthy when it uses hidden, versioned benchmarks, fixed grading criteria, and independent replayable evaluation runs. It should separate model execution from scoring, record the exact prompt and tool context, and prevent public overfitting to the leaderboard by rotating or withholding test items. Clear provenance makes scores comparable and auditable.

Technical breakdown

Why held-out task design matters in agent training

Held-out evaluation only works when near-duplicate tasks, environment state and grading conditions are controlled. In agent training, a model can appear to improve because it memorises patterns, benefits from a permissive sandbox or exploits task artefacts that should have been excluded from training. Separating whole task groups, holding out expert-solved tasks and fixing evaluation presets are ways to preserve the meaning of the score. Without that discipline, a benchmark measures exposure to the test set more than agent capability.

Practical implication: Treat held-out sets as a governance control, not a convenience metric, and prevent task overlap between training and evaluation.

How reinforcement learning changes tool-using model behaviour

Reinforcement learning for agents rewards successful task completion, so it can shape behaviour that supervised fine-tuning alone does not reliably produce. In tool-using systems, the agent must choose actions, observe consequences and persist through multi-step workflows. That means the reward signal is tightly coupled to the environment, the grader and the action space. If the environment is noisy or the reward is too blunt, the model can optimise the wrong thing or learn only partial competence.

Practical implication: Define rewards against the real task outcome and audit the environment so the policy learns the intended behaviour, not a shortcut.

Agent sandboxing and grading are part of the control plane

When agents can click, edit files and invoke tools, the sandbox and grader become part of the security and reliability boundary. A weak sandbox can expose grading logic, permit data leakage or let the model interact with files and contexts it should not see. That is especially relevant for AI infrastructure workload identity, where training jobs, inference endpoints and environment services all need explicit separation of duties. The control plane is not just the model server; it includes the execution environment, job orchestration and result collection path. See the AI Infrastructure Workload Identity Guide.

Practical implication: Isolate training sandboxes from grading logic and restrict what the agent can read, write and discover during evaluation.

NHI Mgmt Group analysis

Agent evaluation is now a governance problem, not just a model-quality problem. Once models are trained to operate tools, browse interfaces and act across software environments, the integrity of the evaluation stack matters as much as the training recipe. The most important question is whether a score represents actual capability or an artefact of the harness. Practitioner conclusion: treat evaluation design, sandbox boundaries and result provenance as first-class controls in any agent programme.

Reproducibility is the real control variable in post-training pipelines. A model that improves once is not enough to justify operational confidence if the run cannot be recreated under the same dataset, code and environment conditions. This is where agent systems differ from static ML workloads: the execution context is part of the output. Practitioner conclusion: require run-level evidence, not just aggregate metrics, before promoting a checkpoint.

AI infrastructure workload identity becomes relevant when the agent stack itself is an execution system. Training jobs, inference endpoints, sandboxes and graders all have distinct runtime identities and permissions, and those identities determine what the agent can access during evaluation. When the platform is broad enough to span training and rollout, identity separation is part of benchmark trust. Practitioner conclusion: map each stage of the pipeline to a scoped runtime identity and keep grading paths isolated.

RL can improve agent behaviour without resolving trust in the underlying environment. The result lift is meaningful, but the same setup also revealed false positives, silent failures and environment weaknesses that a higher score alone would not expose. That means capability growth and governance maturity do not advance at the same speed. Practitioner conclusion: pair RL experimentation with controls that validate the evaluation substrate itself.

Long-horizon agent training will force a new standard for operational evidence. As tasks expand in length and complexity, more of the risk sits in the pipeline that stores trajectories, resumes runs, compares checkpoints and aggregates scores. That pushes teams toward stronger experiment lineage, finer-grained observability and stricter control over task selection. Practitioner conclusion: build for traceability before you scale agent autonomy.

What this signals

Capability gains in agent systems will increasingly be judged by evidence quality, not just raw score movement. As models take on longer-horizon workflows, teams will need to prove that a checkpoint is reproducible, the sandbox is sealed and the grading path is independent. The practical shift is toward evidence-rich experimentation, where provenance and isolation matter as much as the metric itself.

Experiment lineage will become a central design concern for AI operations. The more a team reuses trajectories, resumes runs and compares checkpoints, the more it needs lineage that can explain why a result changed. That is especially true when agents interact with real tools, because the difference between model progress and environment failure can be subtle. Practitioner focus should move to traceability, run isolation and resumable workflows that preserve auditability.

For practitioners

• Separate training, evaluation and grading identities Assign distinct runtime identities and permissions to the training job, the sandboxed task runner and the grading service so the model cannot inspect or influence evaluation logic.
• Hold out near-duplicate tasks across the full pipeline Group reworded or near-copy tasks together and keep each group entirely in either training or evaluation to prevent memorisation from inflating results.
• Record deterministic run metadata for every checkpoint Capture code state, git commit, uncommitted changes, dataset hashes and environment versions so a leaderboard score can be reproduced later.
• Audit the sandbox for unintended visibility Test whether the agent can read grader artefacts, access hidden project files or observe information that should stay outside the task boundary.
• Treat environment failures as a separate control issue Track silent zero-reward runs, node-level faults and false completions independently from model quality so infrastructure problems do not get mistaken for poor learning.

Key takeaways

• Agent training scores can improve materially, but the larger governance question is whether the harness proves real capability or just cleaner evaluation.
• The post-training stack matters because task splits, sandbox boundaries and run metadata determine whether benchmark gains are trustworthy.
• For agent programmes, reproducibility and environment isolation are as important as model optimisation when deciding whether to promote a checkpoint.

Related NHI Mgmt Group resources: AI Infrastructure Workload Identity Guide · Browser and Computer-Use Agent Security Guide · Fine-Tuning and LoRA Security Guide

Key terms

• Supervised Fine-Tuning: Supervised fine-tuning is the stage where a base model is trained on examples of preferred responses before reinforcement learning begins. It helps the model imitate human-written or human-selected outputs more closely. In RLHF pipelines, this step gives the system a better starting point for later preference-based optimisation.
• Reinforcement learning: Reinforcement learning is a training method where a model improves by receiving rewards or penalties based on outcomes. In security research, it is useful when the environment can confirm whether an action succeeded, such as whether a payload actually executed in a browser.
• Held-Out Evaluation: Held-out evaluation is testing performed on data the model did not train on. It is the main check that separates memorisation from genuine usefulness, and it is essential when deciding whether a tuned model beats the base model or a prompt-only approach.
• Sandbox: A sandbox is a controlled execution environment that isolates an agent's runtime from the rest of the estate while still allowing defined actions and tool use. For identity teams, the sandbox is part of the trust boundary because it carries the agent's access, state, and audit trail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management. It helps security and platform teams translate identity controls into operational programmes that can support agentic systems.