Topic starter
06/11/2026 6:36 pm
TL;DR: Supervised fine-tuning plus reinforcement learning lifted held-out GitHub computer-use performance from 35.6% to 47.8%, while Unreal Engine tasks rose from 64.3% to 71.5%, according to Patronus AI research; RL alone performed similarly, and the post-training stack focused on reproducible evaluation and leaderboard comparison. The result suggests that reliable agent improvement depends as much on controlled evaluation and task design as on model updates.
Editorial analysis by NHI Mgmt Group, based on content published by Patronus AI: “We Built Our Own Post-Training Stack to Measure What Our Tasks Are Worth”.
Key questions
Q: How do teams prevent benchmark leakage in agent training?
A: Teams prevent benchmark leakage by keeping training data, prompts, test tasks, and grading rubrics separate, then restricting agent access to hidden evaluation sets.
Q: Why do supervised fine-tuning and reinforcement learning both need controlled evaluation for agents?
A: Both supervised fine-tuning and reinforcement learning need controlled evaluation because agents can optimise for the scoring setup instead of the real task.
Q: How should teams isolate sandboxed agent execution from grading logic?
A: Teams should run the agent in a separate execution environment from the grader, with only the outputs passed across a narrow interface.
Practitioner guidance
- Separate training, evaluation and grading identities Assign distinct runtime identities and permissions to the training job, the sandboxed task runner and the grading service so the model cannot inspect or influence evaluation logic.
- Hold out near-duplicate tasks across the full pipeline Group reworded or near-copy tasks together and keep each group entirely in either training or evaluation to prevent memorisation from inflating results.
- Record deterministic run metadata for every checkpoint Capture code state, git commit, uncommitted changes, dataset hashes and environment versions so a leaderboard score can be reproduced later.
Bottom line: Agent training scores can improve materially, but the larger governance question is whether the harness proves real capability or just cleaner evaluation.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
06/11/2026 6:41 pm
Agent evaluation is now a governance problem, not just a model-quality problem. Once models are trained to operate tools, browse interfaces and act across software environments, the integrity of the evaluation stack matters as much as the training recipe. The most important question is whether a score represents actual capability or an artefact of the harness. Practitioner conclusion: treat evaluation design, sandbox boundaries and result provenance as first-class controls in any agent programme.
A question worth separating out:
Q: What makes an agent leaderboard trustworthy?
A: An agent leaderboard is trustworthy when it uses hidden, versioned benchmarks, fixed grading criteria, and independent replayable evaluation runs. It should separate model execution from scoring, record the exact prompt and tool context, and prevent public overfitting to the leaderboard by rotating or withholding test items. Clear provenance makes scores comparable and auditable.
👉 Read our full editorial: RL and SFT improve agent training on held-out tasks