Notifications
Clear all
Topic starter
21/08/2026 4:50 pm
TL;DR: CyberGym performance is constrained less by raw model capability than by the surrounding agent system, with a purpose-built environment lifting success from roughly 28% to 53% and earlier comparisons topping out near 20%, according to depthfirst. The practical lesson is that agentic AI risk is a system design problem: tooling, runtime feedback, and task structure shape outcomes as much as the model itself.
NHIMG editorial — based on content published by depthfirst: Agent Capability Is a System Design Problem: Lessons From a 90% Improvement on CyberGym
Questions worth separating out
Q: How should security teams govern AI agents that can choose tools at runtime?
A: Security teams should govern runtime agent choice as an access event, not as a simple application action.
Q: Why do AI agents need more than better prompting to perform reliably?
A: Because performance depends on the surrounding environment, not just the model’s language ability.
Q: What breaks when AI agents are given broad standing access?
A: Broad standing access breaks governance because the agent can move from one task to another without a fresh authorization check.
Practitioner guidance
- Define each agent as a governed identity Assign every agent, subagent, and automation component its own identity, ownership, and access boundary.
- Limit tool permissions to the minimum viable workflow Separate file editing, recompilation, submission, and data retrieval into distinct tool permissions.
- Instrument runtime feedback with guardrails Let agents observe execution state, but constrain what they can inspect, change, and retain.
What's in the full article
depthfirst's full blog post covers the implementation detail this post intentionally leaves for the source:
- The exact agent scaffolding changes that produced the performance jump across the development set and final test set.
- The instrumentation workflow used to let the agent inspect runtime behaviour and refine proofs of concept.
- The rationale for using specialised subagents and how the architecture divided responsibilities across tasks.
- The benchmark caveats around dataset quality and why some failures may reflect benchmark construction rather than pure model limits.
👉 Read depthfirst's analysis of how agent system design changed CyberGym performance →
CyberGym and agent design: what security teams should take away?
Explore further
21/08/2026 5:08 pm
Agent capability is a system design problem, not a prompt engineering problem. The article’s core evidence is that a redesigned environment delivered a major jump in benchmark performance without changing the basic task. That means organisations should stop assuming that a better model alone will fix agent risk or agent usefulness. The real control surface includes runtime access, orchestration, tool permissions, and observability. For practitioners, the conclusion is simple: govern the system around the agent, not just the model inside it.
A question worth separating out:
Q: How do teams reduce risk when deploying modular agent architectures?
A: Use separate identities for the main agent, support agents, and any tooling that touches execution environments. Give each component only the permissions needed for its stage, then review their logs independently. Modular design reduces context overload, but it only improves security if the identity model is modular too.
👉 Read our full editorial: Agent system design, not prompting, drives CyberGym performance
