LLM agents still trail humans on complex speedrunning tasks

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Patronus AI's research: Frontier LLM agents can get close to human speedrun performance in simple games, but they remain behind on longer, harder tasks that reward sustained strategy formation.

Why it matters: That matters because security teams assessing agentic AI need evidence of durable decision-making, not just isolated task success, before trusting agents with operational access or autonomous workflows.

Context

Speedrunning is a useful proxy for agent strategy formation because success depends on adaptation, route selection, and the ability to improve from repeated attempts rather than on memorising one fixed answer. In identity and security programmes, that same pattern matters when AI systems must operate across changing conditions, tools, and permissions, especially when they are expected to act with limited oversight.

The benchmark frames a wider governance issue: evaluating an agent only on final output can hide whether it can reason across a long sequence of actions without drifting, stalling, or overfitting to a narrow path. That is relevant to agentic AI governance, where runtime behaviour and recovery from failure are as important as initial capability.

Key questions

Q: How should security teams evaluate long-horizon behaviour in AI agents?

A: Evaluate long-horizon behaviour with repeated runs of the same task, varied initial conditions, and measured success across multi-step workflows, not just single-turn outputs. Look for state drift, tool misuse, compounding errors, and whether the agent stays within its delegated scope when a plan changes. Human review should cover the accountable owner, escalation points, and revocation triggers.

Q: Why do agents that perform well on short tasks still fail in production workflows?

A: Agents often fail in production because short tasks do not expose state management, tool chaining, exception handling, or recovery after partial failure. A model can look competent in isolated prompts yet break when it must preserve context, respect permissions, retry safely, and complete multi-step work without drifting from policy or approved data sources.

Q: What signals show that an AI agent is not really learning from feedback?

A: A common signal is that the agent repeats the same mistake after correction, ignores negative feedback, or changes output style without changing underlying decisions. Another sign is unstable performance across repeated runs, where the agent appears responsive but does not update its action selection, memory use, or tool choices in a durable way.

Q: Should organisations trust an agent that reaches good results only in simple environments?

A: No. An agent that performs well only in simple environments should be treated as unproven for real operations, because tool chains, data quality, permissions and failure handling are usually more complex. Limit its runtime scope, require human approval for high-impact actions, and monitor its decisions with audit logs and rollback controls.

Technical breakdown

Why long-horizon strategy matters for LLM agents

SpeedrunBench tests whether an agent can improve through repeated attempts, not just complete a single instruction. In practice, that means the benchmark measures planning depth, state retention, and whether the system can incorporate feedback from prior runs. For long tasks, the difficulty is not only choosing a good action once, but revising strategy as the environment or route becomes more complex. That makes the benchmark closer to real operational work than many static prompt tests, because the agent must manage progress over time rather than optimise one response in isolation. See the Agentic AI Identity Guide.

Practical implication: treat long-horizon strategy as a distinct capability and test it separately from single-turn accuracy.

What makes a benchmark saturation-resistant

A saturation-resistant benchmark is one where there is still meaningful room for improvement after repeated evaluation. Speedrunning creates that property because there is almost always a faster route, tighter sequence, or better execution path to discover. That is valuable for studying agent development over time, since a task that caps out quickly stops revealing whether a model can continue improving. For practitioners, the lesson is that mature evaluation needs tasks that reward refinement, not just first-pass success, otherwise it becomes too easy to miss capability ceilings or brittle optimisation. Read Top 10 Agent Identity Controls.

Practical implication: use evaluation tasks that continue to expose performance gaps after the first round of tuning.

How game mechanics expose agent reasoning limits

Video games are structured systems with rules, constraints, and repeatable outcomes, which makes them useful for probing agent reasoning. They still demand discovery, timing, and route selection, so they reveal whether the model can infer mechanics rather than merely imitate patterns. In more complex games, the agent must coordinate knowledge across many steps, which is where current systems lose ground. That pattern is relevant beyond games: many agentic workflows also require sustained control, adaptation to changing state, and disciplined execution under practical constraints.

Practical implication: benchmark agents on tasks where understanding the system matters as much as producing the answer.

NHI Mgmt Group analysis

SpeedrunBench is really a test of whether an agent can keep learning inside a live task. The benchmark is not about isolated prompt quality; it is about whether a system can accumulate and apply knowledge across many attempts without losing direction. That is a useful distinction for agentic AI governance, where runtime adaptation is part of the risk profile, not an implementation detail. Practitioners should treat repeated improvement as a control question, not just a model capability question.

The more consequential finding is the gap between first-pass competence and sustained performance. Frontier agents can look strong on shorter, simpler games, yet fall behind when the task stretches over longer horizons and demands course correction. That mirrors the governance problem with autonomous or semi-autonomous systems in enterprise settings: a model that starts well can still fail under drift, complexity, or cumulative state. Security teams should therefore evaluate not just what an agent can do, but how it behaves after the first success or failure.

Strategy formation is becoming a governance issue for agentic systems. The article points to an important concept: adaptive execution debt: the gap between an agent that can act and an agent that can revise its plan responsibly as conditions change. That debt grows when teams rely on static evaluation and assume the first working route is the safe route. For identity and access programmes, the practical conclusion is that agent controls must account for ongoing decision quality, not just initial authorization.

Security and identity teams should read this as evidence that agent oversight must be continuous. The benchmark suggests there is no stable endpoint where evaluation becomes unnecessary, because the system can keep discovering better or worse ways to act. That is directly relevant to agent identity, delegated authority, and runtime monitoring. If an agent can iterate until it finds a better path in a game, it can also iterate within enterprise workflows in ways governance teams did not initially anticipate.

What this signals

Adaptive execution debt: agent programmes accumulate risk when teams validate only the first successful path and do not measure whether the system can keep adapting as conditions change. That gap becomes visible when the environment gets longer, more stateful, or more constrained, which is exactly where benchmark-style evaluation should continue to apply.

The governance lesson for security and identity teams is that agent assurance cannot stop at initial approval. If an agent is allowed to act across tools or sessions, the control problem shifts to whether it can remain disciplined under repetition, failure, and time pressure.

For practitioners

• Define separate tests for short-horizon and long-horizon agent behaviour Measure whether the system can sustain strategy over many steps, not just complete one prompt successfully. See the Agentic AI Identity Guide.
• Assess recovery after failed attempts Check whether the agent revises its plan after setbacks or simply repeats the same ineffective pattern.
• Evaluate route selection under practical budgets Set limits on attempts, time, and context so you can see whether the agent remains useful when resources are constrained.
• Track improvement across repeated runs Record whether performance actually gets better as the agent gains experience, rather than assuming learning from repetition.

Key takeaways

• Frontier LLM agents can approach human performance on simple tasks, but sustained strategy formation remains a harder test than one-shot completion.
• Long-horizon behaviour is the real governance issue, because repeated improvement and recovery matter more than a single successful run.
• Agent programmes need evaluation methods that keep exposing gaps after the first pass, especially where autonomy or delegated access is involved.

Related NHI Mgmt Group resources: Agentic AI Identity Guide · Top 10 Agent Identity Controls · Human vs Non-Human Identity

Key terms

• Long-horizon reasoning: Long-horizon reasoning is an AI system’s ability to plan and carry out many dependent steps towards a goal over time. It involves maintaining state, revising plans when conditions change, and avoiding error accumulation across long workflows, which is especially relevant when the system uses tools or affects real operations.
• Saturation-resistant benchmark: A saturation-resistant benchmark is a test that still separates stronger and weaker systems after models improve on the easy parts of the task. It is designed to reduce score inflation from memorisation or narrow optimisation, so the benchmark keeps measuring capability rather than being quickly maxed out.
• Strategy formation: Strategy formation is the process of choosing a plan of action from available options to achieve a goal. In autonomous systems, it means selecting sequences of steps, contingencies and tool use, rather than only reacting to the next prompt or instruction.
• Adaptive execution debt: Adaptive execution debt is the operational cost created when a system keeps changing its behaviour, workflows or integrations faster than those changes can be stabilised. It shows up as brittle automations, untested branches, and more effort spent fixing drift, which can make access controls and approvals harder to keep aligned.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It helps practitioners connect identity controls to the broader security programme they run every day.