ARE and Gaia2 show why agent evaluation needs time-aware environments

At a glance

What this is: ARE is a platform for building time-aware agent environments, and Gaia2 is a 1,120-scenario benchmark that tests adaptability, collaboration, and verification in a smartphone-like setting.

Why it matters: It matters because AI agents now need governance around timing, tool execution, and action verification, which affects how IAM, NHI, and AI security teams assess delegated access and runtime trust.

By the numbers:

• Gaia2 includes 1,120 scenarios across 10 universes in a Mobile environment with 101 tools.
• The benchmark measures Time tasks where GPT-5 scored 0 in real-time execution but about 34% when generation was treated as instantaneous.

👉 Read Arize’s paper reading on ARE and Gaia2 for time-aware agent evaluation

Context

AI agent evaluation is shifting from static prompts and isolated tool calls toward time-aware environments where events keep happening even while the agent is still deciding. That matters for identity governance because an agent that can read, write, notify, and collaborate is not just a model behaviour problem, it is an access and authority problem.

Meta AI’s ARE and Gaia2 are best understood as a response to evaluation saturation: older tests can confirm execution, but they often miss how agents behave under delay, ambiguity, and asynchronous change. For NHI and agentic AI governance, the core question is whether a system’s permissions, timing, and verification rules still hold once the environment starts moving independently of the prompt.

Key questions

Q: How should security teams govern AI agents that run long, multi-step workflows?

A: Security teams should require durable execution, full event history, and clear ownership for every multi-step agent workflow that touches sensitive data or privileged tools. If the agent can lose state on failure, the organisation cannot reliably audit what happened or prove which actions were completed versus replayed.

Q: Why do slower AI agents create security and operational risk?

A: Because delay can turn a correct decision into a failed one if the environment changes before the agent acts. In access-controlled or message-driven workflows, latency can cause missed alerts, stale approvals, or actions taken against outdated context. Security teams should measure response time alongside accuracy when agents can influence real systems.

Q: What do organisations get wrong about agent evaluation?

A: They often score the final response and assume that means the workflow was safe. That approach overlooks wrong-tool selection, invalid parameters, repeated retries, and bad handoffs. The better model is to score behaviour across the whole trace and to separate containment controls from diagnostic metrics.

Q: What is the difference between static agent benchmarks and time-aware environments?

A: Static benchmarks test a model or agent in a fixed sequence, while time-aware environments keep events moving during execution. That difference matters because agents must react to asynchronous messages, changing state, and deadlines. Time-aware testing is closer to production reality and is better at revealing governance failures in delegated access.

Technical breakdown

Time-driven agent environments and asynchronous state

ARE models an environment as a set of apps and tools where time continues to pass while the agent is thinking. That means the world can change through notifications, user messages, and scheduled events before the agent acts. This is materially different from static evaluation, because success depends on whether the agent can preserve context, react before a deadline, and handle state drift. For governance, the important part is that runtime authority is temporal, not just functional.

Practical implication: Treat agent permissions and response windows as runtime controls, not one-time configuration choices.

Why verifiable scenarios matter for agent evaluation

Gaia2 uses scenarios represented as directed acyclic graphs, which lets evaluators define branches, scheduled checks, and failure conditions ahead of time. That structure makes the benchmark more resistant to gaming because it validates the outcome of meaningful write actions, such as sending a message, rather than only checking whether an answer looks plausible. Verification is central here: if an agent can act in the world, the evaluation must confirm whether the intended state change actually occurred.

Practical implication: Design evaluation and approval workflows around observable outcomes, not only model output quality.

Latency, inverse scaling, and the cost of slow reasoning

The paper’s Time capability highlights a governance gap that teams often miss: higher reasoning quality does not guarantee useful performance when deadlines matter. In real-time scenarios, a slower model can fail even when its underlying answer is strong, because the window to act has already closed. This creates an inverse scaling effect in time-sensitive workflows. For identity and AI control owners, the lesson is that latency becomes part of the security and reliability envelope, not a pure infrastructure metric.

Practical implication: Set latency thresholds for agent actions that influence access, notifications, or approvals.

NHI Mgmt Group analysis

Time-aware agent governance is now an access-control problem, not just an eval problem. When agents operate in environments where events keep arriving, the issue is no longer whether they can complete a task, but whether they still have the right to act when the context changes. That creates a direct intersection with NHI governance, because agents behave like runtime identities with delegated authority. The practitioner conclusion is clear: authority must expire, not linger.

Verification drift is the new benchmark risk. Static evaluations can confirm isolated capability, but they often fail to capture whether an agent’s action remains valid after delays, branches, or partial state changes. Gaia2’s emphasis on write verification shows why outcome checks matter more than answer-style scoring for agent systems. Teams should align evaluation design with control validation, not just model performance.

Temporal responsiveness belongs in the security model. The paper shows that bigger or stronger models can perform worse once time becomes part of the task. That means risk teams need to treat timing as a security property when agents can send emails, notify users, or modify state. The field needs a named concept here: time-window governance, meaning access and action authority that is bounded by deadlines as well as scope. The practitioner conclusion is to bind agent rights to task duration and expected response time.

Agent collaboration expands the blast radius of delegation. ARE explicitly supports scenarios with multiple agents and user interactions, which mirrors how enterprise AI systems will increasingly chain tools and services. Once one agent depends on another, trust decisions propagate across identities, tools, and workflows. That complicates IAM and NHI oversight because the risky unit is no longer a single agent, but the delegation path between agents. The practical conclusion is to govern inter-agent trust as a first-class control surface.

Benchmark sophistication will pressure governance maturity. The move from simple tool-use tests to time-driven, verifiable environments signals where the category is heading. Evaluation is becoming closer to operational reality, which exposes weak assumptions about standing authority, control-plane trust, and delayed execution. That validates stronger identity and runtime governance models for agentic systems. Practitioners should assume future benchmarks will increasingly test controls, not just capability.

What this signals

Time-window governance is likely to become a standard control concept for agentic systems because action authority now has to be bounded by deadlines as well as scope. That shifts programme design toward expiry-based permissions, outcome verification, and latency monitoring for any workflow that can change state or trigger access decisions.

If an agent can act after the environment has already changed, the control failure is not model quality alone. It is the absence of synchronisation between identity, timing, and verification. Teams should expect future agent evaluations to expose more of these governance gaps, especially where delegated access and write actions intersect.

For practitioners

• Define time-bounded agent authority Assign each agent a task window, scope, and expiry condition so permissions do not outlive the workflow they support. This is especially important when an agent can write messages, trigger notifications, or modify shared state.
• Verify write actions, not just model answers Build evaluation and approval steps around the effects of an action, such as whether the right email was sent or the correct state changed, rather than whether the generated text looked plausible.
• Measure latency against business deadlines Set explicit maximum response times for agent actions that influence access, escalation, or customer communication, then test them under realistic load and asynchronous event timing.
• Map inter-agent delegation paths Identify where one agent relies on another agent, service, or tool, and apply least privilege to each link in the chain so one failure does not expand into broad delegated access.

Key takeaways

• ARE and Gaia2 show that agent governance now depends on timing, verification, and delegated authority, not only on model accuracy.
• The benchmark data shows that stronger reasoning can still fail in time-sensitive workflows, which makes latency a security and reliability variable.
• Practitioners should bind agent permissions to task windows, verify write outcomes, and review inter-agent delegation paths before scaling deployment.

Key terms

• Time-Window Governance: A control approach that binds an agent’s authority to a defined task duration as well as a permission scope. It recognises that for software agents, timing affects trust just as much as access does, especially when delayed action can invalidate an otherwise correct decision.
• Verifiable Scenario: A test environment in which the expected outcome can be checked after the agent acts, not merely inferred from its response. This matters for agent governance because real-world utility depends on whether the intended state change occurred safely and within policy.
• Asynchronous Agent Environment: An environment where messages, events, and state changes continue while the agent is still processing. It exposes a common weakness in static evaluation, namely that a model can appear competent when the world is frozen but fail once time and external change are introduced.
• Delegated Agent Authority: The permission granted to an AI agent to act on behalf of a human user or another agent, inheriting some or all of their access rights. Delegated authority must be explicitly scoped, time-limited, and auditable.

What's in the full report

Arize's full paper reading covers the evaluation design and benchmark details this post intentionally leaves for the source:

• The scenario structure behind ARE’s directed acyclic graph evaluation model and how scheduled checks work in practice.
• The full Gaia2 Mobile environment design, including the kinds of apps, tools, and asynchronous events used in testing.
• The comparative results across capability, cost, latency, and time-sensitive performance that are only summarised here.
• The original discussion with Grégoire Mialon on why older tool-use benchmarks became saturated.

👉 Arize’s full post covers the benchmark structure, scenario design, and the Time capability results in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners apply identity controls to agentic and non-human runtime access.