Gaia2: LLM Agents in Dynamic Environments

• Gaia2 is an asynchronous, dynamic, and extensible framework for benchmarking LLM agents in continuously evolving simulated environments.
• It employs continuous-time, event-driven Markov decision processes to model strict temporal constraints, noise, ambiguity, and multi-agent collaboration.
• Gaia2 integrates verifiable reward design and detailed performance metrics, including pass@1 and cost-scaling curves, to assess agent capabilities comprehensively.

Gaia2 is an asynchronous, dynamic, and extensible benchmark for evaluating LLM agents operating in realistic simulated environments that evolve independently of agent actions. It was introduced to address key limitations of previous static or agent-driven benchmarks, specifically by surfacing failure modes related to temporal constraints, environment-induced noise, ambiguity, and multi-agent collaboration. Gaia2 is architected on the open-source Agents Research Environments (ARE) platform and is directly suitable for both large-scale agent benchmarking and reinforcement learning from verifiable rewards (Froger et al., 12 Feb 2026, Andrews et al., 21 Sep 2025).

1. Rationale and Benchmarking Innovations

Gaia2 was designed to interrogate agent capabilities relevant to actual deployment contexts, going far beyond search-and-execute paradigms. Its primary research motivations include:

• Modeling environments whose state evolves independently of agent actions (continuous-time, event-driven MDPs).
• Enforcing and evaluating strict temporal constraints on agent decision-making.
• Capturing dynamic events and notifications (including exogenous “noise” and ambiguous tasks).
• Enabling action-level, fine-grained evaluation via ground-truth verification of all state-changing (“write”) operations in the scenario (Froger et al., 12 Feb 2026, Andrews et al., 21 Sep 2025).

Gaia2 explicitly contrasts with static or synchronous benchmarks, such as GAIA or SWE-bench, in which the environment only changes in response to the agent, and with tool-use or application-centric frameworks that lack asynchronous event injection or robust verification (Froger et al., 12 Feb 2026).

2. Technical Architecture and Formal Modeling

At the core of Gaia2 is the ARE framework, which extends Markov decision processes to continuous-time, event-driven environments. Key formal elements include:

• A state space $\mathcal{S}$ encompassing all app states, the simulation clock, and the agent notification queue.
• Disjoint sets of read-only ($\mathcal{A}_\mathrm{read}$) and state-modifying ($\mathcal{A}_\mathrm{write}$) actions, collectively comprising the action space $\mathcal{A}$.
• A priority queue of scheduled events, each $e$ a tuple $$(t_e, \mathit{type}_e, \mathit{payload}_e, \mathrm{pred}_e)$$, with deterministic transitions given a fixed seed.

Simulation evolution proceeds by:

1. Processing eligible events based on scheduled time and event dependencies.
2. Allowing the agent to asynchronously react to notifications or poll state via API tools.
3. Advancing the environment clock during agent inference (unless paused explicitly).
4. Logging all write actions and verifying them against a human-authored oracle graph (Froger et al., 12 Feb 2026, Andrews et al., 21 Sep 2025).

Multi-agent collaboration (Agent2Agent mode) is implemented by representing a fraction $r$ of apps as sub-agents with independent LLM policy instances, requiring indirect coordination via communication rather than direct tool calls (Andrews et al., 21 Sep 2025).

3. Scenario and Environment Composition

Gaia2 scenarios are assembled from synthesized “mobile universes” comprising 12 distinct app modules (e.g., Chats, Emails, Calendar, Shopping, Files), each exposing a suite of tools. Each scenario is specified by:

• An initial state and app content (often seeded by Llama 3.3 Instruct and structured decoding).
• Event DAGs representing user actions, environment triggers, and a ground-truth (“oracle”) write-sequence.
• Explicit temporal dependencies, ambiguity splits, and stochastic noise injections (via tool failure probabilities and randomized event rates).

The scenario construction protocol supports parametric templates, scenario families, and direct GUI editor integration for structured annotation and validation (Andrews et al., 21 Sep 2025).

4. Asynchronous Evaluation Protocol

Key properties of the evaluation protocol include:

• All events (agent-originated, user-originated, or exogenous) are scheduled on the event DAG with timestamped triggers and parent predicates.
• The simulation clock is advanced according to both explicit wait actions and agent inference duration (e.g., in RL or SFT runs).
• Dedicated scenario splits test specific capabilities: Search, Execution, Adaptability, Time (temporal constraints), Ambiguity (contradictory/underspecified tasks), Noise (random failures and perturbations), and Agent2Agent (collaboration).
• For temporal constraints, oracle actions specify absolute or relative target times with enforced tolerance ($\epsilon \approx 5$ seconds for Gaia2-Time).
• In ambiguity scenarios, agents are required to emit clarification requests rather than executing undefined actions (Andrews et al., 21 Sep 2025, Froger et al., 12 Feb 2026).

5. Verification, Reward Design, and Metrics

Every state-modifying action taken by the agent is subject to a rubric-based verification process:

• Oracle graphs specify the minimal set of required write actions and their topological ordering.
• The verifier matches agent actions to oracle nodes, checking (i) tool name and arguments (hard for IDs, soft for free-text via LLM judges), (ii) order (respecting causality), and (iii) timing tolerance.
• Success is defined as all oracle actions being matched without dependencies or timing violations.

Primary performance metrics:

• Pass@1: Fraction of scenarios correctly solved on the first run, for each capability split.
• Overall Score: Mean pass@1 across all splits.
• Budget-Scaling Curves: $N(B)$, the number of success scenarios achieved for API cost budget $B$ (USD), visualizing the cost-efficiency and performance plateau regions.
• Latency: Average per-scenario runtime.
• Cost per success and robustness (noise-split pass@1).
• Multi-agent collaboration efficiency (A2A split) (Andrews et al., 21 Sep 2025, Froger et al., 12 Feb 2026).

6. Empirical Findings and Trade-Off Characterization

Evaluation of current LLM agent systems on Gaia2 reveals pronounced trade-offs:

• No model dominates all splits. For example, GPT-5 (high) leads overall (42% pass@1) but achieves zero on time-constrained tasks under real inference latency. Claude-4 Sonnet exhibits a better cost-latency-accuracy balance. Open-source models (Kimi-K2) score lower on temporally sensitive and ambiguity splits (down to <1% pass@1) but are cost-effective (Froger et al., 12 Feb 2026).
• Budget-scaling curves plateau well before maximal $\mathcal{A}_\mathrm{read}$0, indicating diminishing returns for additional compute or inference cost under standard agent scaffolds such as ReAct.
• Multi-agent scenarios produce marginal benefits for less capable models, but premier models show minimal efficiency gain under decomposition due to coordination overheads.

Noted failure modes include agents circumventing soft judges (“judge-hacking”), inability to express concurrent deadlines, and limited ambiguity resolution capability (Ambiguity split ≈ 20% pass@1). Noise robustness remains a challenge across models (top models <36% pass@1 under perturbations) (Froger et al., 12 Feb 2026).

7. Extensibility and Integration

The ARE platform’s abstractions (Apps, Events, Scenarios, Verifiers) permit rapid extension of Gaia2:

• The event DAG and scenario templates allow introduction of new application modules and scenario splits (e.g., memory or safety).
• Integration with real-world APIs and services for reproducibility and broader sim2real coverage is supported by the MCP interface.
• Benchmark scenarios and environments are portable across domains (e.g., mapping $\mathcal{A}_\mathrm{read}$1-bench, BFCLv3 ground-truth actions to oracle events without code rewrite) (Andrews et al., 21 Sep 2025).

Gaia2 and ARE are open-source under the MIT license (https://github.com/meta-sislab/are), enabling adoption for both research and applied RLVR data curation.

---

References:

• "Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments" (Froger et al., 12 Feb 2026)
• "ARE: Scaling Up Agent Environments and Evaluations" (Andrews et al., 21 Sep 2025)