OdysseyArena-Challenge: Long-Horizon Stress Test

• OdysseyArena-Challenge is a benchmark for evaluating long-horizon, inductive reasoning by stress-testing agents over 200+ interaction steps with hidden transition dynamics.
• It assesses agents' abilities to infer latent rules, maintain strategic coherence, and avoid repetitive action loops under prolonged and dynamic conditions.
• The challenge regime exposes critical failure modes such as planning collapse and inefficiencies in sustained decision-making that are not evident in shorter evaluation tasks.

OdysseyArena-Challenge is the stress-test extension of OdysseyArena, a benchmark family for evaluating LLM agents under long-horizon, active, and inductive interactions. It reorients agent evaluation away from a deductive setting, in which rules or instructions are given explicitly, toward an inductive setting in which the agent must infer latent transition dynamics from experience, preserve strategic coherence over very long trajectories, and avoid planning collapse or action loops. Within this framework, OdysseyArena-Lite provides a standardized benchmark of 120 curated tasks, whereas OdysseyArena-Challenge uses the same environment family in a much longer-horizon regime, described as “> 200 steps” and, in the task-curation discussion, as often exceeding 1,000 interaction steps (Xu et al., 5 Feb 2026).

1. Conceptual basis and motivation

OdysseyArena was introduced to address three evaluation gaps: long-horizon reasoning is under-tested, active exploration is under-tested, and induction from interaction is under-tested. The benchmark’s central claim is that autonomous agents must infer hidden world dynamics rather than merely follow explicit instructions or apply known rules. OdysseyArena-Challenge was added as the mechanism for probing these requirements under extreme horizons, with emphasis on stability, persistence, and inductive resilience rather than short-run task completion alone (Xu et al., 5 Feb 2026).

The benchmark therefore treats latent transition discovery as the core difficulty. Its target failure modes are not only incorrect actions, but also compounding planning errors, breakdowns in long-term credit assignment, and failure to maintain a coherent internal model over hundreds or thousands of decisions. A common misconception is that the challenge setting is simply a scaled-up version of OdysseyArena-Lite. The benchmark instead distinguishes the two by evaluation regime: Lite is designed to be reproducible, tractable, and fast enough for large-scale benchmarking, whereas Challenge is computationally more expensive and exists specifically to expose failure modes that remain hidden under shorter budgets (Xu et al., 5 Feb 2026).

2. Environment family and formal structure

OdysseyArena formalizes its environments as hidden transition systems,

$(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$

where $s_t \in \mathcal{S}$ is the latent state, $a_t \in \mathcal{A}$ is the action, and $\mathcal{T}$ is the hidden environment law. The agent does not receive $\mathcal{T}$ directly and must infer it through interaction. A key design choice is that the latent transition law is fixed within an episode but varies across episodes or tasks. Task curation also makes the environment deterministic given the task metadata and the agent’s action sequence: time-varying factors are pre-generated and stored in the task configuration, which removes runtime randomness while preserving hidden structure (Xu et al., 5 Feb 2026).

OdysseyArena is organized around four structural primitives that translate abstract latent dynamics into concrete interactive environments.

Primitive	Environment	Hidden structure
Discrete symbolic rules	Turn On Lights	Boolean conditions over lights
Continuous stochastic dynamics	AI Trading	Latent matrix $\mathbf{W}$ and unobserved factors
Periodic temporal patterns	Energy Dispatch	Hidden periodic efficiency
Relational graph structures	Repo System	Dependency graph $G=(V,E)$

These primitives are not merely thematic labels; they define the ontology of the benchmark. In Turn On Lights, toggle effects depend on hidden Boolean conditions, formalized as

$$\phi_i = f(L_{j_1}, L_{j_2}, \ldots, L_{j_k}), \quad j_1,\ldots,j_k < i.$$

In AI Trading, observed prices and news are noisy proxies for latent factors, with a transition of the form

$s_{t+1} = \mathbf{W} z_t + \epsilon.$

In Energy Dispatch, hidden periodicity is central, expressed as

$\mathcal{T}(s,a,t) \approx \mathcal{T}(s,a,t+P),$

and in Repo System the hidden dynamics are governed by a dependency graph over package versions and compatibility constraints (Xu et al., 5 Feb 2026).

A second misconception is that OdysseyArena is primarily a partial-observability benchmark in the classical hidden-state sense. The defining difficulty is instead hidden transition structure: even when surface state is observable, the agent must still induce the governing laws of change from interaction traces (Xu et al., 5 Feb 2026).

3. Lite and Challenge evaluation regimes

OdysseyArena-Lite is the standardized benchmark. It contains 120 curated tasks, with 30 tasks per environment, and serves as the primary evaluation setting. OdysseyArena-Challenge is built from the same environment family but uses a much longer interaction horizon, with 10 tasks per environment in the challenge setting and 1,000+ steps per task in the challenge description. The distinction is therefore not the type of primitive but the persistence requirement imposed on the agent’s reasoning (Xu et al., 5 Feb 2026).

Evaluation is environment-specific in Lite. Turn On Lights uses success rate; AI Trading uses cumulative return or profit rate; Energy Dispatch uses success under a multi-objective criterion; and Repo System uses binary success of making the full project run. Results are reported with Avg.@4 and Pass@4 across four independent runs per task. Avg.@4 is the average performance over four trials, while Pass@4 uses the best of the four attempts; for AI Trading this is based on the highest profit, and for the other environments it is based on success (Xu et al., 5 Feb 2026).

The challenge regime is designed to stress-test “stability,” a term the benchmark uses in two senses. In Energy Dispatch, it denotes an environment-specific success component, where the final average stability must satisfy

$s_t \in \mathcal{S}$0

alongside no early termination over horizon $s_t \in \mathcal{S}$1 and a carbon-intensity constraint

$s_t \in \mathcal{S}$2

At the benchmark level, stability refers to agentic stability: maintaining coherent strategy and avoiding drift, loops, or planning collapse over extreme horizons (Xu et al., 5 Feb 2026).

The benchmark also treats inductive efficiency as a practical proxy for how effectively a model extracts latent rules from interaction. In the appendix discussion, step usage is an explicit proxy: for Turn On Lights and Repo System, fewer steps indicate stronger inductive reasoning, whereas for Energy Dispatch, higher survivable step usage indicates better inductive reasoning. Loop Ratio is defined for Turn On Lights and Repo System as the normalized proportion of actions spent on immediately repeated, non-progress-making $s_t \in \mathcal{S}$3 pairs; lower Loop Ratio indicates better inductive reasoning (Xu et al., 5 Feb 2026).

4. Empirical findings and characteristic failure modes

OdysseyArena reports experiments on 15+ models, including proprietary models such as Gemini 3 Pro Preview, GPT-5, Gemini 2.5 Pro, and Grok 4 Fast, and open-source models such as DeepSeek-V3.2, gpt-oss-120b, Qwen3 variants, Llama 3 variants, and GLM-4 variants. Frontier models perform best, but even the strongest model remains well below human performance across the benchmark (Xu et al., 5 Feb 2026).

The benchmark’s headline empirical claim is that the major bottleneck is inductive discovery rather than rule-following. When rules are given explicitly, top models perform extremely well; when rules are hidden, performance drops sharply. This gap is especially pronounced in Turn On Lights and is used as direct evidence that current LLMs are much stronger at deductive compliance than at autonomous induction of latent structure (Xu et al., 5 Feb 2026).

Energy Dispatch is presented as the clearest failure case. Most models fail all tasks there, including many strong models, and the reported interpretation is that current models cannot reliably infer long-range periodic structure from noisy observations. The benchmark emphasizes that the problem is not merely scale; it is an inability to synthesize periodic patterns over observation windows of roughly 20 steps (Xu et al., 5 Feb 2026).

Long-horizon performance also saturates. As interaction budget increases, performance often rises early and then plateaus, indicating that more steps do not rescue an agent that fails to build a coherent world model. Action loops are another recurrent failure mode: models repeatedly execute ineffective actions, receive negative feedback, and nevertheless fail to update their internal hypotheses. The appendix challenge comparison on Repo System makes this degradation explicit: Gemini 3 Pro Preview drops from Lite 65.83 to Challenge 10.00; Qwen3-235B-A22B-Instruct drops from Lite 15.83 to Challenge 0.00; and Qwen3-30B-A3B-Instruct drops from Lite 26.67 to Challenge 0.00 (Xu et al., 5 Feb 2026).

These results support the benchmark’s central role as a stress test. OdysseyArena-Challenge is not primarily a leaderboard extension; it is an instrument for revealing whether an agent can preserve an induced world model under delayed consequences and extreme interaction length.

5. Relation to adjacent long-horizon benchmark ecosystems

Related benchmark families illuminate how OdysseyArena-Challenge fits into a broader research movement toward persistent, interactive evaluation. AgentOdyssey evaluates test-time continual learning agents in procedurally generated, open-ended, non-resettable text games modeled as a POMDP, with diagnostics for world knowledge acquisition, episodic memory, object and action exploration, action diversity, long-horizon planning, and model cost. Its experiments report that even the top agent remains far below human performance, and that short-term memory benefits multiple agent paradigms; in one harder game, the best long-context agent shows a 34.8% increase in World Knowledge QA accuracy after gameplay (Zhang et al., 29 May 2026).

Odysseys extends the same long-horizon concern to live-web agents. It contains 200 tasks derived from real browsing sessions and replaces binary trajectory judgment with rubric-based evaluation, using an average of 6.1 graded rubrics per task. The strongest reported model, Claude Opus 4.6, reaches 44.5% perfect-task success, while GPT-5.4 attains the highest reported Trajectory Efficiency at 1.15%, defined as rubric score per step. The benchmark’s methodological conclusion is that binary pass/fail is inadequate for long-horizon workflows because partial progress is otherwise discarded (Jang et al., 27 Apr 2026).

Arena 3.0 and Arena 4.0 show a parallel development in embodied navigation. Arena 3.0 is a comprehensive social-navigation development, simulation, and benchmarking platform spanning Flatland, Gazebo, and Unity, with benchmark mode, standardized configurations, and social metrics such as time in private zone, time facing pedestrians, and time seen by pedestrians (Kästner et al., 2024). Arena 4.0 extends the Arena line with a generative-model-based world and scenario generation approach, a semantic 3D asset database, and a full migration to ROS 2 Humble with Nav2, aiming at unified benchmarking in human-centric navigation scenarios (Shcherbyna1 et al., 2024).

A further parallel appears in long-horizon infrastructure planning. “The Battle of the Water Futures” defines a staged-design benchmark for designing and operating water distribution systems over at least 25 years under deep uncertainty, using EPANET in pressure-driven analysis mode and balancing reliability, cost, emissions, fairness, and financial sustainability. Its plan-execute-observe cycle resembles a different, domain-specific form of long-horizon evaluation under latent and partially unobservable structure (Zanutto et al., 28 Nov 2025).

Taken together, these systems suggest a broader transition in benchmark design from short, deductive, single-episode tasks toward settings with persistent interaction, delayed consequences, and more diagnostic evaluation.

6. Significance, interpretation, and common misunderstandings

The significance of OdysseyArena-Challenge lies in its redefinition of what counts as competent agent behavior. It does not ask only whether a model can act, call tools, or follow instructions. It asks whether the model can actively explore, infer hidden rules from interaction, sustain an internal world model over hundreds or thousands of steps, and avoid local optima and repetitive collapse (Xu et al., 5 Feb 2026).

One common misunderstanding is to treat long horizon as a purely scaling problem. The benchmark’s reported findings do not support that view. Frontier models improve on Lite and still degrade sharply on Challenge; more steps often produce early gains followed by saturation; and rule disclosure removes much of the apparent difficulty. This pattern indicates that the unresolved issue is not merely budget, but persistent inductive world-modeling (Xu et al., 5 Feb 2026).

A second misunderstanding is that success on short-horizon or deductive benchmarks transfers cleanly to this setting. Adjacent work on web agents, text-game agents, and social-navigation platforms points in the opposite direction: long-horizon evaluation increasingly requires finer-grained diagnostics, richer environment generation, and explicit measurement of efficiency, memory, or social behavior rather than only endpoint success (Jang et al., 27 Apr 2026, Zhang et al., 29 May 2026, Shcherbyna1 et al., 2024, Kästner et al., 2024).

In that sense, OdysseyArena-Challenge occupies a specific position in contemporary benchmarking. It is a controlled, task-curated stress test for inductive interaction, rather than an unrestricted open-world benchmark; yet its core contribution is precisely to isolate a capability that short-horizon evaluation tends to obscure. Its empirical results identify a persistent bottleneck for current LLM agents: strong deductive competence can coexist with weak inductive discovery, unstable long-horizon planning, and failure to preserve coherent strategy over extended interaction (Xu et al., 5 Feb 2026).