OdysseyArena-Lite Benchmark

• OdysseyArena-Lite is a standardized benchmark suite built on the OdysseyArena framework, featuring 120 deterministic tasks designed to evaluate long-horizon inductive reasoning in LLM agents.
• It integrates four distinct environment families—discrete, continuous, periodic, and relational—to test agents on discovering hidden transition laws rather than relying on explicit rules.
• The benchmark emphasizes reproducibility and computational manageability through fixed configurations, pre-generated trajectories, and standardized evaluation protocols.

Searching arXiv for the benchmark paper and related context. OdysseyArena-Lite is the primary standardized benchmark suite built on top of the OdysseyArena framework for evaluating LLM agents under long-horizon, active, and inductive interactions. It consists of a deterministic, finite subset of 120 tasks, with 30 tasks drawn from each of four environment families, and is designed to measure how effectively an agent discovers latent transition laws from interaction rather than executing tasks from explicitly provided rules. Its core formal object is a hidden transition function, $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t)$, whose structure is fixed per task but not revealed to the agent; the benchmark is therefore centered on inductive efficiency and long-horizon discovery rather than conventional deductive compliance (Xu et al., 5 Feb 2026).

1. Position within the OdysseyArena framework

OdysseyArena-Lite occupies the central evaluation role within the broader OdysseyArena framework. The full framework defines four environment families, each instantiating one structural primitive of world dynamics: discrete symbolic rules, continuous stochastic dynamics, periodic temporal patterns, and relational graph structures. OdysseyArena-Lite is the fixed suite of 120 tasks drawn from these environments, with moderate horizons of 120–200 steps and frozen parameters chosen to keep evaluation tractable, repeatable, and comparable across agents. OdysseyArena-Challenge is a distinct, substantially harder setting with 10 tasks per environment and horizons often exceeding 1,000 steps; it is used to stress-test long-horizon stability rather than routine benchmarking. The paper explicitly describes Lite as “our primary evaluation setting and the basis for all main experiments in this paper” (Xu et al., 5 Feb 2026).

The motivation for the Lite variant is threefold. First, it provides standardization through a fixed set of instances with the same latent rules and pre-generated trajectories. Second, it improves accessibility and computational cost by using horizons that are long enough to be non-trivial but short enough for large-scale evaluation of many models. Third, it enforces determinism and reproducibility: stochastic elements such as market factor trajectories and renewable efficiency series are pre-generated and stored in task metadata, so agents do not encounter runtime randomness.

A recurrent misunderstanding is to treat the “Lite” designation as implying a simplified or shallow benchmark. The task horizons contradict that reading. Turn On Lights allows up to 200 steps, while AI Trading, Energy Dispatch, and Repo System each run for 120 steps. Lite is therefore better understood as a standardized and computationally manageable regime rather than an easy one.

2. Formal structure and environment families

All OdysseyArena environments share the same transition formalism,

$(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$

with the crucial condition that the structure of $\mathcal{T}$ is unknown to the agent and must be induced from experience. From the agent’s perspective, the setting is a deterministic partially observable environment: the latent state and transition law are hidden, while the agent observes textual projections, state summaries, and feedback (Xu et al., 5 Feb 2026).

Environment	Structural primitive	Lite horizon
Turn On Lights	Discrete symbolic rules	200
AI Trading	Continuous stochastic dynamics	120
Energy Dispatch	Periodic temporal patterns	120
Repo System	Relational graph structures	120

Turn On Lights instantiates hidden Boolean structure over a binary state space. The state is $s \in \{0,1\}^N$, the start state is $s_0=\mathbf{0}$, and the goal is $s=\mathbf{1}$. Each light $L_i$ has an activation condition

$$\phi_i = f(L_{j_1}, L_{j_2}, \ldots, L_{j_k}), \quad j_1,\dots,j_k<i,$$

with a strict partial order $L_j \prec L_i$. The minimal element satisfies $\phi_0 \equiv \text{True}$, but its identity is hidden by a random permutation $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$0. When the agent toggles light $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$1, the update rule is

$(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$2

The agent receives only success or failure feedback and the full light states; the logical network itself remains latent.

AI Trading instantiates continuous stochastic dynamics through a latent factor model. The state is continuous, and returns follow

$(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$3

where $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$4 are asset returns, $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$5 are factor changes, and $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$6 is a fixed latent matrix per task. Prices evolve as $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$7. The action is a vector of buy and sell quantities executed as “sell then buy,” and reward is based on portfolio value. In Lite, the entire sequence $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$8 is pre-generated and fixed; the agent must infer the structure of $(s_{t+1}, r_t) = \mathcal{T}(s_t, a_t),$9 and exploit observations and news hints.

Energy Dispatch instantiates periodic temporal structure with a hidden period. The action is $\mathcal{T}$0 for thermal, wind, solar, and battery decisions, and realized power is approximately

$\mathcal{T}$1

For renewables,

$\mathcal{T}$2

with hidden period $\mathcal{T}$3, base patterns formed by piecewise linear segments with rare spikes, cycle offsets, and micro-noise $\mathcal{T}$4. Wind is clipped to $\mathcal{T}$5 and solar to $\mathcal{T}$6; thermal efficiency is approximately $\mathcal{T}$7, and battery efficiency is $\mathcal{T}$8. The environment imposes daily demand and budget constraints and a global carbon constraint,

$\mathcal{T}$9

A scalar stability metric $s \in \{0,1\}^N$0 is also tracked, and the episode terminates early if demand or budget is violated for three consecutive days.

Repo System instantiates a latent graph $s \in \{0,1\}^N$1 over package-version nodes. The state is the installed set $s \in \{0,1\}^N$2, and actions are shell-like commands such as pip install, uninstall, pip list, python <script>, and repo tree. Resolution rules include Ensure, Force-high/low, and Pin, producing non-local side effects: installing one package may upgrade, downgrade, or uninstall others. In Lite, each repo task is generated solution-first: a valid target environment is constructed and then back-propagated into a hidden dependency graph so that solvability is guaranteed.

3. Task construction and interaction protocol

OdysseyArena-Lite contains 30 tasks for each of the four environments. All tasks are frozen, with fixed configurations and, where relevant, fixed trajectories. The benchmark controls diversity and difficulty by varying structural parameters while preserving tractability. Turn On Lights varies the number of lights, operator mix over $s \in \{0,1\}^N$3, and dependency density; each instance is checked to ensure that at least one path to $s \in \{0,1\}^N$4 exists and that the minimum solution length exceeds a threshold. AI Trading varies the number of stocks $s \in \{0,1\}^N$5, the number of latent factors $s \in \{0,1\}^N$6, the sparsity of $s \in \{0,1\}^N$7, and the signal-to-noise ratio. Energy Dispatch varies hidden periods for wind and solar and the carbon and stability thresholds $s \in \{0,1\}^N$8, while choosing budgets and capacities so that some feasible policy exists. Repo System varies package count, dependency depth, and constraint types such as exact version equality and major-version matching (Xu et al., 5 Feb 2026).

The interaction protocol is uniform across environments. A pre-configured environment instance is loaded, including latent rules, initial state, and pre-computed trajectories where applicable. The agent receives a system prompt that describes the goal, the allowed action format, and general environment properties. At each step, it receives the current observation and history, outputs exactly one action in a constrained textual format wrapped in <action>...</action>, and the environment parses, validates, and executes that action.

History management is environment-specific. AI Trading retains the last 50 interactions; Energy Dispatch retains the last 40; Turn On Lights and Repo System retain the entire history. The benchmark does not preserve past chain-of-thought in context. Only actions and environment feedback are kept. This design constrains memory growth while preserving the observable interaction trace that the paper treats as operationally sufficient for evaluation.

The action spaces are deliberately compact. Turn On Lights uses commands of the form Toggle <light_index>. AI Trading expects a JSON action with "Buy" and "Sell" objects. Energy Dispatch expects a JSON object with "Thermal", "Wind", "Solar", and "Battery". Repo System uses shell-like commands. This compact textual interface is one of the reasons the benchmark remains lightweight to deploy.

4. Scoring, metrics, and empirical findings

Success conditions are environment-specific. Turn On Lights is solved when all lights are on within 200 steps. AI Trading has no binary failure state; every episode ends after 120 days, and performance is measured by profit rate. Energy Dispatch is successful if the agent avoids early termination for all 120 days and the final metrics satisfy $s \in \{0,1\}^N$9 and $s_0=\mathbf{0}$0. Repo System is successful only if python run.py executes successfully; partial repairs do not count (Xu et al., 5 Feb 2026).

The main reported metrics are Avg@4 and Pass@4. For Turn On Lights, Energy Dispatch, and Repo System, Avg@4 is the average success rate across tasks when each task is run four times, and Pass@4 is the fraction of tasks solved in at least one of those runs. For AI Trading, Avg@4 is the average profit rate across tasks, each averaged over four runs, and Pass@4 uses the highest profit achieved per task over the four runs. The paper does not define a single global score across environments.

The empirical profile on Lite shows a marked gap between human performance and LLM agents. Human Avg@4 in Turn On Lights is 81.67%, with Pass@4 of 100%; in Repo System, human Avg@4 is 77.50% with Pass@4 of 100%; and in AI Trading, human Avg@4 profit is +92.55%. The top reported model, Gemini 3 Pro Preview, reaches 44.17% Avg@4 in Turn On Lights, 65.83% Avg@4 in Repo System, and +67.71% profit in AI Trading. Energy Dispatch is difficult for nearly all models: Gemini 3 Pro Preview records Avg@4 30.00 and Pass@4 36.67, while human performance is reported as 25.00 / 60.00.

The paper’s most consequential experimental control compares performance without explicit rules to performance with the hidden rules provided in the prompt. In Turn On Lights, Gemini 3 Pro Preview moves from 44.17% without rules to 100% with rules, and GPT-5 moves from 28.33% to 100%. Similar jumps are reported across AI Trading and Repo System. The benchmark therefore isolates an inductive deficit rather than a deductive one: when latent transition structure is made explicit, contemporary models are much stronger.

Several diagnostic measures reinforce that interpretation. The benchmark introduces Loop Ratio for Turn On Lights and Repo System, defined as the proportion of actions where the agent repeats the same consecutive $s_0=\mathbf{0}$1 pair without progress. High Loop Ratio is treated as evidence of inductive stagnation. The paper also analyzes cumulative success as a function of step index and reports performance saturation: after an initial phase, additional steps yield little gain because the agent does not successfully build a correct world model. Increasing reasoning budget helps but does not remove the bottleneck; for gpt-oss-120b, Turn On Lights Avg@4 improves from 7.50 to 16.67 to 27.50 as the reasoning budget moves from low to medium to high.

5. Benchmark niche, conceptual significance, and common confusions

OdysseyArena-Lite is positioned against prior agent benchmarks by emphasizing a distinctive combination of properties: it is interactive, long-horizon, inductive, and computationally manageable. The paper contrasts it with benchmarks that typically focus on short horizons, frequently below 50 steps and only occasionally near 100, and that emphasize deductive compliance by providing full instructions, API documentation, or explicitly specified tools. It also distinguishes Lite from settings that rely on heavy deployment substrates such as Docker, emulators, or browser stacks (Xu et al., 5 Feb 2026).

This design choice gives the benchmark a specific methodological role. It does not primarily test whether a model can follow instructions or retrieve rules from documentation; it tests whether the model can turn feedback into a latent world model. In Turn On Lights, this means symbolic rule induction through trial and error. In AI Trading, it means inference of factor-return relationships under noise. In Energy Dispatch, it means period discovery under safety and budget constraints. In Repo System, it means graph-structured diagnosis and globally consistent repair.

A common conceptual confusion concerns the word “Lite.” In OdysseyArena-Lite, “Lite” refers to a standardized, frozen, API-lightweight benchmark setting with moderate horizons. It does not denote a small-parameter model family. By contrast, in ALLaVA, the term “lite” denotes a 2.7B-parameter vision-LLM built for lower-cost training and deployment, with a frozen CLIP ViT-L/14@336 vision encoder, a 6.3M projector, and a small LM backbone (Chen et al., 2024). The two uses of the term therefore address different objects: one is a benchmark regime for long-horizon inductive agents, and the other is a resource-efficient multimodal model class.

A second misconception is that OdysseyArena-Lite is merely a weaker version of OdysseyArena-Challenge. The data suggest a more precise interpretation: Lite defines the standard operating point for reproducible model comparison, while Challenge probes extreme long-horizon stability. Lite is thus not incidental to the framework; it is the framework’s canonical benchmark suite.

6. Reproducibility, limitations, and future directions

OdysseyArena-Lite is designed for straightforward reproducibility. The code and data are released at https://github.com/xufangzhi/Odyssey-Arena. Evaluation is deterministic because the random elements of each task—such as factor paths, efficiency curves, and budgets—are precomputed and stored. Open-source models are run with vLLM and PagedAttention, while proprietary models are accessed through standard APIs. No heavy simulators, browsers, or GUI emulators are required, which materially lowers the operational overhead of benchmark adoption (Xu et al., 5 Feb 2026).

The benchmark nevertheless has explicit limitations. Energy Dispatch is so difficult that almost all LLMs fail on it in Lite, suggesting either that the tasks remain substantially beyond current model capabilities or that additional scaffolding, memory, or specialized reasoning modules may be needed. The benchmark also does not provide a single scalar inductive reasoning index; evaluation remains environment-specific through success rates and profit measures. Domain coverage is limited to four environment families and excludes robotics, web interaction, and multimodal settings. In addition, the decision to discard past reasoning traces may under-use models that benefit from longer contexts or richer persistent memory strategies.

The Lite–Challenge gap is itself diagnostically important. Challenge tasks often exceed 1,000 interaction steps, and the paper reports sharp degradations when moving to this regime; for Repo System, models that achieve 65.83% on Lite can drop to 10% in Challenge. This suggests that current systems face not only a rule-induction bottleneck but also a stability bottleneck as interaction horizons expand.

The future directions proposed in the paper center on world-modeling architectures that can “distill latent transition laws from raw experience,” broader task families, and more fine-grained diagnostics such as period-detection accuracy in Energy Dispatch and structural inference quality in Repo System. A plausible implication is that OdysseyArena-Lite will remain useful not merely as a leaderboard substrate but as a controlled diagnostic instrument for differentiating deductive competence from genuine inductive adaptation.