OdysseyBench: Unified Multi-Domain Benchmark

• OdysseyBench is a unified evaluation platform providing comprehensive benchmarks for advanced reasoning, numerical analysis, language understanding, code synthesis, and robotic simulation.
• It emphasizes contamination-resistant data, context scaling, and interactive workflows to ensure fair, reproducible assessments across diverse domains such as mathematics, medicine, and physics.
• The platform integrates automated and manual diagnostic tools, including error visualization and curriculum training, to identify model limitations and enhance interdisciplinary performance.

OdysseyBench is a concept and platform encompassing a set of benchmarks, frameworks, and evaluation methodologies for advanced reasoning, manipulation, mathematical problem-solving, and simulation tasks within the Odyssey research family. OdysseyBench aggregates diverse evaluation suites, including benchmarks for numerical analysis, long-context language understanding, mathematical reasoning, code synthesis, general-relativistic simulation, and embodied mobile manipulation, each designed to probe limitations of both current AI and physical systems under demanding, context-rich, or interdisciplinary conditions.

1. Benchmarking Principles and Multi-Domain Coverage

OdysseyBench defines a philosophy of evaluation for complex and high-level tasks, inspired and informed by multiple Odyssey-family works. The benchmarks under this umbrella are characterized by several shared design elements:

• Domain Specialization: Benchmarks such as MathOdyssey (Fang et al., 26 Jun 2024), MedOdyssey (Fan et al., 21 Jun 2024), and ODYSSEY (Wang et al., 11 Aug 2025) focus on tasks requiring expert-level reasoning in mathematics, medicine, or physical manipulation, often at domain boundaries where generic models struggle.
• Context Scaling: Long context understanding is a core theme, with MedOdyssey evaluating LLMs from 4K up to 200K tokens, and ODYSSEY requiring long-horizon planning and execution in physical mobile manipulation.
• Contamination Resistance: Dataset construction methodologies (e.g., in OIBench (Zhu et al., 12 Jun 2025)) emphasize original, unpublished tasks resistant to pretraining leakage, ensuring that benchmarks test true generalization.
• Fairness and Metric Rigor: Principles like "Maximum Identical Context" in MedOdyssey and time/space completion curves in OIBench guarantee standardized, comparable measurements across diverse systems.

OdysseyBench thereby serves as a multi-domain stress test for open-ended reasoning and manipulation, crossing boundaries between symbolic, numeric, multimodal, and embodied tasks.

2. Floating-Point Expression Analysis and Interactive Design Tools

The OdysseyBench framework notably includes extension and adoption of interactive workbenches for expert-driven floating-point expression rewriting, as exemplified by Odyssey (Misback et al., 2023):

• Three-Stage Iterative Workflow: Users diagnose expressions via error visualization (e.g., bit-accurate heatmaps), generate candidate rewritings through automated and manual search (leveraging Herbie), and tune solutions interactively across input ranges.
• Unified Expression Table: All candidate expressions—manual, automated, and derivatively generated—are collected, cross-compared, and visualized for traceable refinement, supporting error-driven decision making.
• Real-Time Diagnostics: Error metrics and local breakdowns are exposed at operator granularity, supporting rapid iteration and facilitating understanding of numerical issues.
• Expert Validation: Quantitative user studies document Odyssey's impact, with experts reporting improved cognitive load and performance on otherwise unsolvable rewriting scenarios.

This workflow embodies OdysseyBench’s philosophy of combining automated reasoning and human expertise with interactive tools for deep numerical evaluation.

3. Long-Context Language Understanding and Specialized Domain Benchmarks

MedOdyssey (Fan et al., 21 Jun 2024) establishes the long-context evaluation paradigm that OdysseyBench generalizes:

• Seven Context Length Levels: Evaluations cover 4K–200K token ranges, revealing performance degradation under extreme scale.
• Needle-in-a-Haystack and Reasoning Tasks: The dual structure combines "needle" detection in vast medical corpora with tasks requiring normalization, graph-based QA, and tabular/case reasoning.
• Counter-Intuitive Reasoning and Novel Fact Injection: Design choices minimize knowledge leakage, enforcing genuine model comprehension.
• Maximum Identical Context: Token-to-character alignment ensures that all models are judged on identical input spans, codified mathematically as

$$\min_{N \in C} \left( \frac{N}{\gamma} - L' \right), \quad L' \leq \frac{N}{\gamma}, \quad C = \{4k, ..., 200k\}$$

• Performance Analysis: SOTA proprietary models outperform open-source in many subbenchmarks; however, all models exhibit decline at greater context lengths and increased task complexity.

OdysseyBench leverages these design, metric, and contamination principles to structure fair, scalable evaluations in specialized domains.

4. Mathematical and Algorithmic Reasoning Benchmarks

MathOdyssey (Fang et al., 26 Jun 2024) and OIBench (Zhu et al., 12 Jun 2025) provide OdysseyBench with coverage of symbolic problem solving and algorithmic synthesis:

• Difficulty and Domain Stratification: MathOdyssey stratifies tasks into Olympiad, high school, and university levels, probing both routine and advanced reasoning (e.g., nontrivial combinatorics, embedded chain rule applications).
• Automated and Human-Evaluable Formats: Problems offered in open-ended, multi-choice, and true/false formats support both LLM and manual evaluation.
• Chain-of-Thought and Symbolic Verification: Benchmarking analyses employ chain-of-thought prompting and symbolic computation tools to rigorously test for mathematically equivalent solutions.
• OIBench Efficiency Metrics: OIBench innovates with time/space completion curves that plot the cumulative correct test case fraction versus normalized runtime/memory, enabling granular performance and efficiency decomposition.
• Contamination Risk Measurement:

$$\text{Risk Score} = \frac{S_{\text{contaminated}} - S_{\text{baseline}}}{1 - S_{\text{baseline}}}$$

confirms negligible leakage.

Comparative studies reveal closed models excel at routine tasks but struggle with advanced problems, while efficient reasoning hints can improve performance universally—shaping OdysseyBench’s approach to robust, leak-resistant benchmarking.

5. Mobile Manipulation and Embodied Planning Evaluation

The ODYSSEY framework (Wang et al., 11 Aug 2025) grounds OdysseyBench in embodied reasoning benchmarks:

• Unified Hierarchical Planning and Whole-Body Control: High-level semantic decomposition (via foundation models and vision-language reasoning) pairs with low-level RL policies for synchronized base+arm control.
• Instance-Level Semantic Maps: RGB and LiDAR fusion, combined with language grounding, yield maps supporting dynamic, goal-driven trajectory and pose planning.
• Vision-Language Manipulation: Wrist-mounted camera and VL model (Qwen2.5-VL-72B-Instruct) enable geometry-constrained contact prediction and end-effector orientation sampling.
• Curriculum RL Training: Locomotion and manipulation are learned in two stages, with terrain invariance and reward-shaping for coordinated gait and interaction.
• Long-Horizon Benchmarks: A comprehensive simulation suite covers short- and long-horizon navigation/manipulation, integrating hundreds of scene/task variations and supporting sim-to-real transfer.
• Performance and Robustness: Real-world deployments on quadruped+arm platforms demonstrate generalization, with robustness under partial failures and environmental stochasticity.

OdysseyBench thereby sets standards for complex manipulation, planning under uncertainty, and whole-system benchmarking in open-world robotic settings.

6. OdysseyBench: Impact, Generalization, and Future Directions

OdysseyBench synthesizes principles from its component benchmarks to create a unified, extensible evaluation ecosystem:

Benchmark Component	Domain	Key Evaluation Principle
Odyssey (GPU GRRT)	Astrophysics	Parallel, modifiable, real-time ray tracing
Odyssey (Expression Rewriting)	Floating-point numerics	Error visualization, workflow integration
MedOdyssey	Medical LLMs	Long-context fairness, domain expertise
MathOdyssey	Mathematical reasoning	Difficulty stratification, CoT
OIBench	Informatics/code	Efficiency curves, contamination resistance
ODYSSEY	Robotics	Semantic planning, RL control, sim-to-real

OdysseyBench’s multi-faceted approach enables deep diagnostics of where models fail: scaling context, reasoning under uncertainty, integrating symbolic and multimodal representations, and bridging simulated and physical domains. Future enhancements, as motivated in member papers, include cross-domain integration, alignment with hardware-specific optimizations, richer visualization, and continual updates for fairness and relevance.

A plausible implication is that OdysseyBench provides a model for rigorous, contamination-resistant, and context-aware benchmarking that is adaptable for new scientific and engineering domains, setting the standard for forthcoming interdisciplinary evaluation platforms in AI and robotics research.