OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving

Abstract: While LLMs demonstrate remarkable reasoning, complex optimization tasks remain challenging, requiring domain knowledge and robust implementation. However, existing benchmarks focus narrowly on Mathematical Programming and Combinatorial Optimization, hindering comprehensive evaluation. To address this, we introduce OptiVerse, a comprehensive benchmark of 1,000 curated problems spanning neglected domains, including Stochastic Optimization, Dynamic Optimization, Game Optimization, and Optimal Control, across three difficulty levels: Easy, Medium, and Hard. The experiments with 22 LLMs of different sizes reveal sharp performance degradation on hard problems, where even advanced models like GPT-5.2 and Gemini-3 struggle to exceed 27% accuracy. Through error analysis, we identify that modeling & logic errors remain the primary bottleneck. Consequently, we propose a Dual-View Auditor Agent that improves the accuracy of the LLM modeling process without introducing significant time overhead. OptiVerse will serve as a foundational platform for advancing LLMs in solving complex optimization challenges.

• The paper introduces the OptiVerse benchmark with 1,000 diverse problems across six domains to evaluate LLMs' optimization reasoning.
• It employs a multi-stage pipeline and a Dual-View Auditor Agent to detect and mitigate semantic modeling errors.
• Results reveal that even advanced LLMs struggle on Hard tasks, exposing systematic weaknesses and domain-specific fragility.

OptiVerse: A Comprehensive Benchmark for Optimization Problem Solving

Motivation and Context

Accurately mapping natural language descriptions to executable optimization models remains a critical challenge for LLMs, despite their progress in mathematical reasoning and code generation. While previous benchmarks have focused narrowly on domains such as mathematical programming and combinatorial optimization, they overlook large swaths of real-world optimization, including stochastic optimization, dynamic optimization, optimal control, and game optimization. This lack of diversity hinders both a rigorous assessment of LLM capabilities and insight into their ability to generalize reasoning across paradigms.

OptiVerse Benchmark Construction

OptiVerse addresses this gap through a rigorously curated set of 1,000 optimization problems, stratified across three difficulty levels and spanning six domains: Mathematical Programming (MP), Combinatorial Optimization (CO), Stochastic Optimization (SO), Dynamic Optimization (DO), Optimal Control (OC), and Game Optimization (GO). Problems are sourced from authoritative textbooks and challenge sets, then standardized and filtered through a multistage pipeline to maximize problem validity and complexity.

Figure 1: The hierarchical taxonomy of OptiVerse benchmark, which provides comprehensive coverage across six distinct optimization domains to comprehensively evaluate the diverse reasoning capabilities of LLMs.

Each domain imposes distinct abstract and formal requirements, ranging from static deterministic structures to dynamic, multi-agent, and probabilistic models.

Figure 2: Domain distribution in OptiVerse benchmark, with each color representing a distinct optimization domain.

Complexity increases systematically between Easy, Medium, and Hard strata, as shown by the problem token and output statistics.

Figure 3: Statistical analysis of question tokens and result counts across optimization problem domain and difficulty, benchmarked against NL4Opt and Optibench.

This stratification, combined with the breadth of domains and integrated tabular/graphical contexts, enforces both generalization and scalability, pushing LLMs beyond rote pattern matching or narrow solver dependency.

Experimental Methodology

Twenty-two recent LLMs—representing open-source “non-thinking” and “thinking” models as well as closed-source proprietary systems—were evaluated purely on textual inputs.

• Response generation leverages structured, code-centric chain-of-thought prompting.
• Output is assessed via an LLM-as-judge paradigm, combining answer extraction and semantically robust verification (tolerance $\epsilon=0.1\%$).
• The evaluation environment incorporates a comprehensive solver suite (e.g., gurobi, casadi, pyomo, nashpy, ortools).

This framework ensures not only rigorous correctness determination but also model flexibility across paradigm and implementation modalities.

Results and Analysis

Aggregate Performance and Scaling

The results expose several persistent systemic weaknesses:

• Significant performance degradation on Hard tasks: Even advanced models such as Gemini-3-Pro and GPT-5.2 fail to exceed 27% accuracy on Hard problems, despite >90% accuracy on Easy examples.
• Reasoning chain superiority: Models configured with explicit “thinking” or chain-of-thought mechanisms substantially outperform instruction-tuned counterparts of equivalent scale.
• Strong scaling law effects: Intra-family scaling (e.g., Qwen3-8B $\rightarrow$ Qwen3-235B) delivers monotonic accuracy gains across all categories.
• Domain-specific fragility: MP and CO tasks remain substantially easier, with consistent underperformance on SO, DO, OC, and GO

This domain fragility underscores that LLMs’ symbolic reasoning and translation capabilities do not generalize reliably across optimization subfields.

Error Taxonomy and Bottleneck Identification

Manual error audit categorizes failures into five main types: Modeling Logic, Parameter Utilization, Feasibility Violation, Optimality/Numerical, and Code Syntax errors.

• Modeling Logic errors are the dominant bottleneck in all models, even the most capable. These errors reflect incorrect translation from natural language to mathematical formulation—not syntactic execution or data extraction.

  Figure 4: Distribution of error types across nine representative models. While Code Syntax errors become negligible with increased capability, Modeling Logic errors remain the predominant bottleneck across all LLMs.

• Syntax-level and API errors are significantly reduced in models with reasoning-optimized architectures.
• A second-order bottleneck persists in numerical/optimality challenges, indicating solver configuration and algorithm selection sub-optimality even when high-level modeling is correct.

Dual-View Auditor Agent (DVA-Agent)

To directly address silent semantic misalignments—where the generated code is syntactically correct but logically incompatible with the problem—OptiVerse introduces the Dual-View Auditor Agent (DVA-Agent).

• Phase 1: Extract requirements from the problem text—including constraints, objectives, and key parameters overlooked by models.
• Phase 2: Perform blind code abstraction, reconstructing the mathematical meaning of the generated code without reference to the original prompt.
• Phase 3: Cross-reference to detect gaps; if discrepancies exist, regeneration is triggered with targeted correction.

  Figure 5: The workflow of the Dual-View Auditor Agent which utilizes a three-phase auditing mechanism to detect semantic discrepancies between the problem and code, determining whether to make modifications.

This semantic triangulation is model-agnostic and only intervenes (≈ 30% of the time) when misalignment is detected, providing significant accuracy gain with minimal additional computational overhead.

• Qwen3-235B-Instruct: +7.66% (Hard), +10.5% (Medium)
• Consistent improvements are observed across all model sizes and families.

Representative Problem Examples

• Easy: Linear scheduling, straightforward arithmetic and algorithmic steps (Critical Path, cost-slope).
• Medium: Game Optimization coupled with stochasticity, requiring backward induction, expectation, and calculus.
• Hard: Multi-period mixed-integer programming with complex constraint translation and variable remapping.

  Figure 6: An Easy Example in OptiVerse.

  

  Figure 7: A Medium Example in OptiVerse.

  

  Figure 8: A Hard Example in OptiVerse.

The progression of complexity in these instances embodies the benchmark’s rigorous challenge profile and highlights the cognitive gaps in LLM abstraction and modeling.

Practical Implications and Theoretical Outlook

OptiVerse establishes a robust platform for quantifying and systematizing the limitations of LLM optimization reasoning. Key implications include:

• Realistic deployment scenarios: Current SOTA LLMs underperform significantly on tasks requiring multi-stage mathematical abstraction, limiting applicability in engineering design, industrial scheduling, game-theoretic analysis, and control.
• Research directionality: The preeminence of semantic modeling errors suggests that advances in LLM alignment, semantic representation, and self-audit mechanisms (beyond naive self-consistency or code verification) are crucial.
• Benchmarking and generalization: The cross-domain diversity and stratified difficulty facilitate nuanced, longitudinal comparisons of next-generation AI architectures and adaptation strategies.

Looking forward, direct extensions include real-world noise integration, further multi-modality (e.g., vision-language or data-to-model), and tighter coupling to iterative or interactive modeling frameworks.

Conclusion

OptiVerse delivers a rigorously designed, domain- and complexity-diverse benchmark for optimization modeling with LLMs, exposing critical failure modes and setting an actionable agenda for future research. The integration of adversarial auditing via DVA-Agent demonstrates measurable mitigation of silent semantic errors, marking a tractable path forward for robust AI deployment in optimization-critical applications.

(2604.21510)