MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling

Abstract: Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although LLMs are increasingly used to generate optimization formulations and solver code, existing benchmarks are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but instance information is conveyed through visual artifacts such as tables, graphs, maps, schedules, and dashboards. We introduce multimodal optimization modeling, a benchmark setting in which models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each with an exact solver, and builds both the model-facing inputs and hidden reference files from the same verified source. We instantiate the framework as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We evaluate 9 multimodal LLMs (MLLMs), including 6 frontier general-purpose models and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The results show that the task remains far from solved: the best two models reach 52.1% and 51.3% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4% on easy instances and 15.9% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Failure attribution shows that errors arise both when extracting instance data from text and visuals and when turning extracted data into solver-correct formulations and code. MM-OptBench provides a testbed for solver-grounded, decision-oriented multimodal intelligence.

• The paper introduces MM-OptBench, a comprehensive benchmark that integrates textual and visual specifications to generate solver-executable optimization models.
• It employs a unified generation-verification pipeline ensuring solver-grounded correctness over diverse optimization families using pass@k evaluation metrics.
• Results reveal that while MLLMs achieve high Valid Code Rates, they often struggle with accurate instance extraction and formal model synthesis in complex settings.

MM-OptBench: Solver-Grounded Evaluation for Multimodal Optimization Modeling

Motivation and Problem Setting

Optimization modeling in practice requires integration of textual specifications with heterogeneous visual artifacts—such as tables, diagrams, maps, charts, and dashboards—which jointly define model variables, constraints, and objectives. Prior work and datasets in automated optimization modeling and multimodal AI have been almost exclusively text-centric, sidestepping the composite input modalities prevalent in real-world operational workflows. MM-OptBench addresses this gap by formalizing multimodal optimization modeling: producing both a mathematical formulation and solver-executable code from structured text-plus-visual specification.

Figure 1: Compact exemplar MM-OptBench instances in which textual rules and structured visual artifacts jointly specify the optimization model.

Unlike existing QA or answer-verification settings, this task requires a model to (a) extract distributed instance-defining data from both modalities, (b) synthesize a symbolic optimization model encoding the correct semantic scaffold, and (c) implement a programmatically verifiable, solver-compatible code artifact. The correctness is solver-grounded: only code returning the verified optimum for the instance passes.

Benchmark Taxonomy and Construction Principles

MM-OptBench is constructed on four criteria: realism of multimodality (semantic rules split between text and visual), solver-grounded verifiability (exact solution for each instance), structural diversity (distinct optimization families rather than template variations), and information-completeness/readability in the artifacts.

Benchmark coverage spans six major optimization families and 26 subcategories, systematically organized by dominant mathematical structure:

Figure 2: Taxonomy of MM-OptBench. The benchmark spans six major optimization families and 26 subcategories, organized by dominant mathematical structure rather than surface application domain.

These families include:

• Network Optimization: Flow, path, and design problems over directed networks.
• Location, Covering, and Assignment: Geometric and combinatorial selection/allocation.
• Scheduling and Sequencing: Temporal and resource-conflict reasoning.
• Multi-Period and System Planning: Time-indexed planning, energy dispatch, inventory.
• Routing and Tour Optimization: Euclidean tour and route construction.
• Pure Combinatorial and Logical Models: SAT, coloring, knapsack, set packing.

Each instance is validated via a unified generation-verification pipeline:

Figure 3: MM-OptBench construction and validation pipeline, involving expert-driven configuration sampling, solver-based verification, artifact synthesis, and quality evaluation.

All instances are derived from a single verified reference—minimizing drift between text, visuals, reference formulation, executable solver, and metadata.

Task Formalization and Evaluation Protocol

Given a multimodal specification $(T, V)$, where $T$ is descriptive text and $V$ is a sequence of structured visual artifacts, the modeling agent must output $(M, S)$: a symbolic formulation and solver-compatible code.

Evaluation is exclusively solver-grounded: Candidate code is executed under a prespecified harness and only accepted if the returned objective numerically matches the verified reference within tolerance. The evaluation protocol defines:

• Pass@k: Fraction of instances where any of $k$ generated outputs is solver-correct.
• Valid Code Rate: Fraction of code outputs syntactically and semantically runnable, irrespective of solution correctness.

A two-stage diagnostic is used for robust failure attribution: Stage 2 extraction probes whether errors stem from misreading the multimodal instance or from subsequent modeling/coding faults.

Empirical Study: Findings on Current MLLMs

Nine state-of-the-art MLLMs (six general-purpose, three math-specialized) are benchmarked without MM-OptBench-specific finetuning.

Overall Performance

Figure 4: Overall MM-OptBench performance for all evaluated MLLMs. The left bar is pass@1 (solver-correct), the right bar is Valid Code Rate.

• Best pass@1: GPT-5.4 (52.1%), Gemini 3.1 Pro Preview (51.3%)
• Mean pass@1 (general-purpose): 29.9%; mean pass@4: 42.2%
• Math-specialized MLLMs solve exactly 0 instances out of 780.

Notably, Valid Code Rate (ability to emit runnable code) substantially overstates solver-correct modeling capability, with some models exceeding 90% code validity yet solving only half the benchmark.

Family-Level and Difficulty Scaling

Model performance is highly dependent on optimization family and difficulty:

Figure 5: pass@1 means for six general-purpose MLLMs by optimization family (left) and difficulty level (right); darker = higher correctness.

• Highest pass@1: Combinatorial/Logical Models (42.5%), Location/Covering/Assignment (37.7%)
• Hardest: Scheduling/Sequencing (20.6%), Multi-Period/System Planning (23.5%)
• Performance degrades monotonically with structural difficulty: average pass@1 drops from 43.4% (easy) → 30.2% (medium) → 15.9% (hard).

Failure Attribution

Figure 6: Per-model failure attribution. Left: proportion of failures due to Stage-2 reading/extraction vs. downstream errors. Right: Oracle-reading failures; the majority are modeling/algorithmic.

• Major bottleneck: Upstream reading/extraction; 62.1% of failures are due to incorrect extraction of data from text and visuals.
• Downstream errors: After correct instance extraction, 32.7% of failures are due to modeling, code generation, or algorithmic mistakes, most commonly structural modeling errors or incomplete/incorrect solution procedures.
• Oracle-reading (ground-truth instance supplied): 51.7% of these are solved, but modeling/algorithmic and runtime errors remain substantial.

Implications and Theoretical/Practical Outlook

The results indicate that current MLLMs—general-purpose or math-specialized—are still considerably limited in assembling optimization models from distributed multimodal evidence. Solver-grounded correctness exposes that even sophisticated systems often fail at the modeling stage, not merely at code or execution.

Theoretical implications:

• The inability of mathematical reasoning finetuning or vision-language pretraining to generalize to model synthesis indicates that the gap between answer-centric and modeling-centric multimodal understanding is substantive.
• The observed family-level performance differentials and sharp difficulty-driven degradation underscore the specialization of existing models for shallow pattern association rather than robust formal reasoning over joint textual-visual domains.

Practical implications:

• Automated optimization modeling in mixed-modality environments—representative of logistics, energy, manufacturing, and other industrial contexts—remains unsolved even for frontier MLLMs.
• Deploying such systems in real-world decision settings, where model failures entail economic or safety risks, would require substantial improvement in both multimodal extraction and formal model construction capabilities.

Outlook for Future AI Developments:

• Further research is needed on end-to-end architecting of models that can robustly translate arbitrary multimodal operational specifications to formal models, possibly involving explicit intermediate instance representations, symbolic reasoning augmentation, and improved visual-semantic grounding.
• Enhanced diagnostic and verification benchmarks—centered on solver-grounded or symbolic correctness rather than answer tokens—will be critical for advancing trustworthy, auditable multimodal intelligence.

Conclusion

MM-OptBench establishes a rigorous testbed for solver-grounded, decision-oriented multimodal intelligence, requiring both multimodal information extraction and formal model construction. Empirical evidence reveals substantial gaps in current multimodal model capabilities, especially at increased structural complexity and across less familiar problem classes. The findings assert that the crucial capabilities for trustworthy semi-autonomous optimization modeling—spanning extraction, abstraction, and code synthesis—are still beyond baseline MLLMs and that architectural, training, and benchmarking advances targeting robust multimodal reasoning and symbolic fidelity are required for progress in formal decision-modeling automation.