Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification

Abstract: Building mathematical optimization models is critical in operations research (OR), while it requires substantial human expertise. Recent advancements have utilized LLMs to automate this modeling process. However, existing works often struggle to verify the correctness of the generated optimization models, without checking the rationality of the constraints and variables or the validity of solutions to the generated models. This hampers the subsequent verification and correction steps, and thus it severely hurts the modeling accuracy. To address this challenge, we propose a novel LLM-based framework with Dual-side Verification (Opt-Verifier) from both structure and solution perspectives, thereby improving the modeling accuracy. The structure-side verification ensures that the modeling structure of the generated optimization models aligns with the original problem description, accurately capturing the problem's constraints and requirements. Meanwhile, the solution-side verification interprets and evaluates the solutions' validity, confirming that the optimization models are logically and mathematically sound. Experiments on popular benchmarks demonstrate that our approach achieves over 20\% improvement in accuracy.

• The paper introduces the Opt-Verifier framework, the tool and methodology with multi-level structure distillation and shows dual-side verification (structure-side and solution-side) using LLMs to significantly improve solving accuracy from 50.6% to 64.3% in benchmark problems
• Opt-Verifier detects key constraints often missed by existing models, corrects them by comparing generated models against canonical components and problem-class requirements for greater reliability.
• The framework outperforms existing state-of-the-art models even with a less powerful LLM backbone, showing robust improvements across multiple task types, baselines, and model scales.

Motivation and problem setting

Automated optimization modeling with LLMs has progressed along two main lines: prompt-based multi-agent or search frameworks such as Chain-of-Experts and OptiMUS, and fine-tuned OR-specific models such as ORLM, LLMOPT, OptMATH, and SIRL. Both lines share a common weakness in their self-correction mechanisms: correction is triggered almost exclusively by solver-code execution errors, so the underlying mathematical model can remain flawed even when the code runs cleanly. The paper identifies two failure modes that motivate its design: (1) LLMs frequently omit implicit constraints required by a problem class — OptiMUS reports such structural mistakes on 36.0% of NL4Opt and 12.6% of ComplexOR instances; and (2) existing debugging modules cannot detect formulation-level bugs, which "requires deeper reasoning" than coding errors, as the authors quote from OptiMUS.

A key motivating observation is that an LLM may fail to produce flow-balance constraints in a maximum-flow formulation from the raw description alone, yet recover them correctly when prompted to recall the standard formulation of the identified problem class. This suggests grounding modeling in canonical problem structures rather than relying on free-form generation.

Multi-level structure distillation

Opt-Verifier first distills a multi-level modeling structure $S$ from the natural-language description $D$ using a dedicated agent:

• High level: the fundamental OR problem type (e.g., maximum flow, knapsack, vehicle routing), associated with a base formulation.
• Medium level: classical variants (e.g., single-commodity vs. minimum-cost maximum flow).
• Low level: standard constraints of the classical model plus problem-specific requirements beyond it (e.g., time-varying capacities).

A formulation agent then generates the initial model as $M = \text{Formulation\_Agent}(D, S)$. The low-level tier is explicitly designed to preserve flexibility for problems that do not fit any canonical classification, avoiding forcing instances into rigid templates.

Structure-side verification

Inspired by dual learning in machine translation, the framework imposes a consistency criterion: back-translating the generated model into structure space should semantically match the structure distilled from the original description. A structure interpretation agent performs this back-translation, producing $\tilde{S} = \text{StruInterp\_Agent}(M)$; an evaluation agent then compares $\tilde{S}$ against $S$ and outputs a binary consistency score $c_c$ together with actionable comments used downstream for refinement. This catches errors of omission — missing constraints or variables that the textual description never states explicitly but that the identified problem class requires.

Solution-side verification

After executing solver code to obtain solution $x$, a solution interpreter renders $x$ as natural language grounded in the original context, and a solution evaluation agent critiques this rendering for logical or mathematical violations (e.g., more outflow than inflow at a reservoir). The output is a validity score $c_v$ plus diagnostic comments. This side exploits the commonsense reasoning capacity of LLMs to expose logical absurdities invisible to execution-based checks. A refinement agent consumes both sides' feedback to produce $D$0.

Notably, the entire pipeline uses GPT4o-mini as the backbone for all agents — a deliberately weak base model relative to frontier reasoning systems.

Main results

Across five benchmarks (NL4Opt, Mamo ComplexLP, ComplexOR, IndustryOR, OptMATH), Opt-Verifier achieves a micro-average solving accuracy (SA) of 64.3%, versus 50.6% for OptiMUS, 46.6% for CoE, 58.1% for OpenAI-o1, and 56.7% for DeepSeek-R1. Per-benchmark results:

Method	NL4Opt	Mamo	ComplexOR	IndustryOR	OptMATH
DeepSeek-R1	82.6	67.2	68.4	32.0	33.1
OpenAI-o1	87.1	66.3	68.4	36.0	32.5
SIRL	96.3	62.1	–	33.0	29.0
OptiMUS	83.0	45.0	73.6	31.0	20.2
Opt-Verifier	96.5	66.7	78.9	45.0	34.3

The claim that a GPT4o-mini-based pipeline outperforms o1 and R1 despite the much weaker backbone is a strong one; it implies verification quality, not raw model capability, is the binding constraint at current performance levels. Ablations confirm all three components contribute: removing structure augmentation drops IndustryOR SA from 45.0 to 34.0; removing either verification side also degrades results substantially. Removing the medium- or low-level structures costs 5–15 points, confirming the coarse-to-fine design matters.

Plug-and-play behavior: appended to OptiMUS, SA rises from 83.0→96.1 (NL4Opt) and 31.0→45.0 (IndustryOR); appended to the fine-tuned ORLM, from 85.1→92.3 and 38.0→42.0. With GPT-4o as backbone, NL4Opt reaches 97.5; with Qwen2.5-14B, gains are consistent though smaller. The framework also improves AlphaOpt (Mamo 62.6→68.7) and OptiMind (IndustryOR 42.0→48.0), suggesting its feedback captures errors missed by hint-based self-correction.

Efficiency: despite adding two verification loops, Opt-Verifier uses fewer tokens (e.g., 5,320 vs. 7,039 for OptiMUS on NL4Opt), fewer agent calls (~9 vs. ~14), and less time than CoE and OptiMUS. The authors attribute this to a fixed, directed workflow without a management agent that can loop redundantly. Verification steps themselves are lightweight (~300–660 tokens per stage versus >2,000 for coding/debugging).

Verifier reliability and scalability

On manually annotated IndustryOR samples (30 positive, 270 negative across difficulty tiers), verifier precision ranges from 83–93% and recall from 68–86%, degrading gracefully on hard problems outside conventional classifications. Error-type analysis shows recall up to 0.95 for variable errors via solution-side checking, but constraint-error recall is lower (0.67–0.79), indicating constraint-level bugs remain the hardest to catch — a limitation the authors acknowledge implicitly through these numbers.

For scalability, the framework generates parameterized formulations whose logic is instance-size agnostic; structure verification operates on abstract relationships, while solution verification runs on small toy instantiations (milliseconds per solve). On enlarged ComplexOR instances averaging ~18,759 variables and ~33,568 constraints (solve times exceeding 1,000 s), modeling accuracy remains comparable to small-scale settings (78.9% vs. OptiMUS's 73.6%). Correctness of large-scale coefficients is delegated to data-loading code rather than LLM transcription — a reasonable but assumption-laden design choice, since it presumes the binding code itself is correct.

Limitations and open questions

Several caveats bear directly on the reported results. First, the verifier reliability study relies on synthetic negative samples created by randomly deleting or rewriting constraints and variables; whether real LLM-generated errors exhibit the same detectability profile is not established, although an appendix analysis on genuine errors partially addresses this. Second, constraint-error detection (recall 0.67–0.79) lags other error types, leaving open how to improve structural recall on complex formulations. Third, the manual annotation effort was confined to IndustryOR with ten problems per difficulty tier, so reliability estimates carry nontrivial sampling uncertainty. Fourth, the scalability argument depends on toy-instantiation verification faithfully reflecting full-scale behavior — supported empirically here but not guaranteed in general. Finally, the framework presupposes that problems resemble some canonical OR classification at least at the high level; performance on genuinely novel problem families is tested only indirectly through the hard-tier results.

Conclusion

Opt-Verifier reframes self-correction in LLM-based optimization modeling around semantic verification rather than execution signals, combining structure-consistency checking via dual back-translation with solution-validity checking via natural-language critique, both grounded in distilled multi-level problem structures. The approach yields roughly 10-point micro-average gains over strong prompt-based baselines, surpasses frontier reasoning LLMs with a weaker backbone, transfers across baselines and model scales, and does so at lower token cost than competing multi-agent pipelines. Its principal open issues are improving constraint-level error recall and validating verifier reliability on organically generated errors at scale.