LogiOR: Optimization Benchmark in Logistics

• LogiOR is an optimization modeling benchmark featuring 92 logistics problems and standardized annotations to support end-to-end evaluation.
• It integrates natural language descriptions, formal mathematical models, and Python-based solver code to assess modeling accuracy and efficiency.
• The benchmark addresses prior limitations with high error rates and narrow evaluation metrics, providing a diagnostic tool for OM research.

LogiOR is an optimization modeling benchmark introduced in “Automated Optimization Modeling through Expert-Guided LLM Reasoning” as a resource for evaluating and automating optimization modeling in the logistics domain (Yang et al., 20 Aug 2025). It was designed to address three limitations identified in prior work: high benchmark labeling error rates reaching up to 42\%, narrow evaluation scope that only considers optimal values, and computational inefficiency due to heavy reliance on multi-agent systems or model fine-tuning. In that setting, LogiOR provides more complex problems with standardized annotations, and serves both as a benchmark for LLM–based optimization modeling and as a structured corpus of logistics and supply chain optimization instances (Yang et al., 20 Aug 2025).

1. Origin and stated purpose

LogiOR is positioned within optimization modeling (OM), where the central task is to transform a natural-language decision problem into a formal mathematical program and an executable solver implementation. The benchmark was introduced together with dataset correction and annotation work on existing resources, and with ORThought, a framework that leverages expert-level optimization modeling principles through chain-of-thought reasoning to automate the OM process (Yang et al., 20 Aug 2025).

The benchmark’s stated purpose is not merely to increase the number of test instances. Its role is to broaden evaluation beyond optimal values alone and to support systematic assessment of whether a model can recover the formal model, executable code, and problem structure. A plausible implication is that LogiOR was intended as an infrastructure benchmark for end-to-end OM research rather than as a narrow answer-only test set.

2. Corpus construction and annotation schema

LogiOR contains 92 carefully curated problems focused on logistics and supply chain optimization (Yang et al., 20 Aug 2025). The problems were developed/adapted with guidance from three Operations Research (OR) experts over two months, and each instance is accompanied by standardized annotations (Yang et al., 20 Aug 2025).

For each problem, the benchmark includes the following elements:

• Natural Language Description: Realistic, context-rich logistics scenario.
• Mathematical Model: Formally defined with LaTeX formulas, structured as sets, parameters, decision variables, objective, and constraints.
• Python (Gurobipy) Code: Full function-based implementation matching the mathematical model, with standardized argument/return conventions.
• Ground-truth Solution: Objective value (or indicator for infeasibility, etc.).
• Problem Metadata: Type (LP, ILP, etc.), size (Toy/Small/Medium), number of variables/constraints/nonzeros. (Yang et al., 20 Aug 2025)

This annotation scheme is central to LogiOR’s identity. A frequent simplification in optimization-modeling evaluation is to treat the final optimum as the only object of interest; LogiOR was introduced specifically against that narrow evaluation scope. Its annotations support peer-verifiable checking of modeling, coding, and solution quality, and the paper describes this as enabling systematic checking for annotation/coding/modeling accuracy (Yang et al., 20 Aug 2025).

3. Problem taxonomy and scale regime

LogiOR spans the main categories of optimization modeling problems found in logistics. The benchmark covers LP, ILP, MILP, and NLP, rather than restricting evaluation to LPs or toy problems (Yang et al., 20 Aug 2025).

Problem type	Number of problems
LP	22
ILP	43
MILP	11
NLP	16

The benchmark also includes an explicit size classification:

• Toy: <5 variables, <10 constraints, <20 non-zeros
• Small: <25 variables, <40 constraints, <80 non-zeros
• Medium: larger/complex instances (Yang et al., 20 Aug 2025)

The paper characterizes LogiOR as notable in including more medium-sized and challenging MILP/NLP instances than previous benchmarks. It also states that this variety makes LogiOR substantially more challenging and representative compared to earlier datasets, which were often limited to LPs or toy problems (Yang et al., 20 Aug 2025).

4. Mathematical and executable representations

A distinctive feature of LogiOR is the explicit alignment between natural-language problem statements, formal mathematical programs, and executable solver code. The mathematical models use rigorous notation for decision variables, objectives, and constraints, with canonical forms such as $x_{ij} \geq 0$ and $y_k \in \{0,1\}$, alongside standard summation and indexing structure (Yang et al., 20 Aug 2025).

An example objective provided for a logistics congestion problem is:

$$\min \sum_{r \in R} \left( \text{LinearCostPerTon}[r] \cdot \text{TonnageOnRoute}[r] + \text{CongestionCoeff}[r] \cdot (\text{BackgroundTraffic}[r] + \text{TonnageOnRoute}[r])^2 \right)$$

with example constraints

$$\sum_{r \in R} \text{TonnageOnRoute}[r] = \text{TotalTonnage}$$

$$\text{TonnageOnRoute}[r] \geq 0 \quad \forall r \in R$$

The executable layer is given as Gurobi implementation code (Python), with variable and constraint names matching the formal model and a direct mapping from the LaTeX-style model to Gurobi code (Yang et al., 20 Aug 2025). This direct model-to-code correspondence is significant because it permits evaluation of whether a system produces a mathematically correct formulation and an operational solver implementation, rather than only a plausible textual explanation.

5. Role in evaluating LLM-based optimization modeling

LogiOR is used in the paper as the principal logistics-domain benchmark for ORThought, which is described as a framework using expert-guided chain-of-thought reasoning and a two-agent design (Model Agent and Solve Agent), to automate OM (Yang et al., 20 Aug 2025). The evaluation uses two main metrics:

• Success Rate: Run is successful if LLM-generated model attains ground-truth optimum (verified by experts).
• Token Consumption: Proxy for computational/memory efficiency. (Yang et al., 20 Aug 2025)

The reported success rates on LogiOR are as follows:

Method	LogiOR Success Rate
Standard baseline	33.34%
Chain-of-Experts	34.78%
Chain-of-Thought	37.32%
Self-Consistency	32.25%
Reflexion	36.59%
ORThought	46.01%

The paper reports that ORThought achieves a 46.01\% success rate—more than 9 percentage points higher than any baseline, which is a ~28\% relative improvement over strong baselines (Yang et al., 20 Aug 2025). It further states that ORThought achieves superior accuracy with lower average token usage versus multi-agent methods like CoE, which can use 50,000+ tokens per problem (Yang et al., 20 Aug 2025).

Granular analysis is also provided. By problem type, ORThought is reported to excel most on ILP (82.33\%), MILP (30.77\%), and NLP (51.65\%), while for LP it is competitive with top baseline performance. By problem size, its advantage is strongest on small and medium problems, with 49.59\% and 43.4\%, respectively, and gains of 6–11 percentage points over second-best (Yang et al., 20 Aug 2025).

6. Relation to prior benchmarks, failure modes, and research significance

The paper explicitly contrasts LogiOR with previous benchmarks such as NLP4LP, ComplexOR, and IndustryOR. In that comparison, LogiOR is described as logistics-focused but generalizable, high in complexity, uniformly annotated with natural language, formal model, code, ground-truth, and metadata, and improved in quality through expert crafting and error correction (Yang et al., 20 Aug 2025). The comparison emphasizes that previous versions had error rates up to 42\%, whereas LogiOR is framed as peer verifiable and standardized.

The benchmark also supports systematic failure analysis. On LogiOR, the most frequent errors are reported as incorrect constraint modeling, followed by issues with variable definitions. Missing constraints are rarer than incorrect ones, and “spurious” (extraneous) constraints are rarest, which the paper interprets as showing principled modeling behavior (Yang et al., 20 Aug 2025). The paper also states that all methods’ performance decreases for larger/more complex problems, describing this as a scaling challenge that highlights automation’s limits in scaling OM (Yang et al., 20 Aug 2025).

These observations are important for interpreting LogiOR. The benchmark is not only a score table for solver outputs; it is also a diagnostic instrument for model formulation quality. A plausible implication is that LogiOR’s main long-term value lies in making it possible to distinguish between superficially correct optimization answers and genuinely correct optimization models, which is essential in logistics OM where constraint fidelity is often more consequential than raw objective matching.