StepORLM: Self-Evolving ORLM Framework

• StepORLM is a self-evolving framework for OR language models that combines solver-grounded outcome verification with generative process supervision for long-horizon reasoning.
• It employs a two-stage training process with initial supervised fine-tuning followed by an iterative co-evolutionary loop that refines both the policy and process verifier.
• The framework achieves state-of-the-art results across six OR benchmarks by shifting supervision from endpoint correctness to holistic, trajectory-level evaluation.

StepORLM is a self-evolving framework for operations research LLMs (ORLMs) that combines solver-grounded outcome verification with generative process supervision in order to train LLMs on optimization modeling and solving tasks. It is introduced as a response to two deficiencies in existing reinforcement-learning-style training for OR: outcome reward suffers from the credit assignment problem, and conventional discriminative process supervision is myopic for long-horizon, interdependent OR reasoning. At the center of the framework is a co-evolutionary loop in which a policy model and a generative process reward model (GenPRM) iteratively improve one another, using dual feedback from an external solver and from holistic retrospective process evaluation. The resulting 8B-parameter system is reported to establish a new state of the art across six benchmarks, and its co-evolved GenPRM also functions as a universally applicable inference-time process verifier (Zhou et al., 26 Sep 2025).

1. Conceptual basis and problem formulation

StepORLM is designed for the setting in which a LLM must solve operations research problems whose reasoning trajectories are long-horizon and interdependent. In this setting, early decisions about variables, assumptions, and constraint structure affect downstream feasibility and correctness. The framework is therefore built around the thesis that OR requires trajectory-level process supervision rather than supervision confined either to final answers or to isolated intermediate steps (Zhou et al., 26 Sep 2025).

The motivating failure mode of outcome-only reward is the standard credit assignment problem. A model may obtain a correct final answer even when its intermediate modeling decisions are flawed, so purely outcome-based reinforcement can strengthen defective reasoning patterns. StepORLM treats this as especially problematic for OR, where a correct final optimum may be compatible with intermediate formulations that are logically unsound or accidentally compensated for later in the trajectory.

The second motivating failure mode is conventional discriminative process supervision. Step-wise process reward models evaluate steps one by one, but OR steps are often only interpretable in context. A variable definition can appear incomplete until later constraints and the objective are introduced; a local classifier that scores each step independently is thus described as “myopic.” StepORLM replaces this with a critic that reasons over the completed trajectory and then retrospectively evaluates interdependencies among steps. This suggests a shift from local step validation to whole-trajectory logical assessment as the relevant unit of supervision for OR reasoning.

2. Two-stage training architecture

StepORLM is organized into two stages: warm-up supervised fine-tuning and iterative co-evolution. The warm-up stage constructs an initial policy, while the second stage jointly refines the policy model and the process verifier in a self-evolving loop (Zhou et al., 26 Sep 2025).

In the warm-up stage, a teacher LLM, specified as GPT-4o, synthesizes OR questions $Q$ together with step-by-step reasoning trajectories $R$. These trajectories are rigorously checked with an external solver, and invalid outputs are refined in an automatic loop until they pass verification. The verified $(Q,R)$ pairs are then used to train an initial policy $\pi_0$ via supervised fine-tuning.

The iterative stage begins from the initial policy $\pi_0$ and a base critic $\rho_{\theta_0}$. The policy model $\pi_\theta$ samples multiple candidate trajectories; the external solver verifies final outcome correctness; the GenPRM $\rho_\phi$ evaluates the entire trajectory and assigns process quality; and the resulting feedback is used to refine both components. The framework characterizes this as a co-evolutionary loop: a better policy generates better trajectories, better trajectories improve supervision for the GenPRM, and a better GenPRM supplies better preference signals for policy training.

This architecture places StepORLM within the ORLM lineage while changing the supervision mechanism. ORLM had already framed optimization modeling as learning a mapping $f:p \rightarrow (m,c)$, where a natural-language problem is converted into a mathematical model and executable solver program (Huang et al., 2024). StepORLM retains the solver-grounded modeling-and-code-generation orientation, but reorients training around self-evolution and process-aware preference optimization rather than supervised fine-tuning alone.

3. Dual-feedback supervision and the role of GenPRM

The central supervisory design in StepORLM is a dual-feedback mechanism that evaluates each sampled trajectory from two complementary perspectives: definitive outcome verification from an external solver and holistic process evaluation from the GenPRM (Zhou et al., 26 Sep 2025).

Outcome verification is performed by executing the final code or solution. The solver produces a definitive success or failure signal. This component grounds the framework in executable correctness rather than textual plausibility. It is the source of the highest-confidence supervision and is prioritized when preference pairs are constructed.

Process evaluation is handled by the GenPRM. Unlike a discriminative local checker, the GenPRM performs holistic retrospective evaluation of the complete trajectory and outputs step-level correctness scores after reasoning over the full sequence. The paper explicitly distinguishes this from conventional step-wise PRMs: the GenPRM is generative, not merely discriminative, and its judgments are contextual rather than step-local.

The combination of these signals serves two purposes. First, it creates preference pairs for policy alignment. Second, it provides labeled process data for continued GenPRM refinement. The practical consequence is that the verifier is not static: it improves on newly collected, solver-checked trajectories generated by the evolving policy. A plausible implication is that the GenPRM is intended to internalize regularities of valid OR reasoning that are not reducible to simple local correctness labels.

A common misconception would be to treat StepORLM as an outcome-reward method with an auxiliary step scorer. That characterization is incomplete. The framework’s defining claim is that OR requires retrospective trajectory-level evaluation, and the GenPRM is presented as the mechanism that supplies precisely that form of supervision.

4. Weighted Direct Preference Optimization and self-evolution

Policy alignment in StepORLM is implemented with Weighted Direct Preference Optimization (W-DPO), using preference tuples $(\tau_w,\tau_l,w)$, where $R$0 is the preferred trajectory, $R$1 is the dispreferred trajectory, and $R$2 weights the informativeness or trustworthiness of the comparison (Zhou et al., 26 Sep 2025).

Preference construction is governed by a solver-first rule. If one trajectory is solver-correct and the other is not, the solver-correct trajectory wins with weight $R$3. If both trajectories have the same solver outcome, the preferred one is the trajectory with the higher CorrectStepRatio as judged by the GenPRM. In those tied-outcome cases, the preference weight equals the difference in correct-step ratios. The design therefore gives solver-distinguished pairs maximal authority while still allowing process-quality gaps to contribute when solver outcomes do not separate candidates.

The W-DPO objective is

$R$4

Here $R$5 is the input OR problem, $R$6 is the policy likelihood of trajectory $R$7, and $R$8 is a scaling factor. The framework interprets this objective as DPO with weighted preference pairs, so that solver-verified comparisons dominate while process-based comparisons contribute in proportion to their quality gap.

GenPRM refinement proceeds in parallel. Verified trajectories from each iteration provide step labels for continued supervised fine-tuning of the GenPRM. The resulting loop is summarized as

$R$9

This feedback structure is the basis for the description “self-evolving.” The policy and critic are not independently optimized components but mutually reinforcing participants in a co-evolutionary training process.

5. Experimental configuration and evaluation regime

The reported StepORLM system uses Qwen3-8B as the base policy model and is trained on 50K synthesized questions. Training is carried out on a single node with $(Q,R)$0 NVIDIA H100 80GB, using AdamW with learning rate $(Q,R)$1, weight decay $(Q,R)$2, maximum length 8192, and special tokens <step> and </step>. In the self-evolving stage, four trajectories are sampled per response, and W-DPO uses $(Q,R)$3 (Zhou et al., 26 Sep 2025).

Solver choices are benchmark-dependent in the reported comparisons. COPT is used for StepORLM and ORLM, Gurobi for generalist and agentic baselines, and Pyomo for LLMOPT. These details matter because StepORLM’s outcome verification is solver-grounded, and solver execution is an integral part of both training and evaluation.

The evaluation covers six public OR benchmarks: NL4Opt, MAMO-EasyLP, MAMO-ComplexLP, NLP4LP, ComplexOR, IndustryOR, and ReSocratic, with the clarification that MAMO is split into EasyLP and ComplexLP in the result table, yielding seven reported columns. The main reported metric is Pass@1 accuracy.

This benchmark regime places StepORLM in continuity with earlier ORLM evaluation on NL4OPT, MAMO, and IndustryOR, but extends the scope to additional OR datasets (Huang et al., 2024). Relative to earlier ORLM-style systems, the experimental design is notable for evaluating both a trained policy and the same system augmented by its co-evolved verifier at inference time.

6. Reported results, inference scaling, and broader significance

The reported 8B StepORLM achieves an average Pass@1 of $(Q,R)$4 and is said to establish a new state of the art, outperforming much larger generalist models such as DeepSeek-V3 (671B) and Qwen2.5-72B-Instruct, agentic GPT-4o-based systems, and specialized OR baselines including ORLM and LLMOPT (Zhou et al., 26 Sep 2025).

When the co-evolved GenPRM is used as an inference-time verifier, the average rises to $(Q,R)$5. The reported gains are especially pronounced on harder benchmarks: ComplexOR improves by $(Q,R)$6 points relative to StepORLM alone, and IndustryOR improves by $(Q,R)$7 points. In this usage mode, inference scaling means sampling multiple candidate solutions and using the GenPRM to rerank or select among them rather than changing the base policy itself.

A major empirical claim is that the final GenPRM generalizes beyond its co-trained policy. It is evaluated as a universal inference-time verifier on both the final StepORLM and an independent open-source ORLM baseline, against majority voting, solver execution, a discriminative PRM, the initial GenPRM, and the final GenPRM. The final GenPRM is described as the strongest or near-strongest inference-scaling method on average, and when paired with ORLM it improves average performance by about $(Q,R)$8. This suggests that the verifier learns model-agnostic principles of valid OR reasoning rather than merely overfitting to the trajectories of a single policy.

In relation to subsequent OR work, StepORLM occupies a distinctive position. OR-R1 also targets automated OR modeling and solving with Qwen3-8B, but its emphasis is data efficiency through supervised fine-tuning plus Test-Time Group Relative Policy Optimization on unlabeled test data, reporting $(Q,R)$9 average solving accuracy with only $\pi_0$0 of the synthetic data used by ORLM (Ding et al., 12 Nov 2025). Step-Opt, by contrast, focuses on iterative data synthesis and structured validation for fine-tuning open-source models, reporting especially large gains on complex OR tasks (Wu et al., 21 Jun 2025). StepORLM’s specific contribution within this landscape is the combination of solver-grounded truth, generative holistic process supervision, and policy-critic co-evolution.

The broader significance of StepORLM lies in its reformulation of supervision for OR LLMs. Rather than treating OR problem solving as a task where only the final solver outcome matters, it treats it as a long-horizon reasoning process whose internal trajectory must be judged retrospectively and holistically. This suggests a general methodological shift for ORLM training: accurate executable outputs are necessary, but they are not sufficient as the sole training signal when the objective is robust optimization modeling rather than isolated end-state correctness.