MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources

Abstract: Achieving strong optimization generalization across diverse optimization problems while requiring limited training resources remains a challenging problem for optimization-oriented LLMs. Existing approaches typically rely on large-scale supervised datasets, costly reasoning annotations, and expensive intermediate step verification, resulting in substantial training overhead. To address these challenges, we propose MiniOpt, a reinforcement learning framework that learns to solve optimization problems through an "reasoning-to-model-and-solve" paradigm. MiniOpt decomposes optimization reasoning into structured optimization modeling and executable solver generation. Building upon this paradigm, we introduce OptReward, a reward function with hierarchical score structure that jointly evaluates formulation and solution, enabling effective policy learning without expert demonstrations. We further develop an optimization-oriented policy optimization strategy that improves exploration efficiency and stabilizes reinforcement learning for compact models. Extensive experiments show that MiniOpt-3B exhibits strong optimization generalization across various optimization types, problem scenarios, and task domains. For models with fewer than 10B parameters, MiniOpt series achieves the highest average solving accuracy (SA). For models with more than 10B parameters, MiniOpt still shows competitive performance. These results suggest that optimization-oriented reward design and reinforcement learning provide an effective pathway for developing compact optimization-specialized LLMs with strong optimization generalization capabilities. The code is available at https://github.com/Hsiang-1/MiniOpt.

• The paper presents a novel reasoning-to-model-and-solve pipeline that integrates RL-verifiable rewards to efficiently address diverse optimization challenges.
• It employs a lightweight SFT warm-up followed by two-stage RL training, achieving high solving accuracy (e.g., 64.76% SA) across multiple benchmarks.
• The framework’s structured reward design and modular pipeline enable small-scale LLMs to generalize across various optimization tasks while reducing inference costs.

MiniOpt: RL-Verifiable Reasoning-to-Model-and-Solve for General Optimization with Resource-Efficient LLMs

Motivation and Contributions

MiniOpt addresses the challenge of imbuing small-scale LLMs (sub-10B parameters) with strong optimization generalization across diverse problem types and scenarios, overcoming the limitations of data scarcity, annotation cost, verification overhead, and inference efficiency prevalent in optimization-oriented LLMs. Existing baselines typically rely on extensive SFT with high-quality data, reflection/debugging mechanisms that introduce computational bottlenecks, and sometimes closed-source solvers that restrict general applicability. MiniOpt integrates RL with verifiable reward (RLVR), introduces a reasoning-to-model-and-solve pipeline, and employs a structured reward function, OptReward, to enable efficient, scalable learning without expert demonstrations.

Methodology

Paradigm: Reasoning-to-Model-and-Solve

MiniOpt enforces a problem-solving pipeline via two output segments: > and <answer>. The <think> segment mandates explicit problem analysis and five-element tuple modeling (Variables, Objective, Constraints, Sets, Parameters), including automatic solver selection. The <answer> segment generates verifiable Pyomo code, executing the previously analyzed problem. This structure restricts the output action space to programmable artifacts, enabling deterministic verification and scoring for reward computation.

Figure 1: MiniOpt’s architecture: Reasoning-to-model-and-solve pipeline, training stages, and OptReward/OptGRPO algorithms.

Training Pipeline: SFT Warm-Up and Two-Stage RL

MiniOpt employs a lightweight SFT phase on operational optimization data to mitigate reward sparsity and provide a foundation for RL training. Subsequent RL training occurs in two stages:

• Stage 1: Paradigm acquisition using relatively easy problems, maximizing signal from structural/formatted reward components, improving modeling and solver code executability.
• Stage 2: Optimization generalization using type-uniform and scenario-aligned, more difficult problems, prioritizing accuracy signal and robust solver selection.

Stage-specific data selection and scheduling, including uniform coverage and scenario distribution preservation, further enhances efficient utilization of limited data.

RLVR: OptReward and OptGRPO

OptReward comprises three hierarchical scores: format correctness, five-element structural completeness, and numerical accuracy. Automated procedures parse response format, check labeled summaries, and execute generated code for solution validation, enabling low-cost verifiable RL. OptGRPO modifies the vanilla GRPO algorithm: removes KL penalty, implements asymmetric importance weight clipping (higher upper threshold for exploration), and adopts token-level policy gradient loss to stabilize learning and amplify correct reasoning patterns. This ensures robust exploration and learning efficiency even under computational and data constraints.

Experimental Results

Optimization Generalization

MiniOpt-7B achieves the highest average Solving Accuracy (SA) (64.76%) across eight operational optimization benchmarks, outperforming all prompt-based and most learning-based baselines among models sub-10B. MiniOpt-3B remains competitive against models several times larger, e.g., outperforming Qwen2.5-14B (+12.19%), matching GPT-5, and achieving strong scaling with size.

Figure 2: MiniOpt achieves Pareto-optimality in performance vs parameter scale, outperforming baselines with fewer resources.

Paradigm and RLVR Effectiveness

Ablations demonstrate essential contributions from each pipeline element: SFT warm-up, two-stage RL, OptReward, and OptGRPO provide additive gains to SA and executable rates (ER). Removing RL drops SA to 27.24% (from 59.65%), and removing two-stage scheduling or structured reward components significantly degrades performance.

Pareto Front: Performance vs Resource

MiniOpt models lie on the empirical Pareto frontier of parameter scale and SA, maximally dense in capability per parameter. Reducing scale from 7B to 3B to 1.5B sees smooth, moderate drop-off in performance, leveraging the reasoning-to-model-and-solve RLVR pipeline to maintain robustness.

Breadth and Scalability

MiniOpt generalizes across multiple optimization categories (LP, MILP, IP, NLP, SOCP, CO, MOP), and scenarios (transportation, supply chain, manufacturing, etc.), with consistent accuracy and executable rates (Figures 5, 6, 7). The model handles dimensionalities up to 80 variables, robustly bridging low-to-high complexity and diverse application domains.

Figure 3: Distribution histogram of optimization types across eight evaluation benchmarks.

Figure 4: Distribution histogram of problem scenarios across eight evaluation benchmarks.

Figure 5: Performance variations of MiniOpt on benchmarks from different domains.

Algorithmic and Theoretical Implications

The hierarchical reward structure in OptReward is essential – sensitivity analysis shows that collapsing reward differentiation impairs policy exploration and solution quality (Table IV, Figure 6). OptGRPO’s entropy trajectory avoids premature convergence (Figure 7), maintaining diversity in reasoning trajectories critical for non-convex search spaces in optimization. The RLVR framework circumvents costly annotation pipelines, reflection/debugging, and intermediary verification, leading to scalable training and inference.

Figure 6: Training dynamics of reward values and policy entropy for MiniOpt-3B under different reward function settings.

Figure 7: Training dynamics of reward function values and policy entropy: OptGRPO vs GRPO, highlighting improved exploration.

Practical Efficiency and General Capability

MiniOpt achieves competitive SA with only 24.40% of the output tokens used by DeepSeek-R1, lowering inference costs and enabling deployment on local and edge devices. Optimization-oriented RLVR training does not impair base model capability in other domains; evaluations on six general benchmarks (math, code, reading comprehension, QA, etc.) show average improvements of +1.24% over the base, indicating positive transfer and structural alignment.

Future Directions

MiniOpt suggests an efficient pathway for the development of compact LLMs specialized in optimization reasoning. The modular RLVR framework is broadly applicable to other code-centric and structured reasoning domains. Refinement of reward hierarchies, automatic scenario/type labeling, and further reductions in data annotation pipeline complexity are promising avenues for expanded generalization and deployment. Extending dimensionality scalability, integrating open solver adapters, and unsupervised reward modeling may enable robust, resource-constrained optimization reasoning at scale.

Conclusion

MiniOpt exemplifies resource-efficient, RL-verifiable optimization reasoning for small-scale LLMs. Its reasoning-to-model-and-solve paradigm, structured reward function, and tailored RL policy optimization facilitate generalization across diverse optimization types and scenarios with minimal computational overhead. MiniOpt strikes a definitive balance between optimization performance and efficiency, advancing the frontier of practical, deployable LLMs for automated reasoning in operations research and allied domains.