Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation (2412.15118v2)

Abstract: LLMs excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision relies on learned reward models requiring costly training data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps. We introduce Outcome Refining Process Supervision, which unifies process and outcome supervision by leveraging executable verification: a tree-structured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that integrate runtime feedback with reasoning. Experiments across 5 models and 3 benchmarks show consistent gains, with 26.9% higher correctness and 42.2% improved code efficiency. The results demonstrate that ORPS enables LLMs to overcome local optima in code generation, suggesting a promising direction for combining verifiable outcomes with structured reasoning to tackle complex challenges. We open-source at: https://github.com/zhuohaoyu/ORPS

• The paper introduces ORPS, a novel supervision method that refines coding outcomes using execution signals rather than relying on extensive reward model training.
• It employs a tree-structured exploration strategy to maintain multiple solution pathways, resulting in a 26.9% increase in code correctness and a 42.2% boost in efficiency.
• The approach reduces dependency on large training datasets and mitigates hallucination, enabling smaller models to outperform larger counterparts in complex coding tasks.

Outcome-Refining Process Supervision for Code Generation

The paper "Outcome-Refining Process Supervision for Code Generation" addresses significant challenges that LLMs face in generating reliable code, especially for complex programming tasks requiring deep algorithmic reasoning. Traditional processes have focused on outcome supervision, which evaluates models solely based on the final output quality, or process supervision, which involves guiding models through reasoning steps using learned reward models. Despite their utility, these methods encounter limitations, including dependence on costly training data and unreliable evaluations due to hallucination issues in models.

To address these constraints, the authors propose Outcome-Refining Process Supervision (ORPS), a novel approach that shifts the paradigm by adopting the refinement of coding outcomes as the supervised process itself. This insight is realized through the use of concrete execution signals which provide objective feedback to guide the reasoning steps rather than relying on Process Reward Models (PRMs) which require extensive training. The framework leverages a tree-structured exploration strategy that maintains multiple solution pathways, allowing smaller models to achieve high success rates—evident in their reported average increases of 26.9% in correctness and 42.2% in efficiency across five models and three datasets.

The methodology underlying ORPS is robust in its design. By integrating execution-driven signals, the authors effectively ground the model's problem-solving capabilities in tangible, verifiable outcomes rather than relying solely on intermediate learned judgments. This method not only reduces hallucination in model predictions but also avoids the substantial data requirement burden of traditional PRM training. The tree-based exploration approach supports the simultaneous search for diverse solutions enabling models to explore alternative algorithmic strategies rather than limiting improvements to minor corrections of initial implementations.

Experiments showcased substantial improvements, as the proposed framework outperformed existing methods in achieving high passthrough rates and efficiency in solving datasets like LBPP, HumanEval, and MBPP. The most notable findings suggest the sufficiency of reasoning space provided by ORPS is more pivotal for tackling complex tasks compared to simply scaling model size. Notably, the method demonstrated that smaller models such as Qwen-7B could outpace their larger counterparts (Qwen-14B) under optimized conditions.

ORPS also departs from discriminatively trained reward models by utilizing a generative reward mechanism, showcasing improved performance due to its ability to incorporate contextual and rational analysis when evaluating intermediate steps. This generative judgement framework not only supports coherence in reasoning but also enhances model capacity to self-correct in structured coding tasks.

The implications of this research are significant for the development and deployment of LLMs in applied code generation where complex problem-solving is required. It demonstrates a capacity to reduce dependency on large-scale annotation efforts while achieving high efficiency and correctness, lending insights into future paradigms for model development. The approach highlights a path toward improved autonomous model capabilities that facilitate not only more efficient software development practices but also potential applications spanning diverse computational intelligence fields. The research sets a precedent for leveraging execution feedback loops and generative reasoning strategies as integral components of model refinement—a valuable direction for future artificial intelligence advancements.