--- title: 'RL-Exec: Reinforcement Learning for Code Reasoning' url: https://www.emergentmind.com/topics/rl-exec type: topic --- # RL-Exec: Reinforcement Learning for Code Reasoning Searching arXiv for the cited papers and closely related usages of “RL-Exec.” RL-Exec, in the sense crystallized by ExecVerify, denotes reinforcement learning for **execution-based code reasoning**, in which a language model is trained not merely to imitate textual explanations but to answer questions about actual program execution using rewards derived from running the code and checking each execution step. ExecVerify is presented as a concrete instantiation of this idea: it synthesizes code problems with controlled difficulty, uses white-box access to the interpreter to construct verifiable stepwise rewards over control flow and data flow, and applies a two-stage RL pipeline that first improves execution reasoning and then transfers those gains to code generation [2603.11226]. ## 1. RL-Exec as a reinforcement-learning formulation ExecVerify treats the language model as a policy in an RL environment where each episode is a reasoning trace about executing a program. For the core Stage-I execution-reasoning setup, there is a synthesized program \(f\) and an input \(x\). Running \(f(x)\) with an instrumented interpreter yields an execution trace \[ \tau = \{(l_t, \sigma_t)\}_{t=1}^{T}, \] where \(l_t\) is the statement executed at step \(t\), and \(\sigma_t\) is the program state after executing that statement, including values and types of in-scope variables [2603.11226]. From the RL perspective, the environment state is encoded implicitly in the prompt: the program text, input(s), and a set of questions derived from the trace. The model observes the code \(f\), an I/O question, and a subset of white-box questions \(Q_s\), such as asking for the value and type of a variable after a given line executes, or asking which line executes immediately after a given line on its \(N\)-th execution. The model’s actions are the tokens of the completion: it first answers the I/O question and then answers the sampled white-box questions. The policy is the LLM \(\pi_\theta\), which defines a distribution over completions \(y\) given a prompt \(p\), \[ \pi_\theta(y \mid p). \] The Stage-I objective is the standard policy-gradient objective \[ J(\theta) = \mathbb{E}_{p \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid p)}[R(p,y)]. \] ExecVerify uses GRPO, described as a PPO-like policy gradient algorithm, with \(n=8\) rollouts per prompt, a scalar reward per rollout, and a KL term whose coefficient is set to \(0\) [2603.11226]. A common misconception is to equate RL-Exec with generic chain-of-thought tuning. In ExecVerify, the defining feature is not the presence of intermediate text, but the fact that intermediate execution claims are tied to interpreter-derived ground truth. This suggests that RL-Exec is best understood as semantic supervision over execution behavior rather than stylistic supervision over explanations. ## 2. Verifiable stepwise rewards and white-box execution The core RL-Exec reward in ExecVerify decomposes into final I/O correctness and intermediate execution correctness. For one completion, the white-box reward is \[ R_{\text{white-box}} = 2 \cdot \left((1-\alpha)\, R^{(\mathrm{I}\rightarrow \mathrm{O})} + \alpha\, R_{\text{white}}\right), \] where \(R^{(\mathrm{I}\rightarrow \mathrm{O})} \in \{0,1\}\) is a binary reward for final output correctness, \(R_{\text{white}} \in [0,1]\) is the average accuracy over sampled white-box questions, and \(\alpha=0.5\) is the default mixing coefficient. The factor \(2\) scales the reward range to \([0,2]\) [2603.11226]. White-box questions are constructed from the full execution trace gathered by the instrumented Python interpreter. ExecVerify uses two categories. **Control-flow questions** test next-statement prediction, for instance asking which line is executed immediately after line \(k\) is executed for the \(N\)-th time. The answer must be the exact source code of the next line, including indentation, but without line number or extra text. **Data-flow questions** test variable value and type tracking, asking for the value and type of a variable after a given line execution, with answers in the strict format `value; type`, such as `1; int` or `[1, 2]; list` [2603.11226]. For each program-input pair, all eligible white-box questions are generated from the trace, and training samples up to \(10\) questions per instance to form \(Q_s\). The white-box component is then \[ R_{\text{white}} = \frac{1}{|Q_s|} \sum_{q_j \in Q_s} \mathbb{I}[a_j = a_j^{*}], \] where \(a_j^{*}\) is the exact ground-truth answer extracted from the trace. Because every question is derived from actual execution, every answer is uniquely verifiable [2603.11226]. For O\(\rightarrow\)I tasks, where the model predicts an input that yields a given output, ExecVerify does not define white-box questions because many inputs may be valid. Instead, if the predicted input produces the target output when executed, the reward is \(2\); otherwise it is \(0\). This keeps the reward range consistent with the I\(\rightarrow\)O plus white-box case [2603.11226]. A second misconception is that RL-Exec must assign rewards along the token timeline. ExecVerify instead uses a single episodic reward whose decomposition is semantic rather than temporal: final I/O correctness plus an aggregate over many control-flow and data-flow micro-decisions. This suggests a reward design in which execution semantics, not token position, define the granularity of feedback. ## 3. Constraint-based synthesis and difficulty-controlled training data ExecVerify builds its environment through constraint-based program synthesis rather than passive data mining. It synthesizes Python code snippets and inputs with structural constraints on built-in types, methods, and control-flow patterns, then filters these instances by execution validity and empirical difficulty [2603.11226]. The synthesis process iterates over built-in Python types and methods and prompts QwQ-32B to generate code under explicit constraints. These include method-call constraints, such as requiring nested method calls or multiple distinct methods within a single function, and control-structure constraints, such as requiring a `for` statement, nesting an `if` statement inside the loop, and, at later levels, adding `while` and deeper nesting. The curriculum levels include: Level 1 with a single built-in method call and no control flow; Level 2 with multiple method calls and at least one shallow `if` or `for` without nested blocks; and Level 3 with multiple method calls and nested control structures with max depth at least \(3\). The final dataset is described as being dominated by level-3-style snippets [2603.11226]. Each synthesized function is paired with concrete inputs. An LLM first generates an initial input, often as an `assert` call, and then type-aware input mutation is applied. The mutation prompt provides candidate integers and strings, encourages longer strings and containers, and encourages recombinations of reference values. This yields multiple unique inputs per code snippet and diversifies execution behaviors [2603.11226]. Filtering proceeds in two passes. First, execution-based filtering removes code-input pairs with runtime errors, timeouts, or huge outputs. Second, difficulty-based filtering uses Qwen2.5-Coder-7B-Instruct as a difficulty probe: each remaining instance is run \(10\) times with temperature \(1.0\) on the I\(\rightarrow\)O task, and only instances with \(k \le 3\) correct predictions are kept. The resulting dataset contains \(119{,}358\) examples [2603.11226]. This dataset design is central to the RL-Exec formulation because it avoids both saturated rewards and degenerate all-zero rewards. The paper states that RL sees tasks whose difficulty is not too low and not too high, and that the structural constraints guarantee varied control-flow and data-flow patterns. A plausible implication is that difficulty control and white-box reward design are coupled components rather than separable engineering details. ## 4. Two-stage training pipeline and quantitative performance ExecVerify uses a two-stage post-training strategy. **Stage I** is RL-Exec for execution reasoning, and **Stage II** is RL for code generation using unit-test rewards. Stage I itself has two sub-phases: a supervised warm-up and then white-box RL [2603.11226]. The Stage-I warm-up performs full-parameter SFT on roughly \(30\)K I/O reasoning traces generated by QwQ-32B and filtered by rejection sampling to keep only traces leading to correct outputs. The SFT objective is standard cross-entropy: \[ \mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(p,y^\star)} \sum_t \log \pi_\theta(y_t^\star \mid y_{