RL-Exec: Reinforcement Learning for Code Reasoning

• RL-Exec is a reinforcement learning framework for execution-based code reasoning that uses white-box rewards derived from interpreter traces to verify program execution.
• It employs constraint-driven program synthesis and difficulty filtering to generate diverse, verifiable code examples for robust training.
• A two-stage training pipeline, combining supervised warm-up with white-box RL and unit-test rewards, improves both reasoning and code generation performance.

Searching arXiv for the cited papers and closely related usages of “RL-Exec.” RL-Exec, in the sense crystallized by ExecVerify, denotes reinforcement learning for execution-based code reasoning, in which a LLM is trained not merely to imitate textual explanations but to answer questions about actual program execution using rewards derived from running the code and checking each execution step. ExecVerify is presented as a concrete instantiation of this idea: it synthesizes code problems with controlled difficulty, uses white-box access to the interpreter to construct verifiable stepwise rewards over control flow and data flow, and applies a two-stage RL pipeline that first improves execution reasoning and then transfers those gains to code generation (Tang et al., 11 Mar 2026).

1. RL-Exec as a reinforcement-learning formulation

ExecVerify treats the LLM as a policy in an RL environment where each episode is a reasoning trace about executing a program. For the core Stage-I execution-reasoning setup, there is a synthesized program $f$ and an input $x$. Running $f(x)$ with an instrumented interpreter yields an execution trace

$\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$

where $l_t$ is the statement executed at step $t$, and $\sigma_t$ is the program state after executing that statement, including values and types of in-scope variables (Tang et al., 11 Mar 2026).

From the RL perspective, the environment state is encoded implicitly in the prompt: the program text, input(s), and a set of questions derived from the trace. The model observes the code $f$, an I/O question, and a subset of white-box questions $Q_s$, such as asking for the value and type of a variable after a given line executes, or asking which line executes immediately after a given line on its $N$-th execution. The model’s actions are the tokens of the completion: it first answers the I/O question and then answers the sampled white-box questions. The policy is the LLM $x$0, which defines a distribution over completions $x$1 given a prompt $x$2,

$x$3

The Stage-I objective is the standard policy-gradient objective

$x$4

ExecVerify uses GRPO, described as a PPO-like policy gradient algorithm, with $x$5 rollouts per prompt, a scalar reward per rollout, and a KL term whose coefficient is set to $x$6 (Tang et al., 11 Mar 2026).

A common misconception is to equate RL-Exec with generic chain-of-thought tuning. In ExecVerify, the defining feature is not the presence of intermediate text, but the fact that intermediate execution claims are tied to interpreter-derived ground truth. This suggests that RL-Exec is best understood as semantic supervision over execution behavior rather than stylistic supervision over explanations.

2. Verifiable stepwise rewards and white-box execution

The core RL-Exec reward in ExecVerify decomposes into final I/O correctness and intermediate execution correctness. For one completion, the white-box reward is

$x$7

where $x$8 is a binary reward for final output correctness, $x$9 is the average accuracy over sampled white-box questions, and $f(x)$0 is the default mixing coefficient. The factor $f(x)$1 scales the reward range to $f(x)$2 (Tang et al., 11 Mar 2026).

White-box questions are constructed from the full execution trace gathered by the instrumented Python interpreter. ExecVerify uses two categories. Control-flow questions test next-statement prediction, for instance asking which line is executed immediately after line $f(x)$3 is executed for the $f(x)$4-th time. The answer must be the exact source code of the next line, including indentation, but without line number or extra text. Data-flow questions test variable value and type tracking, asking for the value and type of a variable after a given line execution, with answers in the strict format value; type, such as 1; int or [1, 2]; list (Tang et al., 11 Mar 2026).

For each program-input pair, all eligible white-box questions are generated from the trace, and training samples up to $f(x)$5 questions per instance to form $f(x)$6. The white-box component is then

$f(x)$7

where $f(x)$8 is the exact ground-truth answer extracted from the trace. Because every question is derived from actual execution, every answer is uniquely verifiable (Tang et al., 11 Mar 2026).

For O$f(x)$9I tasks, where the model predicts an input that yields a given output, ExecVerify does not define white-box questions because many inputs may be valid. Instead, if the predicted input produces the target output when executed, the reward is $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$0; otherwise it is $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$1. This keeps the reward range consistent with the I$\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$2O plus white-box case (Tang et al., 11 Mar 2026).

A second misconception is that RL-Exec must assign rewards along the token timeline. ExecVerify instead uses a single episodic reward whose decomposition is semantic rather than temporal: final I/O correctness plus an aggregate over many control-flow and data-flow micro-decisions. This suggests a reward design in which execution semantics, not token position, define the granularity of feedback.

3. Constraint-based synthesis and difficulty-controlled training data

ExecVerify builds its environment through constraint-based program synthesis rather than passive data mining. It synthesizes Python code snippets and inputs with structural constraints on built-in types, methods, and control-flow patterns, then filters these instances by execution validity and empirical difficulty (Tang et al., 11 Mar 2026).

The synthesis process iterates over built-in Python types and methods and prompts QwQ-32B to generate code under explicit constraints. These include method-call constraints, such as requiring nested method calls or multiple distinct methods within a single function, and control-structure constraints, such as requiring a for statement, nesting an if statement inside the loop, and, at later levels, adding while and deeper nesting. The curriculum levels include: Level 1 with a single built-in method call and no control flow; Level 2 with multiple method calls and at least one shallow if or for without nested blocks; and Level 3 with multiple method calls and nested control structures with max depth at least $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$3. The final dataset is described as being dominated by level-3-style snippets (Tang et al., 11 Mar 2026).

Each synthesized function is paired with concrete inputs. An LLM first generates an initial input, often as an assert call, and then type-aware input mutation is applied. The mutation prompt provides candidate integers and strings, encourages longer strings and containers, and encourages recombinations of reference values. This yields multiple unique inputs per code snippet and diversifies execution behaviors (Tang et al., 11 Mar 2026).

Filtering proceeds in two passes. First, execution-based filtering removes code-input pairs with runtime errors, timeouts, or huge outputs. Second, difficulty-based filtering uses Qwen2.5-Coder-7B-Instruct as a difficulty probe: each remaining instance is run $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$4 times with temperature $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$5 on the I$\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$6O task, and only instances with $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$7 correct predictions are kept. The resulting dataset contains $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$8 examples (Tang et al., 11 Mar 2026).

This dataset design is central to the RL-Exec formulation because it avoids both saturated rewards and degenerate all-zero rewards. The paper states that RL sees tasks whose difficulty is not too low and not too high, and that the structural constraints guarantee varied control-flow and data-flow patterns. A plausible implication is that difficulty control and white-box reward design are coupled components rather than separable engineering details.

4. Two-stage training pipeline and quantitative performance

ExecVerify uses a two-stage post-training strategy. Stage I is RL-Exec for execution reasoning, and Stage II is RL for code generation using unit-test rewards. Stage I itself has two sub-phases: a supervised warm-up and then white-box RL (Tang et al., 11 Mar 2026).

The Stage-I warm-up performs full-parameter SFT on roughly $\tau = \{(l_t, \sigma_t)\}_{t=1}^{T},$9K I/O reasoning traces generated by QwQ-32B and filtered by rejection sampling to keep only traces leading to correct outputs. The SFT objective is standard cross-entropy: $l_t$0 After that warm-up, ExecVerify applies white-box RL on another $l_t$1K synthesized examples. For each prompt it generates $l_t$2 rollouts, computes $l_t$3 and $l_t$4, combines them into $l_t$5, and updates the policy with GRPO. In the reported configuration, the KL coefficient is $l_t$6 (Tang et al., 11 Mar 2026).

Stage II starts from the reasoning-enhanced model and optimizes code generation with unit-test rewards under a Process Reinforcement setup. For each problem, the model generates candidate code, unit tests are executed, and the generation reward is

$l_t$7

GRPO is again used for optimization (Tang et al., 11 Mar 2026).

On code reasoning benchmarks, the reported average scores are $l_t$8 for the base Qwen2.5-Coder-7B-Instruct model, $l_t$9 for I/O RL only, $t$0 for SFT warm-up plus I/O RL, and $t$1 for SFT plus white-box RL, which is the full ExecVerify-7B configuration. The paper states that this $t$2 score is competitive with Qwen2.5-Coder-32B-Instruct at $t$3. It also reports CRUXEval-O improving from $t$4 to $t$5, CRUXEval-I from $t$6 to $t$7, LiveCodeBench-Exec from $t$8 to $t$9, REval-State from $\sigma_t$0 to $\sigma_t$1, and REval-Path from $\sigma_t$2 to $\sigma_t$3 (Tang et al., 11 Mar 2026).

On code generation benchmarks, the reported averages are $\sigma_t$4 for the base $\sigma_t$5B model, $\sigma_t$6 for unit-test RL only, $\sigma_t$7 for I/O RL plus unit-test RL, $\sigma_t$8 for SFT plus I/O RL plus unit-test RL, and $\sigma_t$9 for SFT plus white-box RL plus unit-test RL. The paper describes this as up to $f$0 points over unit-test RL and notes improvements such as LiveCodeBench-Medium from $f$1 to $f$2 and BigCodeBench-Hard from $f$3 to $f$4 (Tang et al., 11 Mar 2026).

The ablation on question types reports that CF-only and DF-only each improve their own area, but the full CF+DF configuration achieves the highest average, $f$5 versus $f$6 and $f$7. The paper also states that, with a fixed budget of $f$8K training examples, SFT on ExecVerify’s synthesized dataset outperforms SFT on SEMCODER’s PYX, CODEI/O’s data, and Grounded-CoT traces on CRUXEval-O and LiveCodeBench-Exec, and that constraints, input mutation, and difficulty filtering each contribute measurable gains (Tang et al., 11 Mar 2026).

5. Relation to prior execution training and semantic alignment

ExecVerify is positioned against prior execution-aware methods that are primarily SFT-based. The paper identifies two broad forms of teacher-generated supervision: I/O prediction chains, as in SEMCODER and CODEI/O, and natural-language descriptions of execution traces, as in TracePile and “Code Execution as Grounded Supervision.” All of these optimize token-level cross-entropy over teacher text (Tang et al., 11 Mar 2026).

The paper identifies two drawbacks of that regime. First, intermediate steps are not explicitly verified: a teacher explanation may be approximate, partial, or wrong in intermediate steps, and SFT still trains the model to match it. Second, there is an objective mismatch: the model is optimized to look like the teacher’s text, not to be semantically correct at each execution step. A model can therefore match generic explanatory patterns without accurately tracking variable states or branches (Tang et al., 11 Mar 2026).

RL-Exec, as instantiated by ExecVerify, replaces textual imitation with interpreter-grounded supervision. The interpreter’s trace is the oracle; next-statement execution and variable value/type are objective facts; and the reward checks exact correctness of those facts. The paper explicitly characterizes this as a tighter alignment between the learning objective and the target behavior of faithfully simulating execution. It also argues that this setup provides dense and fine-grained feedback because the model may answer many white-box questions for each program-input pair and can receive partial credit even when final I/O is wrong (Tang et al., 11 Mar 2026).

This framing addresses another common misconception: that execution reasoning is adequately supervised by natural-language trace descriptions. ExecVerify argues instead that verifiability is the decisive property. A plausible implication is that, for execution-sensitive tasks, the distinction between textual process supervision and executable semantic supervision is more consequential than the distinction between SFT and RL taken in isolation.

6. Broader extensions and terminological scope

Within the supplied research record, ExecVerify is the clearest instance in which RL-Exec names reinforcement learning for execution-based code reasoning. The same record also applies an RL-Exec lens to several adjacent settings in which an RL-trained executor, rollout system, or tool-using model is optimized against execution-grounded signals, but it also includes a fully distinct use of the exact label “RL-Exec” for opportunistic optimal liquidation on BTC-USD limit-order books (Tang et al., 11 Mar 2026).

In code and tool use, the RL-Exec reading extends naturally to execution-grounded LLM systems. R2IF treats function calling as an RL problem over reasoning plus tool-call outputs and uses a composite reward combining format and correctness constraints, Chain-of-Thought Effectiveness Reward, and Specification-Modification-Value reward, all optimized with GRPO (Cheng et al., 22 Apr 2026). ProRL Agent externalizes multi-turn agent rollouts behind a rollout-as-a-service API, with standardized sandbox environments and an init → run → eval lifecycle, and is presented as a scalable infrastructure for RL training of agentic tasks in NVIDIA NeMo Gym (Zhang et al., 19 Mar 2026). TempAct casts autoregressive video diffusion as a planner-executor RL system in which an LLM planner proposes span-aware step prompts and the executor is optimized with local transition-level rewards for prompt-switch behavior and temporal plausibility (Wang et al., 26 Jun 2026).

At the same time, “RL-Exec” is also the title of a separate paper on opportunistic optimal liquidation over BTC-USD limit-order books. There it denotes a PPO agent trained in a replay-with-impact simulator with endogenous transient impact, partial fills, maker/taker fees, and latency, and evaluated against TWAP and a VWAP-like liquidity baseline. That usage is domain-specific and not about code execution reasoning (Duflot et al., 30 Oct 2025).

The broader pattern suggests that RL-Exec is best treated as a family resemblance term rather than a single standardized formalism. In its strongest and most explicit form, exemplified by ExecVerify, it denotes RL in which the environment exposes executable internal state and the reward is defined by semantic correctness of intermediate execution steps. Other papers in the record reuse the same intuition for tool execution, rollout orchestration, planner-executor generation, or decision systems, but the code-reasoning formulation remains the most technically explicit and the most tightly defined (Tang et al., 11 Mar 2026).