CodeRL+: Reinforcement Learning for Code Generation

• CodeRL+ is a methodology that integrates reinforcement learning with variable-level execution semantics to improve both the syntactic and functional correctness of generated code.
• It leverages dense, verifiable rewards from supervising final variable values in code execution, ensuring enhanced training signals and robust learning.
• Empirical evaluations show that CodeRL+ delivers state-of-the-art improvements across benchmarks and effectively stabilizes various RL algorithms and LLM architectures.

CodeRL+ refers to a suite of methodologies for improving code generation in LLMs by integrating reinforcement learning (RL) with explicit alignment to program execution semantics. The primary objective of CodeRL+ is to bridge the gap between text-based code synthesis—where models learn to mimic surface-level syntax—and actual functional correctness as validated by runtime behavior. The approach enhances conventional RL with verifiable rewards by introducing variable-level execution trajectory alignment, yielding dense learning signals, generalization across tasks, and robustness across diverse RL algorithms and LLM architectures (Jiang et al., 21 Oct 2025).

1. Execution Semantics Alignment

CodeRL+ directly targets the semantic discrepancy between textual code representations and executable behavior by incorporating variable-level execution semantics into the RL training loop. Instead of solely relying on sparse, binary pass/fail rewards obtained from unit test execution, CodeRL+ supervises the model to predict the final assigned value for every program variable after execution.

Formally, given a program $p$ with variables $V = \{{var}_1, {var}_2, \ldots, {var}_n\}$ and input $x$, execution alignment is defined as:

$$\hat{\mathcal{f}_p(x)} = \pi_\theta(p, x) \approx \mathcal{f}_p(x) = \{ {var}_i \mapsto v_{t_{\text{last}}}^i \mid {var}_i \in V \}$$

where $t_{\text{last}}^i$ is the maximal timestep at which variable $var_i$ is defined. This intermediate supervision enables dense feedback by inferring the variable-level execution trajectory (i.e., associating every program variable with its final value at the last assignment point), without exhaustive reconstruction of the entire state trajectory, which is impractical for programs with complex control flow.

This alignment is applied directly to failed rollouts—generated samples that do not pass all test cases—and extracts ground truth variable endpoints by executing code on input examples. The resulting alignment promotes learning of control flow, data dependencies, and semantic structure beyond mere surface syntax.

2. Integration with Reinforcement Learning Algorithms

CodeRL+ is designed to be algorithm-agnostic and integrates seamlessly with contemporary on-policy RL variants such as Group Relative Policy Optimization (GRPO), Proximal Policy Optimization (PPO), and REINFORCE++. During training, generated samples (“rollouts”) are divided into two objectives: one for maximizing end-task rewards (e.g., pass@1 via test execution), and one for maximizing agreement with execution semantics alignment.

The composite training objective is:

$$\mathcal{J}_{\text{CodeRL}^+}(\theta) = \mathbb{E}_{q \sim \mathcal{B}_{\text{code}}, p \sim \pi_\theta}[r(\theta)\cdot A_{\text{gen}}] + \mathbb{E}_{q' \sim \mathcal{B}_{\text{align}}, s \sim \pi_\theta}[r'(\theta)\cdot A_{\text{sem}}]$$

where $r(\theta)$, $r'(\theta)$ are importance sampling ratios, and $A_{\text{gen}}$, $A_{\text{sem}}$ are group-normalized advantage estimates for generation and semantic alignment objectives, respectively. All RL algorithms benefit from additional dense semantic alignment, with PPO notably exhibiting substantial performance gains and improved training robustness.

3. Variable-Level Execution Trajectory Supervision

The central methodological advancement of CodeRL+ lies in inferring, extracting, and using variable-level execution trajectories for training supervision. Ground-truth execution traces, computed for failed rollouts by running code on test inputs, yield for each variable the value at its last assignment within the program’s state sequence. This explicit supervision encourages the model to internalize logical properties and dependencies among variables, enhancing its understanding of program semantics and disambiguating subtle logical errors that binary pass/fail checking cannot adequately penalize.

Notably, this scheme circumvents the combinatorial explosion that full state trajectory annotation would induce for deeply nested loops or recursive functions. Instead, it leverages a compact yet functionally rich summary of execution via final variable values, enabling efficient and effective alignment.

4. Experimental Validation and Benchmarks

Extensive experiments demonstrate that CodeRL+ produces state-of-the-art results across diverse tasks. On canonical code generation benchmarks such as HumanEval, LeetCode, and LiveCodeBench, CodeRL+ achieves an average relative improvement of 4.6% in pass@1 compared to RLVR and distillation baselines. For example, on HumanEval, CodeRL+ raises pass@1 rates from approximately 88.4% to 90.9%.

On code reasoning and test-output generation tasks, CodeRL+ yields 15.5% and 4.4% higher accuracy, respectively. Training dynamics figures and evaluation curves indicate increasing divergence in final performance between CodeRL+ and standard RLVR as training proceeds, demonstrating that semantic alignment confers lasting benefits.

Probe analyses, using linear regression on hidden states to predict the final value of program variables, reveal that CodeRL+ representations consistently achieve lower mean squared error (MSE) than baseline or GRPO-trained variants, confirming that hidden activations encode richer, more precise execution semantic information.

5. Generalization to Code Reasoning and Test Output Generation

CodeRL+ generalizes robustly beyond pure code completion. On code reasoning benchmarks (e.g., LiveCodeBench-Reason) and test output generation tasks (LiveCodeBench-Test), the framework’s dense supervision allows models to predict not only observable program outputs but also internal variable states. This capability is critical for tools requiring precise internal analysis and for debugging or automated test-case inference.

Reinforcement via execution semantics alignment ensures reliable transfer to tasks demanding deeper structural and behavioral understanding, contrasting with prior approaches that overfit to surface-level pattern memorization.

6. Applicability Across Model Architectures and RL Algorithms

Empirical studies establish that CodeRL+ yields consistent improvements irrespective of LLM family or model size. Tested architectures include LLaMA-3.1-8B-Instruct, Qwen2.5-Coder-7B-Instruct, and Qwen2.5-Coder-1.5B, among others. CodeRL+ not only improves code generation, reasoning, and test output metrics, but also stabilizes training dynamics—especially where standard RL (e.g., GRPO) exhibits instability or degradation.

This suggests that variable-centric supervision and semantic alignment serve as robust regularizers, mitigating training variance and providing a reliable learning scaffold, especially for models susceptible to overfitting or reward collapse.

7. Implications and Future Research Trajectories

The innovations introduced by CodeRL+ mark a significant advance in aligning LLM code generation with true program semantics. By supervising variable-level execution trajectories, integrating dense semantic rewards into RL, and demonstrating consistent improvements in accuracy, reasoning, and robustness, CodeRL+ sets a benchmark for future code synthesis systems.

A plausible implication is that CodeRL+ will be increasingly adopted in settings demanding rigorous semantic fidelity, such as automated software testing, code review, and program verification. Further research may explore richer trajectory extraction methods, hybrid reward schemes leveraging symbolic and statistical signals, or integration with static analysis and formal verification pipelines.

In summary, CodeRL+ is a methodology for equipping code-generating LLMs with functional correctness via execution semantics alignment in reinforcement learning. Its demonstrated efficacy across models, algorithms, and tasks positions it as a foundational technique for next-generation code synthesis and reasoning systems (Jiang et al., 21 Oct 2025).