Gated Reward Accumulation in RL

• Gated Reward Accumulation (G-RA) is a hierarchical reward processing method in RL that gates immediate rewards based on achieving high-level outcomes to prevent reward hacking.
• It employs a gating mechanism with a threshold (δ) that conditions the accumulation of dense rewards, balancing between sparse outcome rewards and immediate signals.
• Empirical evaluations in long-horizon software engineering tasks demonstrate significant improvements in completion and modification rates compared to naive reward accumulation approaches.

Gated Reward Accumulation (G-RA) is a reward processing method for reinforcement learning (RL) that addresses the challenge of reward sparsity and misalignment by allowing the accumulation of immediate (low-level) rewards only when high-level (long-term) outcome rewards cross a specified threshold. Originally developed in the context of long-horizon, multi-turn software engineering (SWE) tasks, G-RA provides a principled alternative to both naive dense shaping and sparse-only reward specification, preventing policy degradation due to reward hacking and enabling stable RL optimization across extended training (Sun et al., 14 Aug 2025).

1. Problem Formulation in Multi-level RL

The setting for G-RA is a finite-horizon Markov Decision Process (MDP) where trajectories comprise multi-turn agent-environment interaction, such as in automated software engineering tasks. The agent's state at each step encodes the dialogue history, repository state, and other metadata. Actions are structured as text commands mapped onto a set of discrete scaffolds (e.g., shell, editor, web search, submit). The maximum allowed number of turns per episode is fixed (typically $T=15$). Rewards are hierarchical with two levels:

• High-level (outcome) reward $R^{(H)}$: assigned only at the end of the episode (after "submit"); discrete values encode outcome: $R^{(H)} \in \{-2, -1, 0, +10\}$, corresponding to "no submit," "empty patch," "patch fails tests," and "patch passes tests" respectively.
• Low-level (immediate) rewards $r^{(L)}(t)$: assigned by stepwise critics—such as correct output format or valid use of scaffolds—provide dense signal.

Standard RL maximizes the expected return $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[ \sum_{t=1}^T r(t) \right]$$, but direct accumulation of both $R^{(H)}$ and $\sum_t r^{(L)}(t)$ exposes the learning process to spurious incentives, as policies may exploit $r^{(L)}$ without improving $R^{(H)}$, leading to suboptimal behavior ("reward hacking") (Sun et al., 14 Aug 2025).

2. Mathematical Formulation of Gated Reward Accumulation

G-RA introduces a gating mechanism based on the high-level reward. Let $\delta$ be a gating threshold (typically $\delta=0$):

• For a given trajectory $\tau$, let $R^{(H)}(\tau)$ be the final high-level outcome reward.
• Each immediate reward is $r^{(L)}(t)$ for $1 \leq t < T$.

The G-RA accumulated reward is defined as:

$$R_{\mathrm{GRA}}(\tau) = R^{(H)}(\tau) + \sum_{t=1}^{T-1} \mathbb{I}[R^{(H)}(\tau) \geq \delta] \cdot r^{(L)}(t)$$

where $\mathbb{I}[\cdot]$ is the indicator function.

This mechanism can be generalized for $n$-level reward hierarchies $\{R^{(i)}\}$ with integer priorities $o^{(i)}$ and per-reward gates $g_v^{(i)}$:

$$R^{(i)}_{\text{gated}}(s,a) = \begin{cases} R^{(i)}(s,a) & \text{if } R^{(j)}(s,a) \geq g_v^{(j)},\ o^{(i)}<o^{(j)} \ 0 & \text{otherwise} \end{cases}$$

The total per-step reward becomes the sum over gated components. In most practical cases, the two-level scheme suffices (Sun et al., 14 Aug 2025).

3. Algorithmic Workflow and Pseudocode

G-RA is implemented as a simple modification to the reward accumulation logic in training loops compatible with standard policy-gradient or actor-critic RL algorithms. The pseudocode is:

Algorithm: Gated Reward Accumulation (G-RA) Inputs: π_θ: policy network δ: gating threshold (e.g., 0) N: batch size T: max trajectory length RL optimizer hyperparameters Repeat until convergence: 1. Roll out N trajectories under π_θ: - At each step t: π_θ selects action aₜ; environment yields sₜ₊₁ - At terminal (T) or submit, assign R^{(H)}(τ) - At each t, compute r^{(L)}(t) 2. For each trajectory τᵢ, compute return: R_GRA(τ_i) ← R^{(H)}(τ_i) + 𝟙[R^{(H)}(τ_i) ≥ δ] ⋅ ∑_{t=1}^{T−1} r^{(L)}(t) 3. Use R_GRA as return in RL update (GRPO, PPO, A2C, etc.) and update θ

There is no modification to the environment or action interface; the gating is purely implemented within the reward module (Sun et al., 14 Aug 2025).

4. Selection and Tuning of Gate Threshold

The gating threshold $\delta$ controls the balance between purely sparse and purely dense reward regimes. $\delta \to +\infty$ recovers fully sparse rewards (only the outcome matters); $\delta \to -\infty$ gives full access to all shaping rewards (ungated). Empirically, $\delta$ should be just above the highest negative outcome reward (here, $\delta=0$ permits reward shaping only on outcomes of test-fail or better). Theoretical motivation follows from reward transformation invariance (Ng et al., 1999): only improvements to $R^{(H)}$ unlock access to shaping signals, thus eliminating incentive for "reward hacking" on $r^{(L)}$. Choosing appropriate $\delta$ is task-dependent; improper settings can either stifle learning (if set too high) or revert to unstable shaping (if too low) (Sun et al., 14 Aug 2025).

5. Empirical Evaluation and Benchmarking

G-RA was evaluated on SWE-bench Verified (500 validated GitHub issues) and kBench-50 (50 Linux syzbot crash issues). The environment is a dockerized Linux sandbox. Action scaffolds comprise Shell, Editor, WebSearch, and Submit. The model backbone is Qwen2.5-3B-Instruct, with 2 epochs of SFT on 3,000 Deepseek-V3 rollouts, and RL with GRPO for up to 100 steps, on 4×A100 GPUs with a 16k token context. Three key evaluation metrics are used:

• Completion Rate (CR): percentage of episodes invoking Submit
• Modification Rate (MR): percentage of episodes with code edits before Submit
• Resolution Rate (RR): percentage of episodes with a patch that passes tests

Table: Empirical Outcomes for Main Methods (Sun et al., 14 Aug 2025)

Method	SWE-bench Verified CR/MR/RR (%)	kBench-50 CR/MR/RR (%)
SFT	47.6 / 19.6 / 0.2	22.0 / 12.0 / 0.0
D-RA @25 steps	19.4 / 6.6 / 0.0	16.0 / 12.0 / 0.0
G-RA @25 steps	67.0 / 23.8 / 0.2	36.0 / 20.0 / 0.0
G-RA @75 steps	93.8 / 22.4 / 0.2	86.0 / 42.0 / 0.0

Key findings are:

• D-RA, which directly sums high-level and immediate rewards, results in rapid collapse (CR below 2% at 100 steps).
• G-RA produces monotonic improvement in both completion and modification rates, with near doubling of CR on SWE-bench (47.6% to 93.8%) and a fourfold increase on kBench (22.0% to 86.0%).
• RR remains low due to task difficulty but is not degraded by G-RA.
• Ablations confirm $\delta=0$ as optimal; $\delta$ too low leads to instability, $\delta$ too high impedes reward shaping.

6. Implementation Considerations, Limitations, and Extensions

Implementation of G-RA is limited to a change in the reward computation; no modifications are required for environment logic or policy architecture. Multi-level gating is supported by assigning per-critic priorities and gates. G-RA is agnostic to the underlying RL algorithm (compatible with policy-gradient and actor-critic classes).

Limitations include:

• Strict gating can delay useful learning signal, particularly if $R^{(H)}$ is very hard to achieve.
• Uniform thresholds may be suboptimal for diverse or heterogeneous task families.

Potential avenues for extension include:

• Soft gating: Replacing the hard indicator with a continuous gate $\sigma(R^{(H)} - \delta)$.
• Adaptive threshold scheduling: Starting with a permissive threshold and annealing towards the target.
• Hierarchical RL: Integrating G-RA as subgoal-level reward control.
• Meta-learning $\delta$: Optimizing gate thresholds across task classes via cross-validation or Bayesian optimization.

A plausible implication is that such extensions could further balance exploration-exploitation in settings where outcome rewards are extremely sparse or variable (Sun et al., 14 Aug 2025).

7. Significance and Context in Long-horizon RL

Gated Reward Accumulation offers a principled approach to the longstanding problem of aligning dense shaping rewards with sparse ultimate objectives. In SWE and other complex, long-horizon environments, it mitigates degenerate learning dynamics and stabilizes RL over extended optimization (Sun et al., 14 Aug 2025). Its simplicity and compatibility with existing RL frameworks facilitate integration, while empirical results highlight notable improvements in both sample efficiency and policy robustness. The gating criterion directly enforces outcome-conditioned credit assignment, a desideratum frequently identified in hierarchical and subgoal-oriented RL research.