Temporal-Difference Reward Shaping

• Temporal-Difference Reward Shaping is a reinforcement learning technique that augments rewards with a potential function to accelerate convergence and preserve optimal policy.
• It employs various potential designs—including handcrafted, learned, bootstrapped, and ensemble methods—to modify TD errors and align learning dynamics.
• Empirical results demonstrate accelerated learning and improved sample efficiency across diverse domains such as Atari, robotics, and language model tuning.

Temporal-difference (TD) reward shaping refers to a family of techniques in reinforcement learning (RL) that leverage temporal-difference reward modifications—most notably, potential-based reward shaping—to accelerate learning and improve exploration, while preserving policy optimality. These methods systematically inject auxiliary information, either from heuristics, learned potentials, or auxiliary models, into the TD learning process via systematic shaping terms, typically exploiting the temporal structure of Markov decision processes.

1. Core Principles of Potential-Based TD Reward Shaping

Temporal-difference reward shaping is fundamentally grounded in the potential-based shaping framework introduced by Ng, Harada, and Russell (1999). For an MDP $(S, A, P, R, \gamma)$, with standard TD learning based on the update: $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$ potential-based shaping augments the reward with a difference of a real-valued potential function $\Phi : S \to \mathbb{R}$, yielding a shaped reward: $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$ This shaping leaves the optimal policy invariant, since the sum of shaping terms along any trajectory telescopes, only modifying the value function by a state-dependent offset. In the context of TD learning, the TD-error is augmented with the shaping term, thereby directly influencing updates and transient learning dynamics (Adamczyk et al., 2 Jan 2025, Bordelon et al., 2023, Harutyunyan et al., 2014, Okudo et al., 2021).

2. Learning Dynamics and Policy Invariance

The introduction of a TD-based shaping term modifies value function estimation without altering the optimal policy set. When function approximation is linear, as in Bordelon et al. (Bordelon et al., 2023), the potential $\Phi(s)=\psi(s)^\top w_\phi$ (where $\psi$ denotes features) shifts the fixed point of the TD update: $\tilde V(s) = V(s) + \Phi(s)$ for the shaped value function. The Bellman operator becomes: $$\tilde T[V](s) = \mathbb{E}_{s'} [ R(s) + \gamma \Phi(s') ] + \gamma V(s') - \Phi(s)$$ The direction of the fixed point is shifted by $w_\phi$, and the learning plateaus and transient timescales can be favorably affected if $w_\phi$ is chosen to align with fast eigenspaces of the feature covariance operator $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$0 (Bordelon et al., 2023). Shaping itself does not change the asymptotic reducible error plateau but can accelerate convergence by reorienting the target direction in function space.

3. Design and Construction of Shaping Potentials

The construction of the shaping potential $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$1 can be knowledge-driven, learned, or bootstrapped. Manual design often leverages domain priors—e.g., distance to goal, system energy—but empirical and theoretical results indicate that merely scaling or uniformly inflating rewards by $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$2 does not improve learning unless the added structure aligns well with favourable learning dynamics (Bordelon et al., 2023). Tools to construct or learn $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$3 include:

• TTR-Based Potentials: Time-to-reach functions $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$4, computed on low-dimensional surrogates of system dynamics and inverted into potentials $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$5 for robotic control tasks (Lyu et al., 2019).
• Bootstrapped Value Potentials (BSRS): The agent’s own running state value estimate $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$6 or Q-value maximum at iteration $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$7 is used for shaping, yielding an entirely dynamic, policy-dependent shaping term (Adamczyk et al., 2 Jan 2025).
• Subgoal Aggregation: Abstract value functions $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$8 are learned over dynamically constructed abstract states corresponding to human-specified or automatically identified subgoals, updating the shaping function during learning (Okudo et al., 2021).
• Ensembles of Potentials: Multiple domain heuristics are encoded as parallel shaping potentials $$\delta_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t),$$9, with their Q-functions evaluated off-policy and combined by action-rank voting (Harutyunyan et al., 2014).

Shaping Approach	Mechanism	Key References
Handcrafted Potentials	$\Phi : S \to \mathbb{R}$0 designed from priors	(Lyu et al., 2019, Harutyunyan et al., 2014)
Learned Potentials	Abstract V via SARSA-RS, subgoals	(Okudo et al., 2021)
Bootstrapped Potentials	$\Phi : S \to \mathbb{R}$1 from running value/Q	(Adamczyk et al., 2 Jan 2025)
Shaping Ensembles	Multiple $\Phi : S \to \mathbb{R}$2 per domain heuristic	(Harutyunyan et al., 2014)

4. Empirical and Theoretical Effects on Learning

TD reward shaping consistently yields improvements in sample efficiency, early-stage exploration, and occasionally asymptotic performance. Specific highlights include:

• TTR Potential-Shaping: Two to fourfold reduction in episodes required for success and up to 20% higher asymptotic return in robotic navigation/stabilization benchmarks, with the shaped reward form $\Phi : S \to \mathbb{R}$3 operating plug-and-play with any model-free RL method (Lyu et al., 2019).
• Bootstrapped Reward Shaping (BSRS): Median human-normalized performance on Atari improved by 45–60% at intermediate times; more than half the tested games show >10% speedup, and a significant subset sees >100% gains (Adamczyk et al., 2 Jan 2025). The theoretical policy invariance holds, but empirical stability depends on the noise characteristics of the value estimator.
• Dynamic Aggregation via Subgoals: Human-informed subgoal potential functions, updated through n-step TD over sub-episodes, yield up to 1.5x–2x faster learning compared with random or naïve shaping in both discrete and continuous domains, while maintaining final policy equivalence (Okudo et al., 2021).
• Off-Policy Shaping Ensembles: Ensembles yield safe, policy-invariant learning and empirical speedup, leveraging the diversity of domain-knowledge potentials. Under the GTD (Gradient TD) framework, the learning of multiple Q-functions with shaped rewards converges off-policy, and combined policies outperform or match the best constituent in each learning phase (Harutyunyan et al., 2014).
• Learning Dynamics: The primary gain is in transient dynamics, especially the reduction of noise-induced plateaus and more favorable spectral alignment, not asymptotic error magnitude (Bordelon et al., 2023).

5. Extensions, Variations, and Limitations

Several recent directions extend classical TD shaping:

• Bootstrapped and Dynamic Potentials: The uniqueness of BSRS is in its dynamic, iteration-dependent potential—proven not to be reproducible by a static surrogate. Convergence is preserved in the tabular case for admissible scaling $\Phi : S \to \mathbb{R}$4 (shape-scale), with more aggressive values possible in deep RL (Adamczyk et al., 2 Jan 2025).
• Temporal-Difference Regularized Reward Models: In LLM RL, process reward models (PRMs) trained with an explicit TD regularizer $\Phi : S \to \mathbb{R}$5 yield smoother, more temporally stable reward assignments, improving actor-critic stability and data efficiency. When combined with outcome-based verifiable rewards, data requirements are reduced 20-fold, with improved accuracy for LLM policies (Zhang et al., 18 Sep 2025).
• Subgoal-Based and Abstract-State Potentials: SARSA-RS, and its extensions with on-the-fly subgoal aggregation, support the formulation of shaping potentials where explicit state aggregation is infeasible, crucial in high-dimensional settings (Okudo et al., 2021).
• Theoretical Constraints: All variants based on the telescoping potential-difference structure retain optimal policy invariance. Limitations include the requirement to select or design effective potentials, tuning schedule or scale parameters (e.g., $\Phi : S \to \mathbb{R}$6 in BSRS), and potential oscillatory effects if the bootstrapped estimator is noisy (Adamczyk et al., 2 Jan 2025, Okudo et al., 2021).

6. Practical Implementation and Computational Considerations

TD reward shaping is typically easy to integrate: the original reward returned by the environment is replaced by its shaped counterpart in the TD or policy gradient update. For instance, in DQN: $\Phi : S \to \mathbb{R}$7 yields a one-line code change for BSRS (Adamczyk et al., 2 Jan 2025). Computation of potentials can be costly (e.g., solving HJB for TTR potentials), but is typically a one-time offline process, with online incorporation negligible in cost (Lyu et al., 2019). For learned or bootstrapped potentials, overhead is minimal if the estimator is already in place.

Empirical guidelines suggest that:

• Bootstrapped potentials can be scaled as $\Phi : S \to \mathbb{R}$8, with admissible $\Phi : S \to \mathbb{R}$9 given by $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$0, though deep RL can tolerate more aggressive scaling (Adamczyk et al., 2 Jan 2025).
• For spectral efficiency, $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$1 should be designed (data-permitting) to align $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$2 with low-index eigenvectors of the Bellman operator’s feature covariance—yielding faster timescales for error reduction (Bordelon et al., 2023).
• Ensembles and dynamic abstract state approaches may require parallel value learning and auxiliary state tracking, but these are parallelizable and compatible with off-policy architectures (Harutyunyan et al., 2014, Okudo et al., 2021).

7. Representative Results and Impact

TD reward shaping has delivered substantial empirical gains across high-dimensional control, Atari benchmarks, and LLM policy alignment. The following table summarizes principal quantitative effects:

Domain/Method	Performance Gain	Reference
Kinematic car/quadrotor (TTR)	2–4× episode reduction, +10-20% return	(Lyu et al., 2019)
Atari (BSRS, $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$3)	45–60% improvement in median score at $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$41.5M frames	(Adamczyk et al., 2 Jan 2025)
Subgoal HRS (Four-Rooms)	33 vs. 52 episodes to threshold (HRS vs. Baseline)	(Okudo et al., 2021)
LLM RL (TDRM)	$$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$54–6% accuracy in Best-of-$$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$6; $$r'(s, a, s') = r(s, a, s') + \gamma \Phi(s') - \Phi(s)$$7 reduction in required samples	(Zhang et al., 18 Sep 2025)

A recurring theme is that potential-based TD shaping drives down effective delay in credit assignment, enabling reward information to propagate more quickly across state space and improving early-stage exploration without sacrificing long-term policy quality.

---

References:

• TTR-based reward shaping (Lyu et al., 2019)
• Bootstrapped reward shaping (Adamczyk et al., 2 Jan 2025)
• Shaping ensembles and off-policy TD (Harutyunyan et al., 2014)
• Subgoal aggregation and reward shaping (Okudo et al., 2021)
• Statistical mechanics of TD learning and shaping (Bordelon et al., 2023)
• Temporal difference regularized reward models in RLHF (Zhang et al., 18 Sep 2025)