Potential-Based Reward Shaping in RL

• Potential-based reward shaping is a method that adds an auxiliary reward derived from a potential function to accelerate policy learning in sparse environments while preserving optimality.
• It leverages the telescoping property of the potential function, ensuring that the adjusted rewards maintain the original policy ordering in Markov Decision Processes.
• Empirical results in domains like Sokoban and humanoid locomotion demonstrate significant improvements in sample efficiency and robustness to reward scaling.

Potential-based reward shaping (PBRS) is a mathematically principled mechanism in reinforcement learning (RL) for accelerating policy learning in sparse or challenging reward environments, by leveraging an auxiliary shaping reward derived from a "potential" function over the state space. The central feature of PBRS is its guarantee—formally proved by Ng, Harada, and Russell (1999)—that if the shaping reward is constructed appropriately, the set of optimal policies remains unchanged, i.e., policy invariance is preserved. This allows the designer to inject auxiliary guidance signal and densify feedback to the agent, without introducing bias in the final solution. PBRS is broadly applicable, has a clear algebraic form, can be extended or composed with automated and learned potentials, and provides a theoretical backbone for advanced shaping strategies in both classical and deep RL.

1. Mathematical Foundations of PBRS

Potential-based reward shaping operates in Markov Decision Processes (MDPs), $M = (S, A, P, R, \gamma)$, where $S$ is the state space, $A$ the action set, $P$ the transition kernel, $R$ the reward function, and $\gamma\in[0,1)$ is the discount factor. The agent seeks a policy $\pi$ that maximizes expected discounted return: $$G = \mathbb{E} \left[ \sum_{t=0}^\infty \gamma^t R(s_t, a_t, s_{t+1}) \right]\,.$$

PBRS augments the environment reward by a shaping term $F$, generated from a "potential" function $\Phi: S \to \mathbb{R}$, with the canonical form: $F(s,a,s') = \gamma \Phi(s') - \Phi(s)\,.$ The agent instead sees the shaped reward

$R'(s,a,s') = R(s,a,s') + F(s,a,s')\,.$

The policy invariance theorem guarantees that for any bounded $\Phi$, the optimal policy for $R$ is also optimal for $R'$; for all policies $\pi$,

$Q_{R'}^\pi(s,a) = Q_R^\pi(s,a) - \Phi(s)\,,$

so $$\arg\max_a Q_{R'}^\pi(s,a) = \arg\max_a Q_R^\pi(s,a)$$. This extends to finite-horizon, stochastic, and even certain history-dependent settings (Yang et al., 2021, Canonaco et al., 11 Apr 2024, Xiao et al., 2019).

2. Design of Potential Functions

The effectiveness of PBRS depends on selecting a potential $\Phi$ that provides meaningful gradient information towards high-density regions of the optimal policy, often using domain knowledge, heuristics, planning algorithms, or automatically extracted environmental structure. A canonical form is

$\Phi(s) = - d^*(s)$

where $d^*(s)$ is an admissible estimate of the distance-to-goal. In "Potential-based Reward Shaping in Sokoban," $\Phi$ is instantiated as the negative length of the shortest A*-planned path from $s$ to a solved state (Yang et al., 2021).

For high-dimensional systems, such as humanoid locomotion, hand-crafted terms like orientation, height, or joint symmetry are used as $\Phi$, and incorporated via

$$F_c(s,s') = \gamma \Phi_c(s') - \Phi_c(s), \quad c \in \{\text{ori}, h, j\}$$

In hierarchical settings, composite potentials aggregate prioritized predicates over multiple safety, target, and comfort requirements using multiplicative-then-additive structures, as in HPRS (Berducci et al., 2021).

Potential functions can also be learned end-to-end from sampled transitions, for instance via Graph Convolutional Networks to propagate reward signals over transition graphs (Klissarov et al., 2020), or through dynamic trajectory aggregation where $\Phi$ evolves as a value proxy over an abstracted subgoal sequence (Okudo et al., 2021).

Automatically designing $\Phi$ from abstract MDPs, graph-based approximations, or using value estimates (e.g., bootstraping from the agent's own $V$ or $Q$ function as in Bootstrapped Reward Shaping (Adamczyk et al., 2 Jan 2025)) are active research directions.

3. Empirical Performance and Sample Efficiency

PBRS has repeatedly demonstrated ability to improve sample efficiency in RL tasks, especially those characterized by sparse and delayed rewards. In Sokoban, reward shaping based on A* distance yields an order-of-magnitude acceleration in solved fraction, with one-box levels seeing a $4\times$ speedup and two/three-box tasks going from 20% to 100% solve rates within an order of magnitude fewer steps (Yang et al., 2021).

For high-dimensional locomotion, PBRS yields smaller speedup but is markedly more robust to scaling of shaping terms compared to naively additive shaping. In humanoid tasks, PBRS preserved or slightly improved final performance, halved the empirical return variance, and maintained stability under $0.1\times$ to $10\times$ scaling of the shaping weights (Jeon et al., 2023).

Table: Highlights of Empirical Results (Yang et al., 2021, Jeon et al., 2023)

Domain	Baseline Episodes to Mastery	PBRS Episodes	Speedup	Variance Reduction
Sokoban 1 box	~60k	~15k	$4\times$	—
Sokoban 2/3 box	Stagnates at $<20\%$	40-50k	$\to100\%$	—
Humanoid	350	275	marginal	$2\times$ (AUC var)

The robustness of PBRS to reward scaling, agent initialization, and environment stochasticity is highlighted by its stability in continuous control and complex robotics (Jeon et al., 2023, Malysheva et al., 2020).

4. Advanced Extensions and Generalizations

Beyond classical PBRS, several extensions have been proposed to handle more expressive shaping, preserve theoretical guarantees with non-Markovian or action-dependent shaping, or facilitate learning with intrinsic or context-dependent rewards.

• Hierarchical PBRS: HPRS encodes multiple task requirements as compositional potentials with strict partial ordering. This supports multivariate, lexicographically prioritized shaping while maintaining policy invariance (Berducci et al., 2021).
• Bootstrapped and Learning-based Potentials: BSRS dynamically sets the potential to the agent's ongoing value estimate, providing shaping signals adaptable to the agent's knowledge state; this accelerates credit assignment and promotes explorative learning behavior (Adamczyk et al., 2 Jan 2025).
• PBRS for Intrinsic Motivation: PBIM converts generic intrinsic motivation signals into a potential-based form, preserving optimality while supporting complex or learned shaping signals. Variants such as Generalized Reward Matching (GRM) and Action-Dependent Optimality-Preserving Shaping (ADOPS) further extend to action-dependent and sequence-dependent pseudo-rewards while securing policy invariance (Forbes et al., 12 Feb 2024, 2505.12611).
• State Abstraction and Meta-RL Extensions: PBRS is lifted into BAMDPs (Bayes-Adaptive MDPs), establishing criteria for potential-based shaping over belief histories ("BAMPFs") to guard against reward hacking and ensure invariance at the meta-RL and meta-exploration level (Lidayan et al., 9 Sep 2024).
• Ensembles and Off-policy Evaluation: Maintaining ensembles of PBRS agents over multiple heuristics and shaping scales, as in the Horde architecture, allows automated tuning and hedging, crucial in settings where the optimal potential is unknown (Harutyunyan et al., 2015).

5. Implementation Practices and Practical Guidelines

Several implementation strategies are consistently validated across domains:

• Compute or approximate a "cost-to-go" metric (distance-to-goal, abstract value, etc.) as the potential; for complex domains, approximate by search, state abstraction, or learning.
• Use $F(s,a,s') = \gamma \Phi(s') - \Phi(s)$ exactly and consistently with the main RL discount factor.
• When deploying dense, hand-engineered or learned state-action reward terms, prefer converting them to PBRS form rather than direct addition; this preserves robustness to scaling and avoids biasing the policy (Jeon et al., 2023).
• If using function approximation or deep RL, consider learning the potential online (via GCNs, deep value networks, or bootstrapped critics) and ensure boundedness to prevent policy bias due to numeric instability.
• In finite-horizon or episodic RL, be aware of un-telecoped end-of-episode terms (e.g. $\gamma^{H}\Phi(s_H)$), and apply corrections where necessary to ensure policy ordering is preserved (Canonaco et al., 11 Apr 2024).
• Ensure $\Phi$ is set to $0$ in absorbing or goal states for correct return calculation.

Common pitfalls include improper initialization of $Q$-values relative to the potential scale, overscaled or sign-inconsistent shaping terms, and using non-potential-based shaping that violates the telescoping property and thus can bias the optimal policy (2502.01307). In continuous or high-dimensional domains, discretized or function-approximated $\Phi$ must still respect these invariants.

6. Open Problems and Research Directions

Despite the maturity of PBRS theory, several open challenges remain:

• Potential Construction: Automated or data-efficient construction of "good" potentials for complex, high-dimensional, or real-world environments is an ongoing research area, encompassing planning heuristics, graph-based methods, and on-policy bootstrapping (Klissarov et al., 2020, Adamczyk et al., 2 Jan 2025).
• Shaping for Intrinsic Motivation and Meta-Learning: Ensuring shaping does not introduce reward hacking, particularly with complex learned intrinsic bonuses, requires advanced potential-based or action-dependent frameworks (e.g. BAMPF, ADOPS) that go beyond classical PBRS form (Lidayan et al., 9 Sep 2024, 2505.12611).
• Robustness to Horizon and Implementation Bias: PBRS can express nontrivial bias if finite horizons are not properly managed; bounding potential ranges and applying terminal corrections is essential (Canonaco et al., 11 Apr 2024).
• Empirical Scaling: Though policy invariance is proved, practical improvements in sample efficiency are sensitive to the agent's initialization, environment reward structure, and potential scaling (2502.01307).

Potential-based reward shaping remains a foundational tool in the RL practitioner's toolkit, providing theoretically sound, implementation-flexible mechanisms for accelerating and stabilizing policy search, especially in sparse or structured domains (Yang et al., 2021, Jeon et al., 2023, Canonaco et al., 11 Apr 2024, Berducci et al., 2021, Adamczyk et al., 2 Jan 2025).