Potential-Based Reward Shaping in RL

Updated 26 February 2026

Potential-Based Reward Shaping (PBRS) is a reinforcement learning technique that augments rewards with a heuristic potential function to improve learning efficiency.
It employs a difference-of-potentials method that maintains the telescoping property, ensuring that the set of optimal policies remains invariant.
PBRS reduces sample complexity and enhances exploration in sparse reward environments by supporting both hand-engineered and automated potential functions.

Potential-Based Reward Shaping

Potential-based reward shaping (PBRS) is a formal class of reward augmentation in reinforcement learning (RL) that allows the injection of heuristic prior knowledge through an auxiliary term derived from a scalar potential function, while guaranteeing invariance of the optimal policy. Introduced by Ng, Harada, and Russell (1999), PBRS accelerates learning, improves exploration, and reduces sample complexity in both discrete and continuous domains—especially in environments with sparse or delayed reward signals—without introducing suboptimal policy bias when the shaping signal is aligned with the proven potential-difference form.

1. Formalism and Policy Invariance

Let $(S, A, P, R, \gamma)$ denote an MDP with states $S$ , actions $A$ , transition kernel $P$ , reward $R(s,a,s')$ , and discount $\gamma \in [0,1)$ . Potential-based shaping adds to $R$ the difference-of-potentials term: $F(s, a, s') = \gamma\,\Phi(s') - \Phi(s)$ where $\Phi: S \to \mathbb{R}$ is the shaping potential.

The agent learns under the modified reward: $R'(s, a, s') = R(s, a, s') + F(s, a, s')$

Ng et al. (1999) established the policy invariance theorem: for any bounded $\Phi$ , the set of optimal stationary policies in the shaped MDP is identical to that of the base MDP. This is due to the telescoping nature of the sum of $F$ along any trajectory, which adds only a boundary term $-\Phi(s_0) + \gamma^T\Phi(s_T)$ , not depending on actions, to the total return, thus preserving the ordering of $Q$ -values for action selection (Yang et al., 2021, Cooke et al., 2023, Harutyunyan et al., 2015, Jeon et al., 2023, Forbes et al., 2024).

2. Construction and Instantiation of Potential Functions

Heuristic Potentials

Classic practice employs hand-engineered heuristics as $\Phi$ ; e.g., negative shortest-path distance to goal in grid worlds, solution length from Sokoban A* search $d(s)$ via $\Phi(s) = -d(s)$ , or distance-to-target in robotics (Yang et al., 2021, Malysheva et al., 2020).

Data-Driven and Automated Potentials

Several modern methods construct $\Phi$ automatically:

Abstraction-based: Compute the optimal value function of an abstracted MDP (e.g., over rooms, tiles, macro-states), use as $\Phi(s) = V^*(\alpha(s))$ (Canonaco et al., 2024).
Bootstrapped/Adaptive: Use the agent's current value estimate as $\Phi$ (e.g., $Q$ -max), updated online (Adamczyk et al., 2 Jan 2025, Chen et al., 2024).
Representation Learning: Learn $\Phi$ with a neural network via message passing, GCNs, or convolutional planners from sampled transitions (Sami et al., 2022, Klissarov et al., 2020).
Demonstration-based: Encode similarity to video-based demonstrations as a potential via inverse distance in pose-space (Malysheva et al., 2020).
Subgoal Aggregation: Use human-provided or algorithmically computed subgoals in an episodic/sequential order; $\Phi$ is incremented each time a subgoal is achieved (Okudo et al., 2021, Okudo et al., 2021).

Specializations

State-action Potentials: Generalizations allow $\Phi: S \times A \to \mathbb{R}$ , preserving optimality under analogous telescoping arguments (Xiao et al., 2019).
Dynamic and History-Dependent Potentials: PBRS can extend to history-dependent or dynamically learned $\Phi$ if boundary conditions ensuring telescoping are enforced (Forbes et al., 2024).

3. Algorithms and Integration with RL Methods

In discrete tabular and deep RL (e.g., DQN, PPO, A2C, DDPG), integrating PBRS requires replacing the environment reward $r$ with $r' = r + F$ at each step. The potential is computed as a function of state (and/or action, history), possibly using approximators or learned models. For actor-critic methods, the shaping term can be incorporated into the reward signal for both actor and critic updates (Malysheva et al., 2020, Chen et al., 2024).

Algorithmic sketch for PBRS integration:

s = current_state
a = select_action(s)
s_next, r = env.step(a)
f = gamma * Phi(s_next) - Phi(s)
r_shaped = r + f
update_Q_or_policy(s, a, r_shaped, s_next)

For stochastic-policy or actor-critic settings, additional correction terms may be needed to ensure unbiased gradient estimation (Xiao et al., 2019).

4. Impact on Learning Efficiency and Empirical Results

PBRS has been empirically shown to yield orders-of-magnitude reductions in sample complexity relative to unshaped baselines, especially in sparse-reward regimes:

Sokoban: 4× speedup in one-box cases, with essentially no learning in unshaped two-box or three-box domains, but rapid convergence ( $\lesssim 50$ k steps) when shaped (Yang et al., 2021).
Atari and Arcade Learning Environment: Bootstrapped shaping with learned potentials accelerates DQN training by $\sim$ 45% (early) and $\sim$ 60% (final score, median across 40 games) (Adamczyk et al., 2 Jan 2025).
Humanoid locomotion: PBRS reduces variance over runs and is robust to reward term scaling, whereas standard (additive) shaping causes overfitting or performance collapse outside a very small tuning window (Jeon et al., 2023).
IRL subproblems: Potential-based shaping with planning-aware potentials makes inner RL routines more sample efficient, reducing effective planning horizon (Cooke et al., 2023).
Ensemble PBRS: Learning multiple policies under diverse heuristics and scales, voting yields consistently strong performance without any additional environment samples (Harutyunyan et al., 2015).

Sample efficiency gains depend critically on the quality and relevance of $\Phi$ , the impact of finite-horizon truncation (if present), and proper handling of boundary conditions.

5. Extensions: Intrinsic Motivation, Action-Dependence, and Meta-RL

Classical PBRS requires the cumulative shaping reward to be independent of the agent’s actions to guarantee optimality preservation. This restriction prevents naive integration of many intrinsic motivation (IM) signals (e.g., curiosity, count-based bonuses). Recent work has generalized PBRS theory to:

General Potential-based Intrinsic Motivation (PBIM): Convert arbitrary action-independent IM signals into a PBRS-equivalent via episode-wise boundary compensations, preserving optimal policy (Forbes et al., 2024).
Action-Dependent Shaping (ADOPS): Extends PBRS to allow action-dependent shaping while establishing strong optimality preservation by explicitly constructing correction terms; shown to overcome reward-correction explosion in highly sparse, long-horizon Atari environments (e.g., Montezuma's Revenge) where PBRS/GRM/PIES fail (2505.12611).
BAMDP Shaping: Unifies PBRS and intrinsic motivation as potential-based shaping over Bayes-Adaptive MDPs (BAMDPs), showing that policies maximizing BAMDP value under PBRS cannot be reward-hacked and are immune to suboptimal IM-driven fixations (e.g., "noisy-TV" pathology) (Lidayan et al., 2024).

These extensions formalize the safe use of IM in meta-RL and complex environments.

6. Practical Considerations, Hyperparameters, and Limitations

Scale and bias: The effect of PBRS is sensitive to the scale and offset of $\Phi$ relative to the external reward and the initial $Q$ -values. Adding a linear shift to $\Phi$ can drastically improve its effectiveness without altering encoded preferences or requiring re-initialization of $Q$ , as shown in (2502.01307). Overly aggressive scaling may cause the agent to over-prioritize shaping.

Finite-horizon bias: In finite-horizon settings, PBRS can introduce bias through boundary terms (e.g., $\gamma^H\Phi(s_H)$ ). For goal-oriented episodic MDPs with bounded horizon, ordering of optimal policies is preserved; otherwise, explicit horizon-dependent bounds are required (Canonaco et al., 2024, 2502.01307).

Quality of potential: Poorly chosen or misaligned potentials can slow learning or, if continuous, "invert" incentives over small transitions. Exponentially scaled or discretized $\Phi$ can mitigate such sign reversals (2502.01307).

Computational cost: Computing search-based or abstraction-based potentials can be expensive for large or high-dimensional domains. Approximate, learned, or neural $\Phi$ can scale PBRS to these cases (Sami et al., 2022, Klissarov et al., 2020, Malysheva et al., 2020).

7. Applications and Empirical Variants

PBRS has been successfully deployed across:

Planning and puzzle domains: Sokoban (A* path-length heuristics) (Yang et al., 2021), classic gridworlds (distance-to-goal or rooms abstraction) (Canonaco et al., 2024).
Robotics: Hierarchical shaping for multivariate task specifications, including safety, targets, and comfort objectives in F1TENTH and lunar-lander (Berducci et al., 2021); humanoid running with demonstration-based or classic physics-based potentials (Malysheva et al., 2020, Jeon et al., 2023).
Ensemble RL: Multi-potential PBRS for robust shaping signal aggregation (Ensemble Horde) (Harutyunyan et al., 2015).
Deep RL and meta-RL: Dynamic/bootstrapped potentials and potential-based intrinsic motivation (PBIM) for intrinsic signal conversion (Adamczyk et al., 2 Jan 2025, Forbes et al., 2024).

Common patterns: shaping is most beneficial in domains where extrinsic rewards are sparse, delayed, or have long credit-assignment paths; in dense- or low-dimensional domains, PBRS mainly improves robustness and scaling, sometimes exhibiting only marginal speed-up (Jeon et al., 2023).

Key citations: (Yang et al., 2021, Cooke et al., 2023, Harutyunyan et al., 2015, Jeon et al., 2023, Forbes et al., 2024, 2505.12611, Malysheva et al., 2020, Klissarov et al., 2020, Sami et al., 2022, Berducci et al., 2021, Adamczyk et al., 2 Jan 2025, Xiao et al., 2019, 2502.01307, Canonaco et al., 2024, Okudo et al., 2021, Okudo et al., 2021, Lidayan et al., 2024, Chen et al., 2024, Badnava et al., 2019).