---
title: Potential-Based Reward Shaping in RL
url: https://www.emergentmind.com/topics/potential-based-reward-shaping-0682587b-ca8b-4aa5-9644-2510c9516c24
type: topic
---

# Potential-Based Reward Shaping in RL

Potential-Based Reward Shaping

Potential-based reward shaping (PBRS) is a formal class of reward augmentation in reinforcement learning (RL) that allows the injection of heuristic prior knowledge through an auxiliary term derived from a scalar potential function, while guaranteeing invariance of the optimal policy. Introduced by Ng, Harada, and Russell (1999), PBRS accelerates learning, improves exploration, and reduces sample complexity in both discrete and continuous domains—especially in environments with sparse or delayed reward signals—without introducing suboptimal policy bias when the shaping signal is aligned with the proven potential-difference form.

## 1. Formalism and Policy Invariance

Let $(S, A, P, R, \gamma)$ denote an MDP with states $S$, actions $A$, transition kernel $P$, reward $R(s,a,s')$, and discount $\gamma \in [0,1)$. Potential-based shaping adds to $R$ the difference-of-potentials term:
\[
F(s, a, s') = \gamma\,\Phi(s') - \Phi(s)
\]
where $\Phi: S \to \mathbb{R}$ is the shaping potential.

The agent learns under the modified reward:
\[
R'(s, a, s') = R(s, a, s') + F(s, a, s')
\]

Ng et al. (1999) established the policy invariance theorem: for any bounded $\Phi$, the set of optimal stationary policies in the shaped MDP is identical to that of the base MDP. This is due to the telescoping nature of the sum of $F$ along any trajectory, which adds only a boundary term $-\Phi(s_0) + \gamma^T\Phi(s_T)$, not depending on actions, to the total return, thus preserving the ordering of $Q$-values for action selection [2109.05022][2312.09983][1502.03248][2307.10142][2402.07411].

## 2. Construction and Instantiation of Potential Functions

### Heuristic Potentials

Classic practice employs hand-engineered heuristics as $\Phi$; e.g., negative shortest-path distance to goal in grid worlds, solution length from Sokoban A* search $d(s)$ via $\Phi(s) = -d(s)$, or distance-to-target in robotics [2109.05022][2012.08824].

### Data-Driven and Automated Potentials

Several modern methods construct $\Phi$ automatically:
- **Abstraction-based:** Compute the optimal value function of an abstracted MDP (e.g., over rooms, tiles, macro-states), use as $\Phi(s) = V^*(\alpha(s))$ [2404.07826].
- **Bootstrapped/Adaptive:** Use the agent's current value estimate as $\Phi$ (e.g., $Q$-max), updated online [2501.00989][2402.04581].
- **Representation Learning:** Learn $\Phi$ with a neural network via message passing, GCNs, or convolutional planners from sampled transitions [2210.16956][2010.02474].
- **Demonstration-based:** Encode similarity to video-based demonstrations as a potential via inverse distance in pose-space [2012.08824].
- **Subgoal Aggregation:** Use human-provided or algorithmically computed subgoals in an episodic/sequential order; $\Phi$ is incremented each time a subgoal is achieved [2104.06411][2104.06163].

### Specializations

- **State-action Potentials:** Generalizations allow $\Phi: S \times A \to \mathbb{R}$, preserving optimality under analogous telescoping arguments [1907.08823].
- **Dynamic and History-Dependent Potentials:** PBRS can extend to history-dependent or dynamically learned $\Phi$ if boundary conditions ensuring telescoping are enforced [2402.07411].

## 3. Algorithms and Integration with RL Methods

In discrete tabular and deep RL (e.g., DQN, PPO, A2C, DDPG), integrating PBRS requires replacing the environment reward $r$ with $r' = r + F$ at each step. The potential is computed as a function of state (and/or action, history), possibly using approximators or learned models. For actor-critic methods, the shaping term can be incorporated into the reward signal for both actor and critic updates [2012.08824][2402.04581].

Algorithmic sketch for PBRS integration:
```python
# Pseudocode for a single RL step under PBRS
s = current_state
a = select_action(s)
s_next, r = env.step(a)
f = gamma * Phi(s_next) - Phi(s)
r_shaped = r + f
update_Q_or_policy(s, a, r_shaped, s_next)
```
For stochastic-policy or actor-critic settings, additional correction terms may be needed to ensure unbiased gradient estimation [1907.08823].

## 4. Impact on Learning Efficiency and Empirical Results

PBRS has been empirically shown to yield orders-of-magnitude reductions in sample complexity relative to unshaped baselines, especially in sparse-reward regimes:
- **Sokoban:** 4× speedup in one-box cases, with essentially no learning in unshaped two-box or three-box domains, but rapid convergence ($\lesssim 50$k steps) when shaped [2109.05022].
- **Atari and Arcade Learning Environment:** Bootstrapped shaping with learned potentials accelerates DQN training by $\sim$45% (early) and $\sim$60% (final score, median across 40 games) [2501.00989].
- **Humanoid locomotion:** PBRS reduces variance over runs and is robust to reward term scaling, whereas standard (additive) shaping causes overfitting or performance collapse outside a very small tuning window [2307.10142].
- **IRL subproblems:** Potential-based shaping with planning-aware potentials makes inner RL routines more sample efficient, reducing effective planning horizon [2312.09983].
- **Ensemble PBRS:** Learning multiple policies under diverse heuristics and scales, voting yields consistently strong performance without any additional environment samples [1502.03248].

Sample efficiency gains depend critically on the quality and relevance of $\Phi$, the impact of finite-horizon truncation (if present), and proper handling of boundary conditions.

## 5. Extensions: Intrinsic Motivation, Action-Dependence, and Meta-RL

Classical PBRS requires the cumulative shaping reward to be independent of the agent’s actions to guarantee optimality preservation. This restriction prevents naive integration of many intrinsic motivation (IM) signals (e.g., curiosity, count-based bonuses). Recent work has generalized PBRS theory to:
- **General Potential-based Intrinsic Motivation (PBIM):** Convert arbitrary *action-independent* IM signals into a PBRS-equivalent via episode-wise boundary compensations, preserving optimal policy [2402.07411].
- **Action-Dependent Shaping (ADOPS):** Extends PBRS to allow action-dependent shaping while establishing strong optimality preservation by explicitly constructing correction terms; shown to overcome reward-correction explosion in highly sparse, long-horizon Atari environments (e.g., Montezuma's Revenge) where PBRS/GRM/PIES fail [2505.12611].
- **BAMDP Shaping:** Unifies PBRS and intrinsic motivation as *potential-based shaping over Bayes-Adaptive MDPs* (BAMDPs), showing that policies maximizing BAMDP value under PBRS cannot be reward-hacked and are immune to suboptimal IM-driven fixations (e.g., "noisy-TV" pathology) [2409.05358].

These extensions formalize the safe use of IM in meta-RL and complex environments.

## 6. Practical Considerations, Hyperparameters, and Limitations

**Scale and bias:** The effect of PBRS is sensitive to the scale and offset of $\Phi$ relative to the external reward and the initial $Q$-values. Adding a linear shift to $\Phi$ can drastically improve its effectiveness without altering encoded preferences or requiring re-initialization of $Q$, as shown in [2502.01307]. Overly aggressive scaling may cause the agent to over-prioritize shaping.

**Finite-horizon bias:** In finite-horizon settings, PBRS can introduce bias through boundary terms (e.g., $\gamma^H\Phi(s_H)$). For goal-oriented episodic MDPs with bounded horizon, ordering of optimal policies is preserved; otherwise, explicit horizon-dependent bounds are required [2404.07826][2502.01307].

**Quality of potential:** Poorly chosen or misaligned potentials can slow learning or, if continuous, "invert" incentives over small transitions. Exponentially scaled or discretized $\Phi$ can mitigate such sign reversals [2502.01307].

**Computational cost:** Computing search-based or abstraction-based potentials can be expensive for large or high-dimensional domains. Approximate, learned, or neural $\Phi$ can scale PBRS to these cases [2210.16956][2010.02474][2012.08824].

## 7. Applications and Empirical Variants

PBRS has been successfully deployed across:
- **Planning and puzzle domains:** Sokoban (A* path-length heuristics) [2109.05022], classic gridworlds (distance-to-goal or rooms abstraction) [2404.07826].
- **Robotics:** Hierarchical shaping for multivariate task specifications, including safety, targets, and comfort objectives in F1TENTH and lunar-lander [2110.02792]; humanoid running with demonstration-based or classic physics-based potentials [2012.08824][2307.10142].
- **Ensemble RL:** Multi-potential PBRS for robust shaping signal aggregation (Ensemble Horde) [1502.03248].
- **Deep RL and meta-RL:** Dynamic/bootstrapped potentials and potential-based intrinsic motivation (PBIM) for intrinsic signal conversion [2501.00989][2402.07411].

Common patterns: shaping is most beneficial in domains where extrinsic rewards are sparse, delayed, or have long credit-assignment paths; in dense- or low-dimensional domains, PBRS mainly improves robustness and scaling, sometimes exhibiting only marginal speed-up [2307.10142].

---

Key citations: [2109.05022], [2312.09983], [1502.03248], [2307.10142], [2402.07411], [2505.12611], [2012.08824], [2010.02474], [2210.16956], [2110.02792], [2501.00989], [1907.08823], [2502.01307], [2404.07826], [2104.06411], [2104.06163], [2409.05358], [2402.04581], [1902.06239].

Source: https://www.emergentmind.com/topics/potential-based-reward-shaping-0682587b-ca8b-4aa5-9644-2510c9516c24