Policy-Invariant Reward Shaping (PBRS)

• Policy-Invariant Reward Shaping (PBRS) is a reward modification method that augments rewards with a potential difference to preserve the optimal policy in RL.
• It leverages a potential function such that adding the term γΦ(s′) − Φ(s) maintains policy invariance while enabling safe use of intrinsic rewards and expert guidance.
• PBRS is applied in continuous control, robotics, intrinsic motivation, and multi-agent settings to improve learning speed, sample efficiency, and reduce hyperparameter sensitivity.

Policy-Invariant Reward Shaping (PBRS) is a family of reward modification techniques in reinforcement learning (RL) and stochastic games that accelerates credit assignment without altering the set of optimal policies. Under PBRS, an agent’s reward at each step is augmented by a difference of a "potential function" across the current and next states (or, more generally, possibly other variables), ensuring all policies are ranked identically as in the original Markov Decision Process (MDP) or stochastic game. This guarantee enables the safe incorporation of domain knowledge, guidance from suboptimal demonstrations, or intrinsic rewards without incurring the "reward hacking" or policy-rotating pathologies that may arise from arbitrary reward shaping.

1. Formalism and Policy-Invariance Guarantee

Let $\mathcal{M}=(S, A, T, R, \gamma)$ denote an MDP with finite states $S$, actions $A$, Markovian transitions $T$, reward $R$, and discount $\gamma\in[0,1)$. A PBRS scheme defines a potential function $\Phi: S \rightarrow \mathbb{R}$ and a shaping term:

$F(s, s') = \gamma \Phi(s') - \Phi(s),$

which is added to every environment reward. The updated reward is:

$R'(s, a, s') = R(s, a, s') + F(s, s').$

The core theoretical result (Ng, Harada, Russell 1999) is that, for any policy $\pi$, the optimal Q-functions in the original and shaped environments are related as $Q^{\pi'}(s, a) = Q^\pi(s, a) + \Phi(s)$. Since $\Phi(s)$ is independent of $a$, $\arg\max_a Q^{\pi'}(s, a) = \arg\max_a Q^\pi(s, a)$, so the optimal policy set is preserved. This result extends to general-sum stochastic games, with independent potentials $\phi^i$ for each player, and preserves all Nash equilibria (Lu et al., 2014). Algorithmic implementation requires simply augmenting the per-step TD target in any value-based or policy-based method (Malysheva et al., 2020, Lu et al., 2014, Jeon et al., 2023).

2. Generalizations and Boundary Conditions

Classic PBRS assumes time/states-only potentials. Extensions allow for dynamic or non-Markovian shaping:

• For arbitrary (possibly non-Markov) potentials $\Phi_t$ that depend on histories or time, the shaping term becomes $F_t = \gamma \Phi_{t+1} - \Phi_t$.
• The policy-invariance condition is that the return bias $\mathbb{E}_\pi[\gamma^{N-t}\Phi_N - \Phi_t]$ must be independent of the current action $a_t$ ("boundary condition") (Forbes et al., 2024, Forbes et al., 2024).
• A notable case is with shaping derived from arbitrary intrinsic rewards $F_t$: transformation methods such as Potential-Based Intrinsic Motivation (PBIM) or Generalized Reward Matching (GRM) convert these into PBRS-compliant forms that preserve optimal policies even for non-Markovian or trajectory-dependent bonuses (Forbes et al., 2024, Forbes et al., 2024).

In partially observable, multi-agent, or general-sum settings, each agent’s shaping reward can be individualized but must comply with the PBRS form for policy/Nash equilibrium invariance (Lu et al., 2014).

3. Construction of the Potential Function

The informativeness and effectiveness of PBRS hinge on the quality of the potential $\Phi$. Several construction paradigms appear:

• Manual Design: For low-dimensional tasks, $\Phi$ is often chosen as (minus) distance to goal, or as a heuristic reflecting progress (e.g., in grid-worlds, maze navigation) (Lu et al., 2014, 2502.01307).
• Imitation/Data-Driven: In high-dimensional continuous control (e.g., humanoid locomotion), $\Phi(s)$ can be crafted from demonstration data, such as the inverse-squared error between agent and demonstration limb positions extracted from video (Malysheva et al., 2020). Per-part potentials are tested for empirical efficiency (e.g., $g(dx,dy) = 1/(dx^2+dy^2)$ for keypoints).
• Abstraction: In large state spaces, $\Phi$ can be obtained by solving an abstracted MDP (often much lower-dimensional), and "lifting" its value function to the original space (Canonaco et al., 2024).
• Bootstrapped (BSRS): The agent’s own current value function estimate can serve as $\Phi$ (possibly times a scale parameter), yielding an adaptive, task-agnostic shaping signal (Adamczyk et al., 2 Jan 2025).
• State-Action Potentials: For finer shaping, $\Phi$ may depend on both states and actions. Proper construction retains policy invariance (Behboudian et al., 2020).

A well-chosen potential correlates with the true value function, thereby yielding informative guiding signals. However, practical sample efficiency depends critically on aligning the scale and offsets of $\Phi$ with environment rewards and Q-initialization (2502.01307). Simple linear shifts of $\Phi$ can resolve otherwise misaligned initializations in deep RL.

4. Applications and Empirical Impact

PBRS finds application across RL domains:

• Continuous Control/Robotics: PBRS using video-derived or analytic shaping functions can dramatically accelerate learning in humanoid locomotion and robotic manipulation, increasing sample efficiency or final policy quality (Malysheva et al., 2020, Jeon et al., 2023, Tan et al., 29 Dec 2025).
• Intrinsic Motivation: PBRS provides a principled solution to safely deploying intrinsic rewards (curiosity, novelty, count-based exploration) without policy distortion. PBIM and GRM methods enable potential-based conversion for complex, possibly non-Markovian, intrinsic signals (Forbes et al., 2024, Forbes et al., 2024).
• Learning from Advice: Bandit-shaped Policy-Invariant Explicit Shaping (PIES) and related ensemble variants allow leveraging external or expert advice to accelerate learning while provably not undermining extrinsic optimality, including in the presence of adversarial or suboptimal guidance (Behboudian et al., 2020, Satsangi et al., 2023, Harutyunyan et al., 2015).
• Sample Efficiency/Variance Reduction: Abstraction-derived PBRS can yield performance matching highly tuned CNN architectures using orders of magnitude fewer interactions, and acts as a variance-reducing baseline in policy gradient methods (Canonaco et al., 2024).

Empirically, PBRS can double learning speeds (e.g., runner speed in humanoid—baseline 2.5 m/s to 5.0 m/s in 12 hours; higher final performance in ALE games with bootstrapped potentials; more reproducible, tuning-robust results in high-dimensional RL) (Malysheva et al., 2020, Adamczyk et al., 2 Jan 2025, Jeon et al., 2023).

5. Limitations and Remedies

While PBRS is policy-invariant by design, several practical complications arise:

• Finite-Horizon Bias: In finite-episode RL, the final potential term can bias policy ordering if episode horizons are short, especially in goal-directed tasks. However, with sufficiently long horizons, invariance approximately holds (Canonaco et al., 2024).
• Reward/Initialization Dependence: Efficiency is nontrivially influenced by the interaction of $\Phi$ with the Q-value initialization and external reward scale. Precise linear shifting of $\Phi$ can eliminate early mis-signs in Q-updates without policy impact (2502.01307).
• Potential Function Expressivity: Continuous linear potentials may fail to assign shaping with correct incentive signs for arbitrarily small state increments. Exponential potentials offer a remedy but complicate the bias structure (2502.01307).
• Long-Horizon Intrinsic Rewards: PBRS-based intrinsic shaping (including PBIM, GRM, PIES) can be ineffective or unstable in extremely sparse, long-duration domains (e.g., Montezuma’s Revenge), due to large boundary-correction terms. Action-dependent shaping schemes (ADOPS) circumvent this by allowing optimality-preserving, action-conditioned adjustments (2505.12611).

Robust implementation in deep RL often benefits from dynamic, scale-agnostic, or ensemble approaches to shaping design (Adamczyk et al., 2 Jan 2025, Harutyunyan et al., 2015).

6. Extensions and Recent Innovations

Recent PBRS research has expanded the methodology significantly:

• Multi-Agent and Stochastic Games: Extension of PBRS to general-sum stochastic games preserves Nash equilibria under separately defined potentials, enabling principled shaping in competitive or cooperative multi-agent RL (Lu et al., 2014).
• Intrinsic Motivation–Safe Transformation: The PBIM and GRM families yield plug-and-play conversion to PBRS form for arbitrary intrinsic signals, ensuring policy-equivalent exploration bonuses (Forbes et al., 2024, Forbes et al., 2024).
• Ensemble PBRS: Online maintenance of multiple heuristics and shaping scales, and adaptive voting among the resulting policies, achieves robust speedup without a priori tuning, even with off-policy architectures (Horde, Greedy-GQ(λ)) (Harutyunyan et al., 2015).
• Bandit-PIES and Advice Adaptation: Viewing the exploitation of advice as a sequential bandit problem allows the agent to adaptively allocate between unshaped and shaped learning arms, preserving policy invariance while exploiting high-quality guidance (Satsangi et al., 2023, Behboudian et al., 2020).
• Action-Dependent Shaping: ADOPS provides an optimality-preserving transformation for shaping rewards that cannot be written in potential-based form, including for complex, action-dependent, or cumulative intrinsic signals (2505.12611).

A summary of notable recent findings is as follows:

Class of Extension	Key Method	Invariance Ensured	Application Domain
Non-Markovian & Intrinsic	PBIM, GRM (Forbes et al., 2024, Forbes et al., 2024)	Yes, under boundary cond.	IM bonuses, curiosity
Ensemble Shaping	Horde, Off-Policy Ensemble (Harutyunyan et al., 2015)	Yes, by composition	Robust RL pipeline
Advice Safe-Shaping	PIES, Bandit-PIES (Behboudian et al., 2020, Satsangi et al., 2023)	Yes, as $\xi\to0$	Learning from advice
Action-Dependent	ADOPS (2505.12611)	Yes, for stable policies	Sparse, long-horizon RL
Bootstrapping	BSRS (Adamczyk et al., 2 Jan 2025)	Yes, adaptive potential	Deep RL, Atari
Abstraction-Based	Abstraction+PBRS (Canonaco et al., 2024)	Yes, if lift correct	Sample-efficient deep RL

7. Practical Guidelines and Empirical Outcomes

Effective deployment of PBRS requires:

• Potential Selection: Use potentials closely correlated with true value functions, as learned, bootstrapped, abstracted, or demonstration-digitized proxies.
• Bias Alignment: If initial Q-values or external reward scale are nontrivial, apply a linear shift to the potential to align PBRS per-step incentives without breaking invariance, as in $$c = \frac{(1-\gamma)Q_{\text{init}} - r_\infty}{\gamma - 1}$$ (2502.01307).
• Normalization: For nonstationary or episodic PBRS (e.g., PBIM, GRM), episode-level normalization or delay matching may be required for stable credit assignment and temporal distribution of bonuses.
• Algorithmic Simplicity: PBRS requires only an additive correction per TD update, independent of the underlying RL paradigm (Q-learning, DDPG, PPO, etc.).
• Verification: Empirical studies indicate 2x speedups are routine for properly aligned PBRS in challenging continuous control tasks, with orders of magnitude reduced hyperparameter sensitivity (Malysheva et al., 2020, Jeon et al., 2023, Adamczyk et al., 2 Jan 2025).

PBRS thus provides a uniquely principled, flexible, and empirically validated mechanism for accelerating RL without risk to optimality, provided the potential and its scaling are selected with care and, when needed, boundary and action dependencies are addressed with appropriate modern extensions.