Learn Potential-Based Shaping Useful Throughout Training

Develop a procedure to learn a potential function Φ(s) for potential-based reward shaping such that, when added to a recovered reward r to form r′(s,a)=r(s,a)+Φ(s′)−Φ(s), the shaped reward accelerates policy training from scratch throughout the entire training process, including early stages with weak initial policies.

Background

Potential-based reward shaping preserves optimal policies but can dramatically reduce the effective planning horizon, thus improving interaction efficiency. In theory, setting Φ to the optimal value function V* would yield a one-step greedy optimal policy, but computing or approximating V* uniformly over the state space is infeasible in realistic settings.

Given the practical difficulty of obtaining an admissible heuristic or expert critic that is accurate beyond the expert’s state distribution, the authors pose the need for a method that learns a shaping term that is helpful across the full training trajectory when starting from scratch.