Bayes-Adaptive Potential-Based Shaping Framework (BAMPF)

• BAMPF is a reinforcement-learning framework that defines pseudo-reward signals over Bayes-adaptive Markov Decision Process (BAMDP) histories, unifying intrinsic motivation and reward shaping as value corrections aligned with agent knowledge.
• It establishes necessity-and-sufficiency for safe potential-based shaping functions (BAMPFs), showing only BAMPFs guarantee policy-preserving reward design in both meta-RL and regular RL, addressing reward hacking and exploration pathologies.
• The framework presents practical guidelines, including decomposing value into information and opportunity components, and demonstrates empirically how BAMPF-shaped signals align exploration with true task objectives without incentivizing harmful behaviors.

Searching arXiv for the focal paper and closely related work on Bayes-adaptive MDPs and potential-based reward shaping. Search 1: focal paper. The Bayes-Adaptive Potential-Based Shaping Framework (BAMPF) is a reinforcement-learning framework that recasts intrinsic motivation and reward shaping as pseudo-reward design in a Bayes-Adaptive Markov Decision Process (BAMDP), where the agent’s effective state includes both the physical environment state and the agent’s accumulated history or knowledge. In this formulation, pseudo-rewards are interpreted as corrections to the value of information gathered during learning or to the prior value of the physical state, and safe pseudo-rewards are identified with Bayes-Adaptive Potential-based shaping Functions (BAMPFs), which extend potential-based shaping from MDP states to BAMDP histories. The framework is presented as a unified account of why pseudo-rewards can both help and harm exploration, including “noisy TV” behavior, stalling, and reward hacking, and it establishes necessity-and-sufficiency claims for policy-preserving shaping in meta-RL and bounded reward-hacking claims under additional conditions in regular RL (Lidayan et al., 2024).

1. Definition and conceptual scope

The framework begins from the observation that intrinsic motivation and reward shaping both guide reinforcement-learning agents by adding pseudo-rewards. These pseudo-rewards can induce useful emergent behavior, but they can also produce counterproductive exploits, including attraction to unpredictable but task-irrelevant stimuli, fixation on the pseudo-reward itself, and misalignment between the composite reward and the real task objective. The central claim is that these pathologies are not peripheral implementation issues; they arise because pseudo-rewards are often specified relative to the wrong state space.

In standard potential-based shaping for an MDP, the shaping term is defined over physical states:

$F(s_t,a_t,s_{t+1})=\gamma\phi(s_{t+1})-\phi(s_t).$

This preserves the optimal MDP policy. BAMPF generalizes the same logic to the learning process itself by defining shaping over histories, or equivalently over BAMDP states:

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$

The conceptual shift is that the relevant state of an RL system is not only the current physical state $s_t$, but also what has been learned so far. This makes intrinsic motivation and reward shaping instances of a single design problem: shaping over agent knowledge.

A common misconception is that intrinsic motivation and reward shaping are fundamentally distinct mechanisms. In the BAMPF formulation they are unified by treating both as pseudo-reward terms that modify the learning objective inside a BAMDP. This suggests that the crucial distinction is not whether a bonus is called “intrinsic” or “shaping,” but whether it is aligned with Bayes-adaptive value.

2. BAMDP formulation and learning-aware value

The paper defines a BAMDP as

$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$

with augmented state space

$\bar{\mathcal{S}}=\mathcal{S}\times \mathcal{H},$

so that a BAMDP state is

$\langle s_t,h_t\rangle,$

where

$h_t=s_0a_0r_1s_1\ldots a_{t-1}r_ts_t$

is the full history up to time $t$. The history induces a posterior over possible MDPs,

$p(M\mid h_t)\propto p(h_t\mid M)p(M),$

and the BAMDP reward is the posterior expectation of the underlying reward,

$$\bar{R}(s_t,a)=\mathbb{E}_{p(M\mid h_t)}[R(s_t,a)].$$

The BAMDP transition and initial distribution are likewise defined by expectations over the posterior, so the learning process becomes part of the state dynamics (Lidayan et al., 2024).

Under this construction, the expected return of an RL algorithm in an unknown MDP is the BAMDP return:

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$0

The optimal BAMDP policy is therefore the Bayes-optimal RL algorithm. The associated Bellman-style recursion is

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$1

The important feature is that the continuation value depends on the updated history $F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$2, not merely on the next physical state. Exploration, exploitation, and belief updating are therefore evaluated in a single value function.

This formulation provides the theoretical basis for interpreting pseudo-rewards as modifications to a learning-aware objective rather than as local heuristics on the underlying task MDP.

3. Decomposition into value of information and value of opportunity

A major contribution of the framework is the decomposition of optimal BAMDP value into the value of information (VOI) and the value of opportunity (VOO). The value of information is defined as

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$3

and the value of opportunity is defined as

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$4

The decomposition theorem is stated as

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$5

VOI captures the gain attributable to knowledge contained in the history; VOO captures the value of occupying the current physical state under the initial prior, before any learning. In the paper’s interpretation, many pseudo-rewards can be understood as trying to repair one of these terms. Intrinsic-motivation bonuses often target VOI, whereas conventional shaping terms often target VOO by compensating for missing prior structure in the environment (Lidayan et al., 2024).

The paper’s examples make the taxonomy explicit. Entropy bonus, information gain, prediction error, and novelty are VOI-related. Goal proximity, subtask completion, and ball possession are VOO-related. Surprise minimization and information cost are described as repulsive VOO signals. Empowerment is treated as a composite of VOI and VOO signals. The practical significance of the decomposition is that pseudo-reward design becomes a problem of identifying which component of BAMDP value is being misestimated.

A plausible implication is that many reward-design failures can be reinterpreted as failures to distinguish information value from opportunity value. The “noisy TV” pathology is consistent with a pseudo-reward that rewards uncertainty or prediction difficulty without regard to whether that uncertainty contributes to BAMDP value.

4. Formal definition of BAMPFs and invariance

A pseudo-reward $F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$6 is a Bayes-Adaptive Potential-based shaping Function if there exists a potential $F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$7 such that, for all realizable histories,

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$8

The shaped BAMDP reward is

$F(h_t)=\gamma\phi(h_t)-\phi(h_{t-1}).$9

with $s_t$0 and $s_t$1 unchanged.

The paper’s central theorem states that, for a pseudo-reward function to guarantee that the optimal algorithm for any shaped BAMDP is optimal for the original BAMDP, it is necessary and sufficient for it to be a BAMPF. This is presented as the exact analogue of the standard potential-based shaping theorem, but lifted from MDPs to BAMDPs (Lidayan et al., 2024).

The safety argument is telescoping. Over a finite horizon,

$s_t$2

and in the infinite-horizon case the shaped expected return differs by a constant shift,

$s_t$3

Because the modification is a constant offset to return, optimal action choices are preserved.

The framework therefore makes a strong exclusivity claim: only BAMPFs are guaranteed not to distort Bayes-optimal behavior, and any non-BAMPF can be constructed to mislead even an optimal learner in some BAMDP. This directly addresses reward hacking in meta-RL, where the issue is not merely whether the composite reward is learnable, but whether it preserves the correct Bayes-adaptive objective.

5. Safety results, approximate optimality, and limitations

Beyond exact invariance, the paper extends the analysis to near-optimal algorithms. It states an $s_t$4-equivalence result: if an algorithm is near-optimal on the shaped BAMDP, it remains near-optimal on the original BAMDP. It also gives a $s_t$5-step learning-aware regret bound under bounded potential magnitude $s_t$6 and bounded extrinsic reward $s_t$7, with the bound scaling as

$s_t$8

The interpretation given in the paper is that, as $s_t$9 grows, the effect of shaping on the true objective becomes negligible.

The paper also includes a “$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$0-insensitive” remark: if an RL algorithm cannot distinguish returns that differ by less than some threshold $\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$1, then once

$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$2

the BAMPF shaping no longer changes behavior. This is a statement about practical indistinguishability rather than exact policy invariance.

The guarantees, however, are explicitly qualified. Exact Bayes-optimal planning is generally intractable. The framework provides principles for pseudo-reward design, not an exact recipe. Its strongest guarantees apply to sufficiently capable or near-optimal learners, and poorly suboptimal learners can still behave badly even when the pseudo-reward is a BAMPF. The abstract further states that BAMPFs with bounded monotone increasing potentials also resist reward-hacking in the regular RL setting, extending the safety discussion beyond meta-RL (Lidayan et al., 2024).

A common misunderstanding is to read the theorem as showing that any BAMPF will improve learning. The framework does not claim universal improvement. It claims preservation of the true objective under the stated conditions, while leaving open whether a particular potential will be informative or computationally useful for a given learner.

6. Empirical illustrations and design criteria

The paper reports two illustrative demonstrations. In a Bernoulli bandit meta-RL setting, BAMPF shaping is used to show how shaping affects learned algorithms over a distribution of tasks. The purpose is to demonstrate that a meta-learner trained with BAMPF shaping can learn exploration behavior that is helpful for the true reward rather than merely for the pseudo-reward. The pseudo-rewards considered include information gain and other Bayes-adaptive value-correcting terms. Information gain is explicitly presented as a BAMPF with

$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$3

which yields the shaping reward

$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$4

The intended behavior is to explore arms when information is valuable while avoiding pathological attraction to noisy or irrelevant uncertainty. The empirical point emphasized in the text is alignment: BAMPF-shaped signals align better with correct exploration than non-BAMPF pseudo-rewards (Lidayan et al., 2024).

The Mountain Car demonstration serves as a classic shaping example. The figure caption gives the standard potential-based reward-shaping potential

$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$5

This encourages progress toward the goal and preserves the optimal policy when the shaping is potential-based. The paper uses this example to illustrate how the same logic can be applied over histories and agent knowledge, so that a history-dependent potential can reward learning progress or uncertainty reduction while remaining safe at the level of the learning algorithm.

The framework also proposes a practical design sequence. First, identify whether the learner is underestimating VOI or VOO. Second, encode the missing value as a BAMDP potential $\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$6 so that the pseudo-reward takes the BAMPF form

$\bar{M}=(\bar{\mathcal{S},\mathcal{A},\bar{R},\bar{T},\bar{T}_0,\gamma),$7

Third, use the BAMPF structure to preserve the Bayes-optimal objective. Fourth, avoid non-BAMPF pseudo-rewards when safety matters, since heuristic bonuses such as raw prediction error or novelty can misfire on irrelevant stochasticity, create “noisy TV” behavior, or incentivize pathological exploration.

The paper’s taxonomy of signals can be summarized as follows:

Signal class	Examples
Attractive VOO signals	goal proximity, subgoal completion, ball possession
Repulsive VOO signals	negative surprise, information cost
Attractive VOI signals	entropy bonus, prediction error, learning progress, skill discovery, info gain
Composite signals	empowerment, subtask completion

This suggests a unifying criterion for retrofitting existing pseudo-rewards: if a signal can be expressed as a potential over BAMDP states, it inherits the policy-preservation properties of BAMPFs; if it cannot, then safety with respect to the true Bayes-adaptive objective is not guaranteed.