Experience-Guided Reward Shaping

• Experience-guided reward shaping is a family of methods that transform sparse, delayed rewards using prior transitions, demonstrations, and human feedback.
• It incorporates potential-based techniques, kernel methods, and LLM-generated heuristics to create denser reward signals that guide policy learning effectively.
• Empirical studies across domains show enhanced sample efficiency and faster convergence through methods integrating historical data, expert behavior, and adaptive shaping mechanisms.

Experience-guided reward shaping denotes a family of methods that augment or transform the reward signal in reinforcement learning and related sequential decision settings by extracting guidance from prior experience rather than relying exclusively on hand-crafted dense rewards. In the cited literature, the experience source can be a task-agnostic transition dataset, a small set of task-specific demonstrations, historical success and failure trajectories, expert behavior, human feedback, sparse satisfaction surveys, experience-based trust dynamics, reward predictors trained on interaction data, or heuristic plans proposed by LLMs. The common motivation is the same: sparse, delayed, or misaligned rewards slow exploration and impede credit assignment, whereas experience-derived shaping can provide denser signals that accelerate learning or redirect it toward a different operational objective such as user satisfaction or trust (Koprulu et al., 2024, Ma et al., 2024, Christakopoulou et al., 2022, Guo et al., 2023).

1. Problem setting and conceptual scope

A canonical starting point is the sparse-reward finite-horizon Markov decision process $M=(S,A,P,r,\gamma)$ in which the agent must reach a target set $G\subset S$ as quickly as possible. In the formulation used by "Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations," $r(s,a)=1$ if the next state $s'\in G$ and $0$ otherwise, episodes terminate upon first hitting $G$, and the value function is $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$. Because the reward is zero until the goal is reached, even modern off-policy methods such as SAC struggle when the horizon is long and rewarding transitions are rare (Koprulu et al., 2024).

Across the broader literature, experience-guided shaping generalizes this setting in two directions. First, it broadens what counts as “experience”: SASR constructs shaped rewards from historical successful and failed trajectories; kernel IRL reconstructs an expert-like reward from observed expert behavior; FRESH converts human state and action labels into a shaping reward in high-dimensional state spaces; and recommender systems use sparse user surveys to impute dense satisfaction signals over logged interactions (Ma et al., 2024, Rucker et al., 2020, Xiao et al., 2020, Christakopoulou et al., 2022). Second, it broadens the objective of shaping itself. Some methods preserve the original optimal policy while making learning easier; others deliberately alter the operational objective, for example to improve human trust at bounded task-performance cost or to align recommendation policies with user satisfaction rather than engagement alone (Guo et al., 2023, Christakopoulou et al., 2022).

This suggests that “experience-guided” is best understood as a methodological umbrella rather than a single algorithm. The unifying property is not a specific update rule, but the use of empirically grounded side information to synthesize rewards that are denser, more informative, or better aligned with the target behavior than the raw environment reward alone.

2. Major construction families

The literature exhibits several recurrent design patterns.

Experience source	Representative construction	Papers
Prior transitions and demonstrations	Goal-conditioned value estimator, demonstration value function, and potential-based reshaping	(Koprulu et al., 2024)
Historical successes and failures	Beta-distributed success-rate reward estimated with KDE + RFF	(Ma et al., 2024)
Expert behavior or human labels	Kernel IRL potential; confidence-aware feedback reward	(Rucker et al., 2020, Xiao et al., 2020)
Sparse user feedback or trust trajectories	Satisfaction-imputed reward; trust-state potential in a Markov game	(Christakopoulou et al., 2022, Guo et al., 2023)
Learned heuristics or reward candidates	LLM-generated plan potentials; online reward selection; equity-aware LLM reward refinement	(Bhambri et al., 2024, Zhang et al., 2024, Zaregarizi et al., 27 May 2026)
Learned auxiliary predictors or adaptive shaping weights	Reward-prediction potential; bi-level shaping-weight optimization	(Hlynsson et al., 2021, Hu et al., 2020)
Knowledge-space pseudo-rewards	BAMDP potential-based shaping over histories	(Lidayan et al., 2024)

In the demonstration-and-prior-data family, the shaping term is derived from reachability. The prior dataset $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$ is used to fit a goal-conditioned value estimator $\tilde V_g(s;s_d)$, while each expert demonstration $\tau_d^j$ induces a demonstration value $G\subset S$0. A potential $G\subset S$1 then estimates value-to-go by combining reachability to a demonstration state with the remaining discounted path along the demonstration (Koprulu et al., 2024).

In the historical-buffer family, shaped rewards are empirical rather than model-based. SASR labels entire trajectories as successful or failed, places visited states into success and failure buffers, approximates $G\subset S$2 and $G\subset S$3 with KDE, and samples $G\subset S$4. The shaped reward thereby evolves from random early samples to concentrated estimates as evidence accumulates (Ma et al., 2024).

Human-centered variants differ in whether they infer latent reward or directly predict local desirability. Kernel IRL recovers an expert reward $G\subset S$5 in an RKHS and then uses it as a shaping potential for human learners. FRESH instead trains an ensemble feedback neural network on state and state-action labels, uses ensemble variance as a confidence proxy, and augments the environment reward only when the model’s confidence exceeds a threshold (Rucker et al., 2020, Xiao et al., 2020).

Other systems use experience to reshape objectives at application level. Christakopoulou et al. jointly train a policy head and a satisfaction-imputation head, then define the shaped reward as $G\subset S$6, with a hinge transform that zeros out engagement unless the imputed satisfaction is sufficiently high. Guo et al. define a potential over trust counters $G\subset S$7 in an experience-based trust model and use it to alter robot policy while constraining performance loss (Christakopoulou et al., 2022, Guo et al., 2023).

3. Mathematical foundations and invariance properties

The dominant theoretical template is potential-based reward shaping. In its standard form, the shaping term is

$G\subset S$8

or, in action-dependent formulations,

$G\subset S$9

When the shaped reward is $r(s,a)=1$0, the set of optimal policies is preserved. In the dense dynamics-aware construction this appears as $r(s,a)=1$1, with the relation $r(s,a)=1$2. In "Shaping Advice in Deep Reinforcement Learning," the same invariance argument is extended to single-agent and multi-agent actor-critic settings, together with convergence claims showing that convergence under shaping advice implies convergence of the corresponding quantities in the unshaped problem (Koprulu et al., 2024, Xiao et al., 2022).

Several works refine this template rather than merely reusing it. In the dense dynamics-aware setting, Proposition 4.1 establishes that under monotonicity of $r(s,a)=1$3 and a coverage condition, a greedy agent with $r(s,a)=1$4 will in finitely many steps reach $r(s,a)=1$5 by following $r(s,a)=1$6. The associated discount-factor annealing scheme uses a smaller training discount $r(s,a)=1$7 early in learning while retaining the original $r(s,a)=1$8 inside $r(s,a)=1$9, making the agent initially myopic with respect to the potential landscape and then gradually restoring full-horizon optimization (Koprulu et al., 2024).

A second extension moves shaping from physical state spaces to belief or history spaces. "BAMDP Shaping" characterizes all pseudo-rewards as reward shaping in Bayes-Adaptive Markov Decision Processes, where the augmented state is $s'\in G$0. A BAMDP potential-based shaping function satisfies

$s'\in G$1

and the paper proves that such functions preserve Bayes-optimality if and only if they are BAMPFs. The same work decomposes BAMDP value into value of information and value of opportunity, thereby interpreting pseudo-rewards as terms that reward changes in knowledge or in prior physical-state value (Lidayan et al., 2024).

Not all experience-guided methods seek exact invariance. The recommender formulation deliberately modifies the effective objective by multiplying engagement with imputed satisfaction, and the trust-aware HRI formulation explicitly changes the robot’s optimal policy while deriving a finite-horizon performance-loss bound in terms of terminal-state potentials. Conversely, BiPaRS starts from the premise that a supplied shaping function may be imperfect and therefore should not necessarily be applied in full; it formulates shaping utilization as a bi-level optimization problem with a parameterized shaping weight $s'\in G$2 and optimizes $s'\in G$3 for true reward maximization (Christakopoulou et al., 2022, Guo et al., 2023, Hu et al., 2020).

A common misconception is therefore that all reward shaping is policy-invariant. The cited literature supports a narrower claim: invariance holds for the potential-based constructions and their BAMDP analogue, whereas multiplicative shaping, trust-seeking shaping, and adaptive shaping-weight methods may alter the optimized objective by design.

4. Experience sources and algorithmic pipelines

The most explicit “integration of prior experience with demonstrations” pipeline is the three-phase procedure of dense dynamics-aware reward synthesis. First, a goal-conditioned value estimator $s'\in G$4 is pre-trained from the prior dataset $s'\in G$5. Second, each expert trajectory $s'\in G$6 is converted into a demonstration value function $s'\in G$7. Third, the algorithm computes $s'\in G$8 by maximizing over trusted demonstration states $s'\in G$9, where $0$0, and then forms $0$1. The online RL loop computes shaped rewards, stores shaped transitions in a replay buffer, and updates SAC-style policy and Q-networks on minibatches of shaped transitions (Koprulu et al., 2024).

SASR extracts experience more directly from the learner’s own history. Every complete trajectory is labeled successful or failed according to whether the environmental return equals $0$2 or $0$3, states are appended to $0$4 or $0$5, and the empirical success rate $0$6 is replaced by a stochastic Beta model with $0$7 and $0$8. KDE supplies density estimates in continuous spaces, RFF reduces the cost of evaluating Gaussian kernels, and the final reward becomes $0$9 (Ma et al., 2024).

In inverse-reward approaches, the shaping signal is inferred from demonstrations rather than directly counted. Kernel IRL assumes the expert reward lies in an RKHS dual, alternates between witness-reward construction and inner RL, outputs $G$0, and then uses $G$1 as dense feedback for human learners. FRESH uses a different interaction loop: the human labels sampled trajectories from a replay buffer, an ensemble feedback network learns to generalize those labels over high-dimensional observations, and the resulting state and action predictions are transformed into additive shaping rewards gated by confidence estimates derived from ensemble standard deviation (Rucker et al., 2020, Xiao et al., 2020).

Several systems learn dense surrogates from sparse user or environmental signals. The recommender architecture shares a bottom network between a REINFORCE policy head and a satisfaction-imputation head trained with weighted binary cross-entropy on sparse survey responses; it then computes $G$2 at every logged $G$3 pair. Hlynsson and Wiskott pretrain a reward predictor $G$4 on 10 k random transitions, freeze the predictor, and use $G$5 with an episode-wise decay factor to shape downstream ACKTR or PPO2 training (Christakopoulou et al., 2022, Hlynsson et al., 2021).

LLM-mediated approaches use model-generated heuristics as experience proxies. Bhambri et al. query an off-the-shelf LLM either on a deterministic low-level abstraction or on a hierarchical STRIPS-style abstraction, optionally with a verifier in a generate-test-repair loop, and translate the resulting plan into a state-action or state potential. ORSO treats candidate shaping rewards as base learners in an online model-selection problem, uses the D³RB rule $G$6, and reallocates training rounds toward rewards whose resulting policies evaluate well under the task reward. OccuReward closes the loop at a higher level: Gemini 1.5 Flash generates reward-function code over three rounds, with Round 3 incorporating per-profile satisfactions and the Comfort Equity Index into the reward (Bhambri et al., 2024, Zhang et al., 2024, Zaregarizi et al., 27 May 2026).

5. Empirical findings across domains

The empirical record is heterogeneous because the target domains range from continuous control to recommender systems, human learning, Atari, and grid-interactive buildings, but a repeated pattern is that experience-guided shaping improves sample efficiency relative to sparse-reward baselines. In the Robosuite puck-pushing benchmark with a three-walled U-slot, "Buffer+Potential+Annealing" achieves approximately $G$7–$G$8 success on the single-grid state-observation variant in about $G$9 K steps, about $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$0 on the two-grid variant, and about $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$1 on the point-cloud variant, whereas SAC with buffer but without shaping remains below $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$2 success and pure imitation baselines either fail or reach the goal suboptimally in about $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$3 steps instead of about $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$4 (Koprulu et al., 2024).

SASR reports improvements on 7 MuJoCo tasks, 2 robotic tasks, and MountainCar. The paper states that SASR learns successful policies in as few as $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$5–$$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$6 K steps on tasks where baselines require $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$7–$$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$8 K steps, with final returns such as $$V^\pi(s_0)=\mathbb E_\pi[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\mid s_0]$$9 on AntStand versus $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$0 for ReLara and $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$1 for SAC, $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$2 on AntFar versus $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$3 for ReLara and $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$4 for ExploRS, and $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$5 on RobotPush versus $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$6 for ReLara and $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$7 for RND (Ma et al., 2024).

Human- and user-centered systems show that reward shaping can improve behavior even when the environment is not a conventional RL benchmark. In the online game experiments on human learning, the final dataset contains $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$8 participants, the mean pretest touch count is about $P=\{(s_t^p,a_t^p,s_{t+1}^p)\}_{t=1}^{T_p}$9 across groups, and the high-touch expert shaping treatment HH yields a combined fixed-effects meta-estimate of $\tilde V_g(s;s_d)$0 touches with $\tilde V_g(s;s_d)$1, while LL shows a trend of $\tilde V_g(s;s_d)$2 with $\tilde V_g(s;s_d)$3. In the industrial recommender deployment, the hinge-shaped reward yields $\tilde V_g(s;s_d)$4 in satisfied engagement and $\tilde V_g(s;s_d)$5 in unsatisfying engagement, with real 5-star responses rising by $\tilde V_g(s;s_d)$6, 1–2-star responses falling by $\tilde V_g(s;s_d)$7, likes increasing by $\tilde V_g(s;s_d)$8, dislikes decreasing by $\tilde V_g(s;s_d)$9, and dismissals decreasing by $\tau_d^j$0 (Rucker et al., 2020, Christakopoulou et al., 2022).

In high-dimensional visual settings, FRESH reports average scores of $\tau_d^j$1 on Bowling and $\tau_d^j$2 on Skiing, compared with a human expert at $\tau_d^j$3 in Bowling and $\tau_d^j$4 in Skiing; the paper states that FRESH exceeds the human expert by $\tau_d^j$5 in Bowling. Reward prediction for representation learning and shaping reports, for ACKTR on Four-room after 1 M steps, mean returns of $\tau_d^j$6 for deep RL, $\tau_d^j$7 for successor features, $\tau_d^j$8 for the raw predictor without shaping, and $\tau_d^j$9 for the raw predictor with shaping (Xiao et al., 2020, Hlynsson et al., 2021).

LLM-assisted and reward-selection results further broaden the empirical picture. Bhambri et al. report that in BabyAI DoorKey-5×5, PPO and A2C with LLM+verifier shaping reach optimal return in about $G\subset S$00 steps versus about $G\subset S$01 for vanilla training, and that in hierarchical domains Q-learning with complete LLM plans converges $G\subset S$02–$G\subset S$03 faster than baseline. ORSO reports up to $G\subset S$04 reduction in computational time, uses $G\subset S$05–$G\subset S$06 fewer environment interactions than naive selection, and reaches at least $G\subset S$07 of human-baseline performance in about $G\subset S$08 of the iterations naive uniform selection requires. OccuReward reports that Round 3 equity-aware refinement reduces normalized energy cost from $G\subset S$09 to $G\subset S$10, reduces CEI from $G\subset S$11 to $G\subset S$12, and raises satisfaction from $G\subset S$13 to $G\subset S$14 for Elderly Female while also improving Young Male, Mid-aged Female, and Health Sensitive profiles (Bhambri et al., 2024, Zhang et al., 2024, Zaregarizi et al., 27 May 2026).

6. Limitations, failure modes, and open directions

A central limitation is coverage. In dense dynamics-aware synthesis, if $G\subset S$15 is ill-fitted under dynamics shift or $G\subset S$16 is set too low, many states fall outside $G\subset S$17, $G\subset S$18, and shaping collapses to pure sparse RL; if demonstrations are too suboptimal and leave gaps, the agent may initially be led astray, although annealing $G\subset S$19 back to $G\subset S$20 preserves eventual optimality. The same paper notes that extremely long-range reachability remains challenging without a good short-horizon prior, and that raw-pixel extensions require coupling with representation learning in latent goal spaces (Koprulu et al., 2024).

A second limitation is dependence on auxiliary modeling assumptions. SASR relies on KDE and RFF approximations to state visitation counts; LLM-guided plan shaping depends on either a deterministic abstraction or a symbolic abstraction and often on a verifier with access to valid actions or fluent preconditions; ORSO requires that at least one good shaping reward exist in the candidate set and that the true task reward be available for evaluation; OccuReward shows that even after reward-level intervention, demographic disparities persist (Ma et al., 2024, Bhambri et al., 2024, Zhang et al., 2024, Zaregarizi et al., 27 May 2026).

Human-in-the-loop methods face annotation and calibration burdens. Kernel IRL for human learning reports modest effect sizes and notes that kernel choice and post-processing smoothing were hand-tuned. FRESH relies on periodic human labeling of replayed trajectories, and its advantages are strongest in settings where people can rapidly recognize good and bad states or actions even if they cannot supply exact scalar rewards. These observations support the broader claim that experience-guided shaping is often most effective when the auxiliary signal is easier to obtain locally than the task reward is to obtain globally (Rucker et al., 2020, Xiao et al., 2020).

A further open issue concerns imperfect priors. BiPaRS is explicitly motivated by the possibility that human-translated shaping rewards are biased or harmful, and its empirical results show that adaptive shaping weights can ignore unbeneficial shaping rewards or even transform them into beneficial ones. "BAMDP Shaping" sharpens this concern theoretically by showing that pseudo-rewards can induce counterproductive exploits when they align poorly with actual value, while bounded monotone BAMPFs resist reward-hacking in both meta-RL and regular RL (Hu et al., 2020, Lidayan et al., 2024).

Taken together, the literature does not support a single best formulation of experience-guided reward shaping. It supports a design space. When policy invariance is essential, potential-based constructions over states, state-actions, or histories provide the cleanest guarantees. When the goal is objective modification rather than mere acceleration, multiplicative satisfaction shaping, trust-seeking potentials, or equity-aware reward refinement become appropriate. The main research frontier suggested by these works is therefore not whether experience should shape reward, but how to choose the experience source, shaping form, and theoretical constraint that match the intended operational objective.