Prioritized Expert Demonstration Replay (PEDR)

• PEDR is a prioritized replay technique that selectively samples expert and synthetic demonstrations based on informativeness to enhance policy learning in deep RL and IL.
• It adapts prioritized experience replay by computing loss-based priorities and importance-sampling weights to optimally balance real, pseudo-expert, and agent-generated data.
• Empirical evaluations show that PEDR improves convergence speed, peak performance, and stability in complex tasks such as urban driving and locomotion.

Prioritized Expert Demonstration Replay (PEDR) is a replay buffer technique designed to accelerate and stabilize policy learning in deep reinforcement learning (RL) and imitation learning (IL) frameworks by selectively emphasizing expert demonstrations according to an adaptive, informativeness-based priority mechanism. PEDR originated as an augmentation of prioritized experience replay concepts and has been deployed in both adversarial imitation learning with synthetic trajectories (Li et al., 21 Dec 2025) and reinforcement learning from demonstrations in complex domains such as urban driving (Liu et al., 2021). Its primary objective is to sample the most valuable expert demonstrations and agent-generated experiences to improve learning efficiency, especially in scenarios with limited or uneven-quality demonstration data.

1. Motivation and Conceptual Foundations

In imitation learning tasks, leveraging expert demonstrations is crucial for policy performance and stability, yet acquiring large, high-quality datasets is often prohibitively expensive or infeasible. The integration of synthetic (pseudo-expert) trajectories, such as those produced by diffusion-based generative models, compounds the challenge by introducing a large but highly variable pool of instructional data. Uniform sampling from such heterogeneous buffers leads to inefficient learning and increased risk of policy degradation due to low-quality samples.

PEDR addresses these challenges by incorporating prioritized sampling based on a formal measure of demonstration informativeness or surprise. By drawing inspiration from Prioritized Experience Replay (PER), PEDR ensures that the training process emphasizes the most informative or challenging transitions—those with highest discriminator error in adversarial imitation (Li et al., 21 Dec 2025) or highest composite loss in joint RL–IL setups (Liu et al., 2021). This results in accelerated convergence, improved policy robustness, and better exploitation of both real and synthetic demonstrations.

2. Mathematical Formulation

PEDR operates on the principle of assigning each demonstration or transition a real-valued priority reflecting its learning value. The formulation differs slightly with context, but generalizes as follows:

• Let $i=1,\ldots,N$ index transitions in the replay buffer(s).
• A scalar priority $p_i$ is computed using error or loss-based criteria:
  • In AIL with synthetic data (Li et al., 21 Dec 2025): For each demo $(s_i, a_i)$, $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$, with priority $p_i = |\delta_i|$.
  • In SAC+IL for urban driving (Liu et al., 2021): $p_i$ is a sum of relevant RL/IL losses, e.g., for self-generated transitions,

  $$p_i^{RL} = L_{\pi,RL}(\phi)_i + \tfrac{1}{2}\left[ L_Q(\theta_1)_i + L_Q(\theta_2)_i\right] + \epsilon,$$

  and for expert demos,

  $$p_i^{IL} = L_{\pi,IL}(\phi)_i + \tfrac{1}{2}\left[ L_Q(\theta_1)_i + L_Q(\theta_2)_i\right] + \epsilon.$$

• Sampling probabilities are computed as

$P(i) = \frac{p_i^\zeta}{\sum_{k=1}^N p_k^\zeta}$

with $\zeta$ (AIL) or $p_i$0 (RL/IL) controlling prioritization strength.

• To compensate for the induced sampling bias, importance-sampling (IS) weights are computed:

$p_i$1

where $p_i$2 is annealed over training. Normalization $p_i$3 is sometimes applied.

• In batch construction, demonstrations are typically drawn in fixed ratio from real and pseudo-expert buffers, with each buffer prioritized independently.

3. Algorithmic Realization

PEDR is implemented as a subroutine within the training loop, synchronizing priority computation, prioritized sampling, IS correction, and priority updates upon each model parameter change. Typical steps are as follows:

1. Maintain separate prioritized buffers for different sources (real expert, pseudo-expert, agent-generated).

2. For each update:

• Sample $p_i$4 real and $p_i$5 pseudo-expert demonstrations from respective buffers proportional to $p_i$6.
• Construct batch $p_i$7 and compute IS weights $p_i$8.
• Compute the weighted loss for discriminator or Q/policy networks.
• Update model parameters via gradient descent.
• Recompute priorities for the sampled transitions and update buffer entries.

1. Mixing ratios and prioritization coefficients are controlled via hyperparameters, allowing dynamic adaptation as training progresses.

The following table summarizes key PEDR algorithmic components across major frameworks:

Component	SD2AIL (AIL) (Li et al., 21 Dec 2025)	Urban Driving (RL+IL) (Liu et al., 2021)
Priority $p_i$9	$(s_i, a_i)$0	Composite of RL/IL + Q-losses
Sampling exponent	$(s_i, a_i)$1	$(s_i, a_i)$2
IS exponent (annealed)	$(s_i, a_i)$3: $(s_i, a_i)$4	$(s_i, a_i)$5
Buffer Types	Real + pseudo-expert (separate)	Agent-generated + expert (separate)
Mixing Ratio (per batch)	$(s_i, a_i)$6	$(s_i, a_i)$7 (annealed RL:IL mix)
Capacity	$(s_i, a_i)$8: tens-of-thousands demos	Agent: $(s_i, a_i)$9, Expert: $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$0

4. Integration with Learning Architectures

PEDR is architecturally agnostic and can be integrated into a wide variety of RL/IL frameworks. Illustrative integrations include:

• SD2AIL Framework (Li et al., 21 Dec 2025):

  • PEDR underpins discriminator updates, determining the composition of real and pseudo-expert batches.
  • The discriminator, trained on PEDR-weighted samples, produces surrogate rewards to guide off-policy SAC policy optimization.
  • Diffusion models periodically generate new pseudo-expert trajectories, which are filtered and prioritized before addition to the replay pool.
  • Sampling probabilities are dynamically updated after each outer-loop iteration to reflect evolving informativeness.
• SAC+IL for Urban Driving (Liu et al., 2021):
  • PEDR orchestrates mixed-batch updates over expert and self-exploration transitions.
  • The mixing ratio $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$1 is adaptively incremented toward full agent-driven updates as the policy surpasses expert performance.
  • Both agent and expert buffers are prioritized and sampled independently, ensuring both exploration and expert coverage.

These integrations allow PEDR to simultaneously maintain diversity, maximize informative coverage, and suppress the destabilizing effects of low-quality (including synthetic) demonstrations.

5. Hyperparameterization and Design Choices

Effective deployment of PEDR hinges on careful tuning of several hyperparameters:

• Priority exponent ($\delta_i=1-D_\phi(s_i, a_i, \epsilon)$2, $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$3): Governs the focus on high-error transitions. Higher values increase selectivity; recommended values in the range [0.4, 0.8] in AIL, and 0.6 in RL+IL.
• Importance-sampling exponent ($\delta_i=1-D_\phi(s_i, a_i, \epsilon)$4, $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$5): Controls bias–variance tradeoff in loss correction; annealed from 0.4 to 1 in SD2AIL; fixed at 0.4 in urban driving tasks.
• Mixing ratio: Proportion of expert vs. non-expert samples per batch; set at $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$6:$\delta_i=1-D_\phi(s_i, a_i, \epsilon)$7 (pseudo:real) in SD2AIL, and adaptively increased in RL+IL as agent matches/exceeds expert return.
• Buffer capacities: Must accommodate diverse trajectory sources, with synthetic/agent buffers often much larger due to data volume.
• Other: Small offsets ($\delta_i=1-D_\phi(s_i, a_i, \epsilon)$8) prevent vanishing priorities; learning rates and Polyak averaging typical for deep RL/IL.

All hyperparameters in the cited works were selected by grid search around standard RL/IL defaults and validated by ablation (Li et al., 21 Dec 2025, Liu et al., 2021).

6. Empirical Outcomes and Evaluation

PEDR has demonstrated substantial empirical gains in both adversarial imitation learning and RL+IL settings:

• Learning Efficiency: PEDR accelerates convergence. In SD2AIL (Hopper), PEDR achieved convergence in $\delta_i=1-D_\phi(s_i, a_i, \epsilon)$9K steps versus $p_i = |\delta_i|$0K for baselines (Li et al., 21 Dec 2025). In urban driving, PEDR-SAC surpassed expert reward within $p_i = |\delta_i|$1K steps and stably peaked $p_i = |\delta_i|$2 at $p_i = |\delta_i|$3K steps versus $p_i = |\delta_i|$4K for baseline SAC (Liu et al., 2021).
• Asymptotic Performance: PEDR consistently improves peak returns; for example, on Walker with one expert trajectory, “PEDR Only” achieved $p_i = |\delta_i|$5 versus $p_i = |\delta_i|$6 for uniform replay (Li et al., 21 Dec 2025).
• Stability: Return variance across multiple seeds was reduced by up to $p_i = |\delta_i|$7, indicating more stable learning (Li et al., 21 Dec 2025).
• Reward Fidelity: The Pearson correlation between surrogate and true rewards was substantially higher with PEDR (90–93%) compared with uniform/naive methods (77–81%) (Li et al., 21 Dec 2025).
• Distribution Alignment: The Fréchet distance between pseudo and real demonstration features shrank faster under PEDR, suggesting an accelerated alignment toward expert manifolds.
• Task Outcomes in RL+IL: In urban navigation, PEDR policies achieved $p_i = |\delta_i|$8 success rate, $p_i = |\delta_i|$9 collision rate, and improved episode rewards and efficiencies compared to SAC, DQfD, SQIL, and on-policy RL methods (Liu et al., 2021).

Empirical ablations confirm that PEDR alone provides significant advantages over uniform replay, and its combination with synthetic demonstration generation yields further state-of-the-art results.

7. Significance and Context

PEDR generalizes and extends prioritized replay paradigms from vanilla RL to settings demanding tight integration of demonstration data, including adversarial and hybrid imitation-learning regimes. Its adaptability to both synthetic and real demonstrations enables robust scalability to problems where data quality and informativeness vary widely. PEDR’s widespread empirical validation underscores its impact on accelerating learning, enhancing stability, and improving policy quality across challenging domains such as locomotion and urban autonomous driving (Li et al., 21 Dec 2025, Liu et al., 2021).

A plausible implication is that continued advances in generative modeling will further increase the need for robust prioritization schemes like PEDR to harness synthetic data effectively. As the distinction between real and synthetic expertise dissolves, mechanisms for demonstration curation and prioritization will play a central role in next-generation imitation and reinforcement learning systems.