Predictive Preference Learning from Human Interventions

Abstract: Learning from human involvement aims to incorporate the human subject to monitor and correct agent behavior errors. Although most interactive imitation learning methods focus on correcting the agent's action at the current state, they do not adjust its actions in future states, which may be potentially more hazardous. To address this, we introduce Predictive Preference Learning from Human Interventions (PPL), which leverages the implicit preference signals contained in human interventions to inform predictions of future rollouts. The key idea of PPL is to bootstrap each human intervention into L future time steps, called the preference horizon, with the assumption that the agent follows the same action and the human makes the same intervention in the preference horizon. By applying preference optimization on these future states, expert corrections are propagated into the safety-critical regions where the agent is expected to explore, significantly improving learning efficiency and reducing human demonstrations needed. We evaluate our approach with experiments on both autonomous driving and robotic manipulation benchmarks and demonstrate its efficiency and generality. Our theoretical analysis further shows that selecting an appropriate preference horizon L balances coverage of risky states with label correctness, thereby bounding the algorithmic optimality gap. Demo and code are available at: https://metadriverse.github.io/ppl

• The paper introduces a novel IIL algorithm that propagates human corrective signals over a predicted trajectory horizon to reduce intervention frequency.
• It integrates behavioral cloning loss with contrastive preference optimization to improve policy performance across simulated and control tasks.
• Empirical results demonstrate higher success rates and smoother control in both autonomous driving and robotic manipulation compared to existing methods.

Predictive Preference Learning from Human Interventions: A Technical Analysis

Overview and Motivation

The paper introduces Predictive Preference Learning from Human Interventions (PPL), a novel interactive imitation learning (IIL) algorithm that leverages trajectory prediction and preference learning to improve sample efficiency and reduce the cognitive burden on human supervisors. The core insight is to proactively forecast the agent’s future trajectory, visualize it for a human expert, and, upon intervention, propagate the corrective signal not only to the current state but also to a horizon of predicted future states. This approach addresses the limitations of prior IIL methods, which typically only correct the agent at the current state and require continuous, high-attention human monitoring.

Figure 1: PPL forecasts the agent’s upcoming trajectory and visualizes it for the human expert, who intervenes if a failure is anticipated; a single intervention is interpreted as preference signals across predicted states, guiding the policy to avoid risky maneuvers in similar contexts.

Methodology

Trajectory Forecasting and Human-in-the-Loop Preference Collection

At each decision point, the agent proposes an action, and a trajectory prediction model simulates the agent’s future states over a short horizon. This predicted rollout is visualized for the human expert, who intervenes only if the forecasted trajectory is likely to result in failure (e.g., collision, task failure). If an intervention occurs, the corrective action is recorded, and preference pairs are constructed for each state along the predicted trajectory, indicating that the expert prefers the corrective action over the agent’s original action at those states.

Figure 2: At each decision point, the agent’s predicted trajectory is visualized; interventions are converted into contrastive preference pairs over the predicted future states, propagating expert intent into regions the agent is likely to explore.

Preference Propagation and Policy Optimization

The key innovation is the bootstrapping of each intervention into a set of preference labels over a “preference horizon” $L$. For each predicted state $\tilde{s}_i$ ($i=1,\ldots,L$), a tuple $(\tilde{s}_i, a^+, a^-)$ is added to the preference dataset, where $a^+$ is the expert’s corrective action and $a^-$ is the agent’s original action. The policy is then trained using a combination of behavioral cloning (BC) loss on the expert demonstrations and a contrastive preference optimization (CPO) loss on the preference dataset:

$$\mathcal{L}(\pi_\theta) = \mathcal{L}_{\text{pref}}(\pi_\theta) + \mathcal{L}_{\text{BC}}(\pi_\theta)$$

where $\mathcal{L}_{\text{pref}}$ is a contrastive classification loss over preference pairs, and $\mathcal{L}_{\text{BC}}$ is the standard negative log-likelihood loss on expert actions.

Theoretical Analysis

The paper provides a theoretical bound on the performance gap between the learned policy and the expert policy, decomposing it into three terms: (1) state distribution shift between the agent’s on-policy distribution and the preference dataset, (2) the quality of the preference labels (i.e., the mismatch between the corrective action at the current state and what would be optimal at predicted future states), and (3) the optimization error. The preference horizon $L$ is a critical hyperparameter: increasing $L$ improves coverage of risky states but may degrade label quality due to compounding model errors and action mismatch.

Implementation Details

Trajectory Prediction

For simulated environments, the trajectory predictor is implemented as an $H$-step simulator rollout, applying the agent’s action repeatedly. For real-world driving, a kinematic bicycle model is used for short-term prediction, assuming static obstacles. This model achieves real-time performance (up to 3,000 fps on CPU) and is robust to moderate prediction noise.

Human Interface

The system visualizes predicted trajectories and allows the human expert to intervene via a gamepad or keyboard. The interface is updated every $H=10$ steps, balancing responsiveness and cognitive load.

Figure 3: Human interfaces for MetaDrive, Table Wiping, and Nut Assembly; predicted trajectories are visualized, and interventions are performed via gamepad or keyboard.

Empirical Results

Sample Efficiency and Performance

PPL is evaluated on MetaDrive (autonomous driving) and Robosuite (robotic manipulation: Table Wiping and Nut Assembly) benchmarks, using both real human experts and neural experts as proxies. Across all tasks, PPL achieves higher test-time success rates with significantly fewer expert interventions and lower total data usage compared to baselines such as BC, GAIL, Ensemble-DAgger, Thrifty-DAgger, PVP, IWR, EIL, and HACO.

Figure 4: Test-time performance curves for PPL and PVP across three environments; PPL achieves higher success rates with fewer environment interactions.

In MetaDrive with real human experts, PPL achieves a 76% success rate within 10K steps, requiring only 2.9K human-labeled transitions (0.29 intervention rate), outperforming all baselines in both efficiency and final performance.

Training Dynamics

During training, PPL rapidly reduces the frequency of human interventions and safety violations, converging to autonomous, high-performing policies with minimal expert oversight.

Figure 5: Training process in MetaDrive: test success rate, intervention rate, and safety cost; PPL quickly becomes autonomous, requiring fewer interventions.

Ablation and Robustness

Ablation studies confirm that the construction of contrastive preference pairs is essential; discarding either positive or negative actions, or using only BC or CPO loss, degrades performance. PPL is robust to moderate noise in the trajectory predictor and achieves optimal efficiency with a preference horizon $L=4$ in MetaDrive.

(Figure 6)

Figure 6: PPL maintains superior performance under increasing trajectory prediction noise.

(Figure 7)

Figure 7: Success rate as a function of preference horizon $L$; optimal efficiency is achieved at $L=4$.

Qualitative Analysis

PPL produces smoother, more human-aligned control sequences compared to PVP, especially in safety-critical scenarios.

(Figure 8)

Figure 8: Steering control sequences for PPL and PVP; PPL yields smoother, more consistent behavior near obstacles.

Practical and Theoretical Implications

PPL demonstrates that integrating trajectory prediction and preference learning into IIL can substantially reduce the human effort required for safe and efficient policy learning. The approach is general, requiring no task-specific reward engineering or offline pretraining, and is applicable to both control and manipulation domains. The theoretical analysis provides guidance for selecting the preference horizon to balance coverage and label quality, and the empirical results validate the method’s robustness and efficiency.

From a practical perspective, PPL enables scalable human-in-the-loop training for real-world robotics and autonomous systems, where safety and sample efficiency are paramount. The reduction in cognitive load and intervention frequency is particularly relevant for deployment in safety-critical applications.

Limitations and Future Directions

The method assumes that human interventions are optimal and consistent, which may not hold in practice, especially with non-expert users or in highly complex environments. All experiments are conducted in simulation; real-world deployment will require addressing sim-to-real transfer, model inaccuracies, and potentially suboptimal human feedback. Future work should explore adaptive preference horizon selection, integration with uncertainty-aware predictors, and large-scale real-world validation.

Conclusion

Predictive Preference Learning from Human Interventions advances the state of interactive imitation learning by leveraging proactive trajectory forecasting and preference propagation. The method achieves strong empirical performance and sample efficiency, supported by theoretical guarantees, and offers a scalable framework for human-in-the-loop training of autonomous agents. Its generality and robustness make it a promising foundation for future research in safe, efficient, and human-aligned policy learning.