Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret

Abstract: Robot reinforcement learning from demonstrations (RLfD) assumes that expert data is abundant; this is usually unrealistic in the real world given data scarcity as well as high collection cost. Furthermore, imitation learning algorithms assume that the data is independently and identically distributed, which ultimately results in poorer performance as gradual errors emerge and compound within test-time trajectories. We address these issues by introducing the "master your own expertise" (MYOE) framework, a self-imitation framework that enables robotic agents to learn complex behaviors from limited demonstration data samples. Inspired by human perception and action, we propose and design what we call the queryable mixture-of-preferences state space model (QMoP-SSM), which estimates the desired goal at every time step. These desired goals are used in computing the "preference regret", which is used to optimize the robot control policy. Our experiments demonstrate the robustness, adaptability, and out-of-sample performance of our agent compared to other state-of-the-art RLfD schemes. The GitHub repository that supports this work can be found at: https://github.com/rxng8/neurorobot-preference-regret-learning.

• The paper presents a novel self-imitation framework (MYOE) that minimizes preference regret to optimize neurorobot policies with extremely limited demonstration data.
• It utilizes an innovative QMoP-SSM to jointly learn hidden representations and goal preferences, enabling effective planning through imagined rollouts.
• Empirical results from simulated and real robotic tasks demonstrate faster convergence and higher success rates compared to traditional imitation learning methods.

Preference-Regret Minimization for Data-Constrained Robot Policy Learning

Introduction and Motivation

Robotic Reinforcement Learning from Demonstrations (RLfD) is fundamentally hindered by the scarcity and non-i.i.d. nature of expert trajectories in practical settings. Imitation learning (IL) algorithms, particularly behavior cloning (BC), suffer from compounding (cascading) errors due to distribution shift between expert and learned policy behaviors, often resulting in significant performance degradation when faced with limited expert data. This work addresses these deficiencies through a self-imitation framework—termed Master Your Own Expertise (MYOE)—centered around the joint estimation of internal representations and future goal preferences, enabling robust neurorobot policy optimization under stringent data constraints.

Model Overview: Active Preference-Guided World Modeling

The agent architecture leverages the active inference framework for integrated perception-action loops, constructing a generative world model that simulates both environmental dynamics and an explicit trajectory of "preferences" guiding task behavior.

Figure 1: The proposed architecture encodes observations into hidden states, generates imagined future states and preferences via the QMoP-SSM, and uses these for policy/value learning.

Queryable Mixture-of-Preferences State Space Model (QMoP-SSM)

QMoP-SSM extends classical state-space modeling by partitioning latent state learning into two synergistic processes:

• Representation Learning: Neural posterior and prior models infer hidden world states ($s^o_t$) from observations and past actions, facilitating temporally coherent world prediction.
• Preference Learning: Parallel models predict desired preference states ($s^p_t$) conditioned on provided and learned goal specifications, modeling both observed and imagined expert behaviors over trajectory rollouts.

A mixture modeling scheme is introduced to alleviate the limitations imposed by multi-modality in the set of plausible trajectories toward a goal, thereby increasing mode coverage, exploration, and policy robustness.

Figure 2: QMoP-SSM features parallel learning of representations and preferences, supporting both standard world modeling and explicit goal-driven rollouts.

Behavioral Optimization via Preference Regret

Policy training departs from naive imitation. Instead, the framework computes an intrinsic reward based on "preference regret," quantifying the discrepancy between rewards for actual and preference-driven trajectories. Concretely, the policy seeks to minimize regret—the difference between realized return and predicted preference-driven reward—while still maximizing actual environmental reward and maintaining high entropy for exploration.

The optimization pipeline integrates generalized advantage estimation (GAE-$\lambda$), mixtures of imagined rollouts for planning, and actor-critic updates based on preference-guided value targets. This design enables the agent to:

• Exploit high-quality demonstrations when informative
• Avoid blindly mimicking suboptimal expert behavior when direct environmental rewards are superior
• Circumvent cascading error by grounding online exploration in self-improving preference estimation, not only offline demonstrations

Empirical Results: Simulated and Physical Robotic Tasks

The experimental evaluation covers 32 tasks in Franka Kitchen, Meta-World, and Robosuite benchmarks, all with sparse reward signals and with policies trained on only 5 expert demonstration episodes per task. The results are benchmarked against DreamerV3+BC, PPO+BC, LAIfO, and various behavior cloning variants, all operating with controlled model capacity and interaction budgets.

The MYOE framework consistently achieves higher final success rates and faster convergence across the majority of manipulation/control tasks, including pronounced margins on sparse-reward environments and tasks with significant distribution/goal perturbations engineered to induce cascading error in typical BC/PPO variants.

Figure 3: Cumulative reward as a function of training steps demonstrates rapid, stable convergence of preference regret-optimized agents compared to RLfD baselines.

Ablative evidence from real-robot experiments (7bot/6-DOF and PX100/4-DOF arms) confirms the superiority of MYOE, with significant gains in both mean and variance of episodic success rates under acute data and evaluation constraints.

Figure 4: Real robot deployments of MYOE in “reach” (7bot) and “block picking” (PX100) tasks, highlighting successful real-world transfer under limited demonstration data.

Theoretical and Practical Implications

The explicit formulation of preference regret as an augmentation to intrinsic motivation, rooted in contemporary extensions to bandit regret and expected free energy, enables the agent to learn in a manner more closely approximating optimal exploration-exploitation than standard IL approaches. Preference-guided behavior operates both as a stabilizer (aligning with distributional properties of successful expert rollouts) and as a safety valve (permitting deviation when evidence favors alternative strategies).

From a practical standpoint, the method offers robust adaptation in high-dimensional, partially observed robotic domains without requiring large-scale demonstration curation or costly reward engineering. Empirical and theoretical sections jointly demonstrate that the approach bridges gaps left by BC and adversarial IL under sparse, noisy, or suboptimal supervision.

Conclusion

This framework demonstrates that principled integration of preference modeling, mixture-based goal prediction, and regret-minimizing intrinsic rewards can make data-efficient neurorobot policy optimization viable in regimes that elude traditional RLfD and IL methods. Future extensions—such as improved latent trajectory inference via probabilistic filters or enhanced pixel-level abstraction learning—can further bolster real-world applicability in high-complexity settings where perception and control remain tightly coupled.