SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Exploration (2509.19292v1)

Abstract: Intelligent agents progress by continually refining their capabilities through actively exploring environments. Yet robot policies often lack sufficient exploration capability due to action mode collapse. Existing methods that encourage exploration typically rely on random perturbations, which are unsafe and induce unstable, erratic behaviors, thereby limiting their effectiveness. We propose Self-Improvement via On-Manifold Exploration (SOE), a framework that enhances policy exploration and improvement in robotic manipulation. SOE learns a compact latent representation of task-relevant factors and constrains exploration to the manifold of valid actions, ensuring safety, diversity, and effectiveness. It can be seamlessly integrated with arbitrary policy models as a plug-in module, augmenting exploration without degrading the base policy performance. Moreover, the structured latent space enables human-guided exploration, further improving efficiency and controllability. Extensive experiments in both simulation and real-world tasks demonstrate that SOE consistently outperforms prior methods, achieving higher task success rates, smoother and safer exploration, and superior sample efficiency. These results establish on-manifold exploration as a principled approach to sample-efficient policy self-improvement. Project website: https://ericjin2002.github.io/SOE

• The paper introduces the SOE framework that leverages on-manifold exploration to safely improve robot policies using a compact latent space.
• It employs a variational information bottleneck to disentangle task-relevant factors, ensuring stable, diverse, and sample-efficient learning.
• Experimental results on real-world and simulation tasks demonstrate significant performance gains and robustness compared to baseline methods.

Sample-Efficient Robot Policy Self-Improvement via On-Manifold Exploration

Introduction and Motivation

The paper introduces SOE, a framework for sample-efficient robot policy self-improvement through on-manifold exploration. The central challenge addressed is the inefficiency and safety risks of exploration in high-dimensional, continuous action spaces typical of robotic manipulation. Existing methods, such as random action perturbation or noise injection, often result in unsafe, erratic, and temporally inconsistent behaviors, especially when actions are chunked over time. SOE constrains exploration to the manifold of valid actions by learning a compact latent representation of task-relevant factors, ensuring that exploration remains both safe and effective. This approach is designed to be modular, allowing seamless integration with arbitrary policy models without degrading their performance.

Figure 1: Overview of SOE. By constraining exploration to the manifold of valid actions, SOE generates diverse yet temporally coherent behaviors, enabling structured and efficient exploration. The collected rollout data is used to refine the policy, leading to efficient self-improvement.

Methodology

On-Manifold Exploration via Variational Information Bottleneck

SOE leverages a variational information bottleneck (VIB) to learn a latent space $Z$ that captures only the information essential for policy execution. The encoder $p_\theta(z|o)$ and decoder $q_\phi(a|z)$ are trained to maximize mutual information between $Z$ and actions $A$, while minimizing mutual information between $Z$ and observations $O$, controlled by a hyperparameter $\beta$:

$\max_\theta I(Z;A) - \beta I(Z;O)$

This yields a compact, disentangled latent space where exploration can be performed by sampling $z_t \sim \mathcal{N}(\mu_t, (\alpha \sigma_t)^2)$, with $\alpha$ controlling the exploration scale. This ensures that generated actions remain on the task manifold, avoiding unsafe or unrealistic behaviors.

Dual-Path Architecture and Plug-in Integration

SOE is implemented as a plug-in module with a dual-path architecture:

• Base Path: Standard policy execution using the original observation encoder and action decoder.
• Exploration Path: Observation embeddings are encoded into the latent space, stochastically perturbed, and decoded back for diverse action proposals.

Both paths are optimized jointly, with the base path using standard imitation loss and the exploration path using the VIB loss. This design ensures that exploration does not compromise the base policy's performance and allows for end-to-end training with minimal overhead.

Figure 2: Dual-Path Architecture of SOE. The observation embedding is processed through two parallel paths: the base path for stable policy execution and the exploration path for generating diverse actions. Each path outputs distinct noise predictions and is optimized with a separate loss function.

User-Guided Steering

The disentangled latent space enables user-guided exploration. By identifying effective latent dimensions via signal-to-noise ratio (SNR), users can steer exploration toward preferred behaviors by perturbing specific dimensions. Farthest point sampling (FPS) is used to select diverse action proposals for user selection, further improving sample efficiency and controllability.

Figure 3: Visualization of action proposals from different latent dimensions. Each subfigure corresponds to a specific latent dimension, with the title indicating the dimension index and its SNR value (measured in dB). Activating high-SNR dimensions leads to meaningful and diverse action variations, while low-SNR dimensions yield negligible diversity.

Experimental Evaluation

Real-World Manipulation Tasks

SOE was evaluated on three real-world tasks (Mug Hang, Toaster Load, Lamp Cap) using a Flexiv Rizon4 arm and Robotiq 2F-85 gripper. Policies were initialized with limited human demonstrations and improved via self-collected rollouts.

Figure 4: Overview of real-world manipulation tasks: (a) Mug Hang, (b) Toaster Load, (c) Lamp Cap.

SOE consistently outperformed baselines (Diffusion Policy and SIME) in terms of task success rate, sample efficiency, and motion smoothness. Notably, SOE achieved an average relative improvement of 50.8% on real-world tasks after a single round of self-improvement. User-guided steering further boosted performance, achieving up to 62% relative improvement on Lamp Cap.

Figure 5: Task success rate over rounds. SOE consistently improves policies, whereas baselines perform unstably.

Figure 6: Number of rollouts and exploration success rate over rounds, averaged across tasks. SOE achieves higher Pass@5 rates while requiring fewer environment interactions compared to baselines, indicating more efficient and effective exploration.

Simulation Benchmark

On RoboMimic simulation tasks (Lift, Can, Square, Transport), SOE demonstrated robust multi-round improvement, with stable gains in success rate and reduced rollout requirements. Baselines exhibited unstable or declining performance in certain tasks, highlighting the advantage of structured on-manifold exploration.

Ablation Studies

Ablation experiments on the Can task revealed:

• Removing the KL term in the VIB loss reduced final success rate by 5.25%, confirming the importance of compact latent representations.
• Exploration scale $\alpha$ and KL weight $\beta$ critically affect diversity and stability; optimal values are task-dependent.
• The number of effective latent dimensions correlates with task complexity and remains invariant to the nominal latent dimension, suggesting the existence of a policy-agnostic intrinsic dimension for each task.

Figure 7: Task success rate under ablation. Ablations on the Can-20 task demonstrate the impact of KL regularization, noise scale, and latent dimension on policy improvement.

Theoretical and Practical Implications

SOE establishes on-manifold exploration as a principled approach for sample-efficient robot policy self-improvement. By constraining exploration to the manifold of valid actions, SOE ensures safety, diversity, and effectiveness, addressing key limitations of prior methods. The modular plug-in design enables broad applicability across policy architectures, and the disentangled latent space facilitates both autonomous and user-guided exploration.

The identification of intrinsic task dimensions via SNR analysis provides a foundation for future research on task complexity and representation learning in robot policies. The demonstrated sample efficiency and robustness suggest that SOE can significantly reduce the cost and risk of real-world robot learning, making autonomous self-improvement feasible in practical deployments.

Future Directions

Potential future developments include:

• Extending SOE to multi-agent and collaborative manipulation scenarios.
• Integrating SOE with reinforcement learning for hybrid policy improvement.
• Investigating automated latent dimension selection and adaptive exploration scaling.
• Exploring transfer learning of latent manifolds across related tasks.

Conclusion

SOE presents a structured, sample-efficient approach to robot policy self-improvement via on-manifold exploration. By learning a compact, task-relevant latent space and constraining exploration to valid action manifolds, SOE achieves superior effectiveness, safety, and efficiency compared to prior methods. The framework's modularity and support for user-guided steering further enhance its practical utility. These results underscore the importance of structured exploration in advancing autonomous robot learning and open avenues for future research in scalable, self-improving robotic systems.