- The paper proposes 1001 DEMOS, a trajectory-level augmentation method that generates diverse and physically consistent action-view pairs from limited demonstrations.
- It employs a fisheye-based 3D scene reconstruction and optimized trajectory planning to enhance data efficiency and policy robustness in robotic manipulation tasks.
- Empirical evaluations demonstrate up to a 56% improvement in task success rates and 100% success in challenging obstacle scenarios, highlighting its practical impact.
Detailed Summary of "One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies"
Introduction and Motivation
The paper introduces 1001 DEMOS, a trajectory-level data augmentation framework for eye-in-hand visuomotor policy learning. Visuomotor policies, especially for robotic manipulation, typically suffer from sensitivity to minor changes in scene configuration, with catastrophic failures arising from out-of-distribution observations caused by small variations in the robot’s pose or the environment. Conventional solutions demand labor-intensive data collection with numerous demonstrations covering the space of possible configurations. This work addresses this inefficiency by proposing a principled method to mechanistically expand the available training data from a small set of demonstrations, producing visually and physically diverse observation-action pairs while maintaining their correspondence and adhering to physical constraints.
Methodology
Fisheye-Based 3D Scene Reconstruction
1001 DEMOS leverages eye-in-hand data collection using a portable parallel gripper equipped with a single fisheye camera. The core pipeline encompasses three tightly integrated modules:
- 3D Scene Reconstruction: Utilizing the UMI dataset format and ARKit VIO, high-fidelity point clouds are constructed from fisheye image sequences through COLMAP, permitting robust motion planning. Critically, the paper introduces a modified 3D Gaussian Splatting (3DGS) framework that integrates a KB8-based fisheye ray sampler. This allows rendering photorealistic images from arbitrary novel viewpoints, reflecting the extensive FoV characteristic of fisheye optics.
- Trajectory Optimization: The pipeline generates two types of augmented trajectories:
- Free-space trajectories: Smooth, collision-free paths initialized from randomly sampled start poses within the workspace, converging to the pre-contact pose of the expert demonstration.
- Obstacle-avoiding trajectories: By augmenting the reconstructed scene with obstacle geometries (from Objaverse), the method synthesizes collision-avoidant behaviors not present in the original demos, ensuring feasibility through strict TSDF-based constraints and convex hull checks.
The optimization combines funnel loss (to preserve convergence), render consistency losses, collision penalties, and smoothness regularization, jointly ensuring spatial diversity, action realism, and view-rendering fidelity.
- Paired Observation Generation: For every optimized trajectory, novel-view observations are generated using the fisheye-extended 3DGS representation, ensuring each action sequence is tightly coupled with visually consistent image sequences, both in the presence and absence of added obstacles. Gripper overlays and segmentation (via SAM2) maintain realism and domain fidelity.
Policy Learning
Augmented datasets are subsequently used to train Diffusion Policies with CLIP-pretrained ViT-B/16 vision encoders. The training set comprises both original and synthesized action-view trajectories, improving domain coverage without the exhaustive cost of manual data collection.
Empirical Evaluation
Simulation Results
The approach is benchmarked on the RoboMimic "square" peg-in-hole task. Key findings include:
- Performance: With identical numbers of original demonstrations, action-view augmented policies attain success rates close to those achieved with ideal ground-truth renderings (oracle). The gap is only 8% in low-data and 11% in high-data regimes.
- Data Efficiency: Augmentation yields a 56% improvement in task success rate over policies trained without it.
- Ablation: Methods that only perturb actions (but not visuals) or perform single-step NeRF-based augmentation (e.g., SPARTN) underperform relative to 1001 DEMOS, particularly in tasks requiring spatially extended, collision-aware behaviors.
Real-World Results
On a real-world cup serving task using UMI-collected demonstrations:
- Free-space augmentation achieves a 55% success rate boost over unaugmented baselines for tasks with OOD initial states.
- Obstacle-augmentation allows for 100% success (vs. 10% for free-space alone and 5% for no augmentation) in scenarios featuring unseen obstacles, demonstrating the method's efficacy in transferring implicitly learned collision avoidance to policy execution.
- Challenging scenarios with complex obstacles reaffirm these results, where only obstacle-augmented policies succeed (100% vs. 0% for others).
Scalability and Trade-off Analyses
- Diversity vs. Rendering Fidelity: The authors empirically determine that a rotation cone of 50° maximizes both novel view coverage and rendering quality; further increases degrade visual fidelity without substantial gain.
- Optimal Use: Policies are most robust when action-view augmentation is performed for the full trajectory (not just per-step), and when both actions and views are jointly varied in the augmentation pipeline.
Implications and Future Impact
The framework substantially pushes the boundary on data-efficient visuomotor learning:
- Practical Utility: Robotic systems can now be robustified to unseen states and obstacles using just a handful of real-world demonstrations, minimizing on-robot training or dangerous exploration. The approach is highly compatible with offline learning paradigms and policy pretraining for real-world deployment.
- Behavior Extension: The method not only regularizes against minor pose variations and view shifts but directly operationalizes collision avoidance from demonstration alone, an ability previously requiring explicit programming or exhaustive teleoperation.
- Scalable Data Generation: By automating the expansion from one demonstration to thousands of high-fidelity action-view pairs, the pipeline reduces annotation burden while preserving physical feasibility, crucial for scaling robot learning toward generalist agents.
Limitations and Prospects
Current constraints include:
- Viewpoint Coverage: The single-camera setup enforces a static-scene assumption and limits novel view synthesis to neighborhoods of original trajectories. Larger viewpoint excursions lead to rendering artifacts due to multi-view inconsistency inherent in 3DGS.
- Kinematic Transfer: Generated trajectories are not explicitly checked for downstream robot kinematic feasibility, potentially limiting hardware transferability—future work could incorporate robot-specific constraints in the trajectory optimizer.
- Dynamic Scenes: Extension to reconstructing and synthesizing dynamic or deformable scene elements will require advances in dynamic radiance fields or sensor fusion with ToF/multicam rigs.
The authors propose further investigation into few-shot 3DGS, multi-camera setups, and employing view-consistent 2DGS or configuration-space-aware planning to further enhance generalization, view realism, and embodiment transfer.
Conclusion
1001 DEMOS presents a robust, physically grounded action-view data augmentation framework that dramatically increases the spatial coverage and behavioral robustness of visuomotor policies through synthetic, yet realistic, augmentation of both actions and observations. The approach achieves marked improvements in both simulated and real-world environments, particularly in generalization to OOD scenes and obstacle avoidance. This work provides a compelling direction for scaling robot learning with minimal manual annotation and opens avenues for further data-driven policy generalization via principled, scene-aware synthetic data generation.