Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

Abstract: Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution. The standard remedy is to collect multi-view teleoperation data for every failure case, but this scales poorly in both cost and time. We introduce Pose6DAug, a failure-driven data augmentation framework that turns a policy's own successful episodes into targeted demonstrations for its failure modes, without any new data collection. Our key insight is that each successful episode already encodes a physically valid action trajectory together with calibrated multi-view observations. By swapping only the manipulated object while preserving this trajectory, we obtain new and physically grounded demonstrations. However, naive 2D video editing breaks multi-view consistency and physical plausibility, particularly under heavy occlusion and egocentric viewpoints. Our method instead operates directly in 3D, anchoring the target object with an explicit mesh driven by a temporally coherent 6D pose trajectory, ensuring geometrically consistent renderings across all camera views. Fine-tuning a VLA on data augmented by our method improves success rates by 16.5% relative to the state-of-the-art baseline on novel objects, while preserving in-distribution performance. These results show that multi-view and physically consistent augmentation is a practical path to scalable VLA generalization.

• The paper presents a novel framework using 3D pose extraction and mesh reconstruction for physically plausible object swapping in robot data augmentation.
• It employs an object-centric augmentation pipeline incorporating rotation, translation, and scaling to ensure multi-view consistency and physical plausibility.
• Empirical results demonstrate a 16.5% relative improvement in success rate and robust recovery of challenging OOD objects compared to 2D and simulation-based methods.

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

Overview

Pose6DAug introduces a 3D-grounded framework for robot data augmentation aimed at overcoming generalization failures in Vision-Language-Action (VLA) manipulation policies when faced with novel object instances. The method leverages existing successful robot episodes by substituting manipulated objects with new target objects using explicit 3D mesh extraction and a temporally coherent 6D pose trajectory, maintaining strict physical plausibility and multi-view consistency. This approach departs from 2D video editing and simulation-only augmentation, offering significant improvements in out-of-distribution object handling while being simulation-free and maintaining data scalability.

Figure 1: Failure-driven object-swap augmentation pipeline, transforming failed episodes into successful demonstrations for novel objects by swapping objects in successful source rollouts.

Methodology

Pose6DAug is built on three main modules: target mesh reconstruction with 6D pose extraction, object augmentation via geometric perturbations, and 3D mesh-pose-guided multi-view video composition.

• Target Mesh Reconstruction & Pose Extraction: A single observation of the target object is used with an off-the-shelf image-to-3D model (SAM3D) to reconstruct the 3D mesh. The episode’s pose sequence (in $SE(3)$ per timestep) is extracted from either simulator or external pose estimators, with all objects operating in a shared canonical coordinate frame to preserve gripper interaction configurations.
• Object-centric Data Augmentation: Geometric perturbations are applied to the target mesh before rendering:
  • Rotation: Randomly rotating or flipping the mesh to expose the policy to diverse, plausible object orientations.
  • Translation: Shifting the mesh along the gripper's approach axis to vary contact offsets.
  • Scaling: Adjusting the target mesh to remain within the feasible range for the gripper, ensuring plausible manipulation post-substitution.
  • 

  • Figure 2: Examples of applied object-centered augmentations—rotation, translation, scaling, and their composition—demonstrably maintain physical plausibility and 3D consistency.

• Multi-view Video Composition: For each timestep and view, the background plate is generated by removing the source object (using segmentation masks and inpainting). The transformed target mesh is rendered at the kinematically coupled pose, and the robot/gripper layers are composited to guarantee correct occlusion and interaction. Because mesh rendering is driven by a shared 6D trajectory, all synthesized videos are guaranteed to be consistent across camera perspectives and over time.

Empirical Results

Pose6DAug undergoes extensive benchmarking on RoboCasa365, emphasizing failures arising from OOD object geometries and leveraging the Counter-to-Cabinet pick-and-place task as a challenging scenario. The methodology is compared against:

• VACE: 2D video editing via multimodal diffusion transformers, applied per camera view.
• MimicGen: Simulation-based data generation by trajectory retargeting, requiring extensive rollout filtering and simulator access.

Results are analyzed by success rate, turnover ratio (fraction of previously failed instances successfully manipulated post-augmentation), and per-object recovery rate, using GR00T-1.5 as the foundation model.

Numerical Highlights

• On failure episodes (i.e., OOD objects where the base VLA policy fails 100%), Pose6DAug outperforms VACE and MimicGen by a relative 16.5% success rate improvement and achieves a 24.5% turnover ratio, a significant margin over the baselines.
• When training solely on augmented data for the hardest OOD samples (no support from the original dataset), Pose6DAug attains 21.2% success versus VACE's 15.0% and MimicGen’s 5.7%.
• Aggregate metrics confirm that Pose6DAug enhances OOD performance (42.9% vs. 36.9% for the base policy) without degrading in-distribution performance.
• The method robustly recovers 7 out of 8 hardest OOD object classes, in contrast to 5 (VACE) and 2 (MimicGen), demonstrating explicit robustness gains attributable to trajectory-level physical plausibility and geometric consistency.

  Figure 3: Qualitative augmentation comparisons highlighting pose/identity consistency and physical plausibility; VACE exhibits multi-view inconsistencies while MimicGen's rollouts may lack feasible grasps for challenging geometries.

Qualitative Assessment

Pose6DAug consistently generates visually and kinematically coherent robot-object interactions across all multi-view configurations. In contrast, VACE's per-view 2D edits result in identity and geometry drift, and MimicGen's simulation-constrained approach suffers from low coverage due to the inability to generate valid trajectories for complex object geometries or occlusion scenarios.

Failure Modes and Limitations

Pose6DAug’s main limitations are associated with the affordance mismatch between source and target objects—significant variations in scale or geometry can result in physically implausible grasps despite the canonical pose coupling. The framework also inherits potential artifacts from external 3D mesh or inpainting models, and compositional rendering can neglect photorealistic interaction effects (e.g., shadows, reflections).

Figure 4: Typical Pose6DAug failures stem from mesh scale and geometry mismatches, producing visually implausible robot-object intersections.

Implications and Future Directions

Practically, Pose6DAug’s scalable, simulation-free data augmentation pipeline circumvents major bottlenecks in VLA robot learning: cost, diversity, and multi-view expertise. By combining generative 3D object substitution with rigorous kinematic grounding, it enables targeted failure recovery and robust OOD handling not previously attainable by 2D generative or simulation-only strategies.

• Theoretical Implications: The work advocates that robust robot policy generalization requires explicit 3D physical grounding throughout the augmentation and training pipeline. It demonstrates that geometric trajectory alignment and view consistency can be algorithmically enforced to provide effective training signals, triggering new design paradigms for multimodal robot learning systems.
• Future Directions: Research thrusts include automated affordance-aware substitution algorithms, mesh-inpainting co-design for photorealistic synthesis, and generalization to unaligned object categories and multi-object manipulation scenes. Neural relighting and dynamic interaction rendering could further enhance realism.

Conclusion

Pose6DAug provides a technically rigorous, physically and geometrically consistent augmentation framework for robotic manipulation, substantiated by both quantitative and qualitative advances over 2D and simulation-based alternatives. Its integration of explicit 3D reasoning, mesh-based object swapping, and multi-view trajectory alignment establishes a new baseline for scalable, generalizable robot learning infrastructure.