Task Editing for Generalizable 3D Visuomotor Policy Learning

Abstract: 3D visuomotor policies offer a promising direction for complex robotic manipulation, as depth maps and point clouds provide rich geometric information for spatial reasoning. However, their success often depends on large-scale real-world demonstrations, which are costly and time-consuming to collect. To this end, existing methods commonly use demonstration generation strategies to improve data efficiency by applying object-centric transformations to human-collected demonstrations, such as varying object poses or scales. While effective for local variation, these transformations largely preserve the original scene structure and skill sequence, limiting their ability to synthesize diverse scene-skill-object combinations for complex tasks. In this paper, we propose Task-Edit, a novel demonstration generation framework that generates diverse trajectories from a task-centric editing perspective. The key insight of Task-Edit is to decompose a task into scene, skill and object components, and flexibly recombine them. In this way, Task-Edit enables scalable demonstration generation and significantly improves generalization for long-horizon manipulation tasks. We evaluate Task-Edit through extensive real-world experiments and demonstrate three advantages: (1) Effectiveness: Task-Edit significantly improves 3D visuomotor policies across various real-world tasks and robot embodiments. (2) Generalizability: Task-Edit improves model generalization across different scenario setups. (3) Applicability: Task-Edit enables models to handle scenarios that are difficult to collect in the real world, including disturbance resistance, obstacle avoidance and unseen cluttered scenes.

• The paper presents Task-Edit, a novel framework that decomposes tasks into scene, skill, and object components to generate diverse training data.
• It achieves significant performance gains with a mean success rate of 81.3% on real tasks and 93.7% in simulated in-distribution settings.
• The approach reduces manual data collection by enabling flexible editing and recombination, boosting robustness in out-of-distribution scenarios.

Task-Edit: A Task-Centric Framework for Generalizable 3D Visuomotor Policy Learning

Overview and Motivation

This paper introduces Task-Edit, a novel data generation framework for 3D visuomotor policy learning. The motivation stems from the high cost and limited diversity of real-world demonstrations for training policies capable of complex, long-horizon manipulation tasks. Existing demonstration generation methods focus largely on object-centric variations (e.g., changing object poses), with limited ability to synthesize diverse scene-skill-object combinations. Task-Edit addresses this by decomposing tasks into independent scene, skill, and object components, enabling independent, targeted editing and flexible recombination for scalable generation of diverse training trajectories.

Task-Edit Framework

The Task-Edit framework operates through a Real2Sim2Real paradigm and consists of the following key stages:

• Task Decomposition: Demonstrations are factorized into scene (object composition and environment layout), skill (manipulation primitives, sequence, cyclic/collaborative nature), and object (6-DoF pose, geometry, texture) attributes.
• Real2Sim Process: Human demonstrations are reconstructed in simulation using foundation models for object segmentation, 6-DoF trajectory estimation, and image-based 3D asset generation. This enables accurate replay and sets the foundation for independent editing.
• Component Editing:
  • Object Editing: Skill transfer across object categories via keypoint matching and spatial transformations. New trajectories are synthesized by modifying object pose, geometry, and texture, with action trajectories updated via SE(3) transformations.
  • Skill Editing: Trajectory decomposition into motion and skill segments. Skill sequences are manipulated based on Skill-DAG representations, permitting valid reordering and re-combination while respecting physical and task constraints, with constraint rules enforced by collision-checking and dependency analysis.
  • Scene Editing: Diversity added by altering workspace backgrounds and object compositions. Scene generation uses divide-and-conquer occupancy mapping for distractor placement, maintaining collision-free layouts and object diversity.
• Sim2Real Process: To mitigate the sim-to-real gap, Task-Edit employs a depth restoration pipeline, leveraging segmentation and completion foundation models to repair noisy or incomplete point clouds from real sensors. Outlier filtering and real-time mask extraction further enhance observation quality.

  Figure 1: Architecture of the Task-Edit framework highlighting scene, skill, and object decomposition and the editing mechanisms for each component.

Experimental Analysis

The efficacy of Task-Edit is demonstrated through extensive experiments involving various robot embodiments (dual-arm, single-arm, humanoid), a suite of single- and dual-arm manipulation tasks, and both real and simulated environments.

Quantitative Results

• Across nine real-world tasks, models trained on Task-Edit-generated trajectories outperform those trained on both DemoGen-generated and human-collected data, with Task-Edit yielding a mean success rate of 0.813 compared to 0.656 for DemoGen and 0.719 for human data, even though Task-Edit builds from a single demonstration per task.
• In simulation, Task-Edit further surpasses baselines, achieving a mean success of 0.937 (in-distribution) and 0.887 (out-of-distribution), with strong results maintained as the number of generated trajectories and object combinations increases.
• Robustness in generalization: Task-Edit-founded policies display enhanced performance in settings with unseen objects or novel skill-object combinations, with the gap widening as data diversity scales.

  Figure 2: Time-series visualization contrasting the execution process of DP3 trained on DemoGen vs. Task-Edit data on the ABB Sorting task.

  

  Figure 3: (a) Task-Edit enables stable grasps in in-distribution settings where DemoGen-trained models fail. (b) With greater trajectory diversity, Task-Edit-trained models can grasp unseen small-scale objects.

Qualitative Observations and Ablation

• Qualitative visualizations demonstrate improved policy robustness and stability, particularly for fine-grained manipulations and in the presence of distractors or clutter.
• Ablation studies reveal that the depth restoration module significantly reduces the sim-to-real gap and that Task-Edit remains robust to pose estimation noise.
• Human time cost is sharply reduced: 14.6% of the time required for equivalent manual collection, with a 93.26% improvement in data collection efficiency for long-horizon tasks.

Handling Complex Manipulations

• Skill Sequence Editing: Task-Edit-generated data supports the learning of policies that handle varied and cyclic skill sequences, outperforming DemoGen across both dual-arm/long-horizon and single-arm/short-horizon tasks (mean: 0.895 vs. 0.684).
• Cyclic Manipulation: From a single demonstration, Task-Edit enables successful policy learning for repeated actions (e.g., shaking, stirring) of varying cycle lengths and novel object categories.

Challenging Scenarios

• Task-Edit-generated data enables robust policy deployment in dynamic, cluttered, and obstacle-rich scenarios, where models trained without explicit exposure to such scenarios nearly always fail.

Implications and Future Directions

Task-Edit represents a substantial step toward efficient and scalable demonstration generation for large-scale, generalizable visuomotor policy learning. The explicit decoupling and independent editing of scene, skill, and object attributes allow for richer compositional diversity, which is critical for out-of-distribution robustness and sample complexity reduction.

Practically, Task-Edit provides an avenue to bypass manual enumeration of skill/object/scene configurations, positioning it as a valuable tool for both academic and industrial applications where exhaustive data collection is impractical.

From a theoretical standpoint, Task-Edit reframes the problem of data generation as a compositional, task-level operation, opening future research in automated task synthesis, transfer to non-rigid and articulated objects, and further sim-to-real reduction. Integrating language-conditioned policy learning or generative models for novel task derivation ("task generation") is an anticipated evolution, as is the incorporation of richer object categories and human-in-the-loop adaptation to handle the rare edge cases not captured by current 3D generation methods.

Conclusion

Task-Edit establishes a new protocol for data-efficient, generalizable 3D visuomotor policy learning via systematic task decomposition and component-centric editing. Empirically validated across diverse tasks and robots, it enables state-of-the-art policy performance and generalization from minimal demonstrations, substantially lowering the barrier for deploying robust manipulation policies in complex environments. Task-Edit has clear implications for scaling robotic learning and serves as a foundation for future research on automated, compositional task and skill acquisition.