Constraint-Preserving Data Generation (CP-Gen)

• Constraint-Preserving Data Generation (CP-Gen) is a method that decomposes a single expert demonstration into free-space and skill segments, encoding task-relevant keypoint constraints.
• It applies geometry-aware optimization by sampling pose and geometry transforms to generate valid robot demonstrations under novel object shapes and poses.
• Empirical results show CP-Gen achieves up to 83% real-world success, outperforming baseline methods and enabling zero-shot sim-to-real policy transfer.

Constraint-Preserving Data Generation (CP-Gen) is a method for robot demonstration generation and visuomotor policy learning that uses a single expert trajectory to generate robot demonstrations containing novel object geometries and poses. The method first decomposes expert demonstrations into free-space motions and robot skills, then formulates each skill as a keypoint-trajectory constraint defined relative to a task-relevant object. For a new scene, CP-Gen samples pose and geometry transforms for each task-relevant object, applies those transforms to the object and its associated keypoints or keypoint trajectories, re-solves robot joint configurations so that keypoints on the robot or grasped object track the transformed reference trajectory, and motion plans a collision-free path to the first optimized joint configuration. The generated demonstrations are then used to train closed-loop visuomotor policies that transfer zero-shot to the real world and generalize across variations in object geometries and poses (Lin et al., 5 Aug 2025).

1. Motivation and problem setting

Large-scale demonstration data has powered key breakthroughs in robot manipulation, but collecting that data remains costly and time-consuming. The formulation targeted by CP-Gen begins from the observation that collecting demonstrations is extremely expensive, and that even when many demonstrations exist, policies trained on them typically do not generalize far beyond the exact objects and poses seen during collection.

Prior data-generation approaches such as MimicGen, SkillGen, and IntervenGen address this by taking a small set of teleoperated demonstrations and replaying or pose-transforming them under new object poses. In that setting, if an $\mathrm{SE}(3)$ transform $T$ is applied to an object, and the same transform is applied to the robot’s end-effector trajectory, the result is a valid demonstration in the new pose so long as the object’s shape and size remain fixed. CP-Gen is explicitly designed to remove that restriction. A common misconception is that pose transformation alone is sufficient whenever task instances vary; CP-Gen treats this as a pose-only regime, not a geometry-aware one.

The motivating counterexample given for the method is that a wine glass with a tall, narrow stem cannot be hung on a spiral rack by simply rotating and translating the end-effector trace collected on a short, thick glass. CP-Gen’s central idea is therefore to generate geometry-aware demonstrations from as few as one expert trajectory by decomposing the trajectory into free-space and skill segments, encoding each skill segment as a keypoint-trajectory constraint anchored in the task object’s frame, and re-solving for robot joint angles that satisfy those constraints under novel object poses and shapes.

2. Demonstration decomposition and state representation

CP-Gen assumes a single expert demonstration given as a time-indexed trajectory

$\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$

where $o_t$ contains sensor data, including cameras and proprioception $q_t$, and $a_t$ contains end-effector pose $a_{\mathrm{eef},t}\in\mathrm{SE}(3)$ plus gripper action $a_{\mathrm{grip},t}$.

The trajectory is parsed into alternating segments:

$$\tau_{\mathrm{src}} = [\tau^{(1)}_{\mathrm{motion}},\, \tau^{(1)}_{\mathrm{skill}},\, \tau^{(2)}_{\mathrm{motion}},\, \tau^{(2)}_{\mathrm{skill}},\dots].$$

Free-space motion segments, $\tau^{(i)}_{\mathrm{motion}}$, can be replaced by standard collision-free point-to-point motion planning. Skill segments, $T$0, involve contact, insertion, or other fine-manipulation that requires preserving a relative relationship between robot and task objects.

For each skill segment, CP-Gen defines a set of actor keypoints

$T$1

expressed in the local frame $T$2 of either the robot’s gripper or an object being grasped. These keypoints are manually annotated, with the paper noting that they could be detected automatically in future work. This representation shifts the invariant object-robot relationship away from a raw end-effector trace and toward a set of task-relevant geometric correspondences.

3. Keypoint-trajectory constraints and geometry-aware optimization

For each skill segment, CP-Gen extracts a reference trajectory for the actor keypoints, but expressed in the task object’s local frame $T$3. If the task object’s world pose at time $T$4 is $T$5 and the actor pose is $T$6, then the reference keypoints in the object frame are

$T$7

Collecting these over time yields the keypoint-trajectory $T$8 in the task object frame.

When the scene changes, CP-Gen introduces both a new object-to-world pose $T$9 and a geometry transform $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$0 in $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$1, for example a nonuniform scale. The transformed target keypoints are

$\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$2

If the actor itself is geometrically transformed, local keypoints are updated by

$\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$3

At each skill time $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$4, CP-Gen solves for a robot joint configuration $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$5 so that the forward kinematics map tracks the transformed target:

$\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$6

where $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$7 enforces temporal smoothness in joint-space. Solving this optimization produces a new joint trajectory $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$8 that preserves the original keypoint-trajectory constraint under novel object geometry and pose.

This formulation is the distinctive technical move in CP-Gen. Prior pose-only methods preserve an end-effector trajectory under rigid transformation; CP-Gen instead preserves a trajectory of task-relevant keypoints defined in an object frame and transformed by geometry-aware operators. This suggests a more direct treatment of shape variation in tasks where the relevant invariant is not the absolute gripper trace but the relation between functional object geometry and the actor.

4. Generation pipeline and motion planning

For each new scene, CP-Gen performs three top-level operations. It samples a random pose transform $\tau_{\mathrm{src}} = \{(o_t, a_t)\}_{t=1}^H,$9 and a random geometry transform $o_t$0 for each task object. It applies these transforms to the object’s mesh and to all extracted $o_t$1 as in Eq. (2). It then solves the keypoint-matching optimization in Eq. (3) for every skill segment to produce a candidate joint-space trajectory.

In parallel, each free-space motion segment is replaced by a collision-free motion plan between the ending joint angle of the previous skill and the start joint angle of the next skill segment. If one skill segment ends at $o_t$2 and the next begins at $o_t$3, CP-Gen calls a planner such as OMPL’s RRT-Connect to find

$o_t$4

The method therefore stitches optimized skill segments with plan-based motion segments to form a full demonstration in joint space for the new scene.

During planning, collision geometry for all scene objects is included; end-effector to object grasp transforms are fixed when the next segment requires a grasped object; and the returned collision-free interpolation $o_t$5 is executed via a low-level inverse-kinematics controller in the end-effector–pose action space. This ensures that the robot never executes the optimized skill unless it can safely reach its starting joint pose.

All demonstrations that fail collision checks or keypoint-matching are discarded by a task-specific success detector. The remaining demonstrations, generated on the order of 1,000 per task, form a dataset for imitation learning.

5. Empirical evaluation

Experiments were reported on 16 simulation tasks and four real-world tasks, featuring multi-stage, non-prehensile and tight-tolerance manipulation. Policies trained using CP-Gen achieved an average success rate of 77%, outperforming the best baseline that achieved an average of 50% (Lin et al., 5 Aug 2025).

In simulation, the evaluation used 8 single-arm Franka tasks from the MimicGen benchmark under two reset distributions: Pose Only (D1), in which only object poses vary as in MimicGen, and Geometry Generalization (TaskG), in which object scale is sampled nonuniformly in a task-specific range. The policies were Diffusion Policies taking either RGB or (Depth + SegMask) inputs. The reported metric was zero-shot success rate of closed-loop policies trained on CP-Gen data versus MimicGen and a physics-enabled variant of DemoGen.

Setting	Baseline results	CP-Gen
Pose Only (D1), RGB	MimicGen 63% avg; DemoGen 86% avg	85% avg
Geometry Generalization (TaskG), RGB	MimicGen 35%; DemoGen 44%	73%
Real-world avg	MimicGen 40%	83%

The pose-only comparison is important because it shows that CP-Gen remains competitive when the problem is restricted to pose variation: MimicGen reached 63% average success, CP-Gen 85%, and DemoGen 86%. The geometry-generalization comparison isolates the regime that motivates the method: MimicGen reached 35%, CP-Gen 73%, and DemoGen 44%. The real-world results used four multi-stage or tight-tolerance tasks—Mug Cleanup, Hammer Cleanup, Mug Hanging on hook, and Wine Glass Spiral Hanging—and policies trained entirely in simulation on CP-Gen data transferred zero-shot to real hardware with an average success of 83% versus MimicGen’s 40% (Lin et al., 5 Aug 2025).

These results support the paper’s claim that geometry awareness via keypoint-trajectory constraints lets CP-Gen generate valid demonstrations across a wide range of object shapes, not just poses, while remaining compatible with closed-loop visuomotor training.

6. Limitations, open problems, and methodological position

The method has several stated limitations. It relies on manual segmentation into skill versus motion segments and manual keypoint annotation, although future work could automate both via vision-language or learned keypoint models. It assumes fixed skill order and therefore does not generalize to novel task sequences. It requires a geometric model, specifically a mesh, of each task object to sample and apply transforms. Generating for a new object instance within the same category but with an unknown transform remains nontrivial. The current scope is restricted to single-arm setups, with extension to multi-arm or mobile platforms identified as future work.

These limitations clarify the specific sense in which CP-Gen is constraint-preserving. The preserved object is not a symbolic task specification or a globally optimal policy, but the original keypoint-trajectory constraint extracted from an expert skill and re-imposed under transformed object geometry and pose. This makes CP-Gen a data-generation method rather than an equivariant policy architecture. The paper explicitly positions it as bridging $o_t$6 pose-only data generation and fully equivariant policy architectures by embedding geometry into the data generation process via keypoint-trajectory constraints.

Only one expert demonstration per task is required. In the context defined by the paper, that property is paired with geometry-aware generation, closed-loop visuomotor policies, and zero-shot sim-to-real transfer on multi-stage, tight-tolerance tasks. A plausible implication is that CP-Gen is most useful when the task structure is stable, the relevant object geometry can be parameterized by known transforms, and the primary bottleneck is demonstration coverage rather than online planning or task-sequence recomposition.