A Principled Approach for Creating High-fidelity Synthetic Demonstrations for Imitation Learning

Abstract: Recent advances in 3D Gaussian Splatting (3DGS) have enabled visually realistic demonstration generation from a single expert trajectory and a short multi-view scan. However, existing 3DGS-based synthesis pipelines typically generate new motions using sampling-based planners or trajectory optimization, which often deviate substantially from the expert's demonstrated path. While such deviations may be acceptable for tasks insensitive to motion shape, they discard subtle spatial and temporal structure that is critical for contact-rich and shape-sensitive manipulation, causing increased demonstration diversity to harm downstream policy learning. We argue that demonstration synthesis should treat the expert trajectory as a strong prior. Building on this principle, we propose a framework that synthesizes diverse task demonstrations while explicitly preserving expert motion structure. We model the expert trajectory using Dynamic Movement Primitives (DMPs) and retarget it to new goals, object configurations, and viewpoints within a reconstructed 3DGS scene, yielding phase-consistent, shape-preserving motion by construction. To safely realize this expert-preserving diversity in cluttered scenes, we introduce an analytic obstacle-aware DMP formulation that operates directly on the continuous density field induced by the 3DGS representation. This enables collision avoidance while minimally perturbing the nominal expert motion, unifying photorealistic rendering and geometric reasoning without additional scene representations. We evaluate our approach on a Spot mobile manipulator across three manipulation tasks with increasing sensitivity to trajectory fidelity. Compared to planner- and optimization-based synthesis, our method produces trajectories with lower deviation and collision rates and yields higher task success when training diffusion-based visuomotor policies.

• The paper introduces the Follow-the-Expert (FTE) framework that fits Dynamic Movement Primitives to expert trajectories, ensuring minimal Dynamic Time Warping deviation.
• It couples DMP trajectories with a unified 3D Gaussian Splatting density field to achieve obstacle-aware motion retargeting without auxiliary simulation tools.
• Empirical results demonstrate that FTE outperforms planner-based and optimization-based methods in tasks like writing and push–pick–place through improved trajectory fidelity and reduced collisions.

High-Fidelity Synthetic Demonstration Generation for Imitation Learning

Motivation and Context

Imitation learning (IL) is a principal route to acquiring dexterous, semantic-rich robotic manipulation skills from human demonstration. However, policies trained from limited demonstration data are prone to overfitting to narrow distributions of object pose, viewpoint, and scene context, resulting in brittle behavior in unseen settings. Recent augmentation pipelines leveraging 3D Gaussian Splatting (3DGS) reconstruct photorealistic scenes and generate synthetic demonstrations spanning diverse visual conditions, enabling robust sim2real transfer. Despite substantial advances in visual generalization, prior 3DGS-based methods often ignore or distort the expert demonstration’s trajectory structure during motion synthesis, undermining policy efficacy in tasks where success depends on spatial and temporal fidelity (e.g., contact-rich manipulation, stroke-based actions).

Follow-the-Expert (FTE) Demonstration Synthesis

The paper introduces the FTE paradigm, which treats the full expert trajectory as a strong prior for synthetic demonstration generation. Rather than planner-centric or optimization-based motion synthesis, FTE fits Dynamic Movement Primitives (DMPs) to segmented end-effector paths and retargets the learned dynamical system to new goals, object configurations, and viewpoints. This guarantees phase-consistent, shape-preserving reproduction, systematically varying endpoints while enforcing the trajectory’s spatiotemporal structure.

The pipeline consists of:

• Scene reconstruction: Photorealistic 3DGS models are constructed from short multi-view scans, parameterizing density, appearance, and occupancy for explicit geometric reasoning.
• Trajectory prior fitting: Demonstrations are decomposed into interaction-aware segments and modeled via DMPs for position and orientation, using quaternion log mapping for smooth orientation retargeting.
• Goal perturbation: Segment boundaries are perturbed via sampled translation/rotation, generating systematic but expert-faithful trajectory diversity.

  Figure 1: FTE pipeline: expert demonstration encoding as DMP prior, obstacle-aware retargeting within aligned 3DGS, and multi-view synthetic data production.

Obstacle-Aware Motion Retargeting via 3DGS Density

A major contribution is the analytic coupling of DMP trajectories with the density field induced by 3DGS. The aligned 3DGS model not only serves as a renderer but also as a continuous geometric substrate for collision avoidance. Obstacle awareness is achieved by modulating DMP dynamics using density-gradient repulsion; rollouts deform locally around dense regions while minimizing deviation from the expert trajectory. This approach obviates the need for auxiliary simulators, meshes, or point clouds, streamlining pipeline complexity.

Figure 2: Obstacle-aware retargeting: density-gradient coupling locally deforms rollouts around obstacles, preserving expert motion structure.

Empirical Evaluation and Results

Experiments are conducted on the Spot mobile manipulator across three tasks, each requiring semantic, shape-sensitive trajectory fidelity: Sweep Clean, Push–Pick–Place, and Writing a Letter “A”.

Figure 3: Tasks where trajectory encodes both semantics and context.

The FTE synthesis is benchmarked against two baselines:

• Planner-based keyframe stitching (MPLib)
• Demo-anchored trajectory optimization (TrajOpt)

Metrics include task success rate, path deviation (Dynamic Time Warping), collision rate, and normalized writing error.

Trajectory Fidelity and Policy Transfer

FTE achieves minimal DTW deviation to the expert trajectory, both in synthesized demonstrations and executed policy rollouts, significantly outperforming planner-based and optimization-based approaches. Crucially, diffusion-based visuomotor policies trained on FTE data inherit this fidelity, reproducing expert motion profiles in deployment.

Figure 4: Comparison of synthesized and rollout trajectories across tasks.

Figure 5: Dynamic Time Warping to expert demonstration; lower is better and indicates superior trajectory fidelity.

On the writing task, FTE produces the lowest normalized pixel error, underscoring superior shape preservation.

Obstacle-Aware Synthesis: Safety vs Motion Deviation

The density-driven obstacle-aware variant (FTE+OA) reduces collision rates compared to optimization baselines, while minimally perturbing expert motion. This enables safe trajectory augmentation in cluttered scenes without sacrificing task semantics.

Implications and Future Directions

The results substantiate that effective demonstration augmentation in imitation learning is not merely about maximizing diversity, but generating diversity that preserves expert intent and motion structure, especially when task semantics are trajectory-encoded. The unified photorealistic and geometric reasoning via 3DGS density fields integrates visual realism and safety, supporting scalable, safe, and semantic policy learning.

The method’s closed-form DMP rollouts, analytic density coupling, and compatibility with diffusion-based visuomotor policies facilitate practical adoption in both academic and industrial settings. Limitations stem from reliance on static scene density proxies and lack of explicit contact modeling; dynamic environments and force-sensitive tasks warrant extension via hybrid contact DMPs or uncertainty-aware clearance.

Conclusion

FTE delivers a principled, computationally efficient framework for high-fidelity synthetic demonstration generation. By enforcing expert trajectory priors and leveraging unified density-based obstacle avoidance within 3DGS, the system produces safe, semantically accurate augmentations that directly translate to improved policy performance, especially in trajectory-sensitive manipulation. Future directions include extending to dynamic scenes, deformable objects, and closed-loop replanning.