---
title: 'DynaMimicGen: Dynamic Data for Robot Learning'
url: https://www.emergentmind.com/topics/dynamimicgen
type: topic
---

# DynaMimicGen: Dynamic Data for Robot Learning

DynaMimicGen is a scalable, real-time data generation framework for efficient robot policy learning in dynamic manipulation environments. It enables the synthesis of large, high-diversity imitation learning datasets from a minimal set of human demonstrations, explicitly supporting adaptation to dynamic task settings. Utilizing Dynamic Movement Primitives (DMPs) and automated subtask segmentation, DynaMimicGen produces robust, task-consistent trajectory data that generalizes across variable scene layouts, object instances, and robot configurations, with validated advantages in both simulation and real-world robotic applications [2511.16223].

## 1. Problem Scope and Pipeline Overview

DynaMimicGen ("D-MG") is designed to address the data bottleneck in training robust manipulation policies for contact-rich and long-horizon robot tasks, where collecting large-scale, human-curated demonstrations is costly or impractical. D-MG operates as follows:

- Given a small dataset $\mathcal{D}_{\mathrm{src}}$ (1 or 2 human demonstrations) for a manipulation task $\mathcal{M}$, modeled as an MDP with state space $\mathcal{S}$ and action space $\mathcal{A}$ (absolute end-effector pose and binary gripper control), D-MG generates a large, diverse dataset $\mathcal{D}_{\mathrm{gen}}$ suitable for imitation policy training.
- The pipeline: (1) Collection of a few demonstrations, (2) Automatic segmentation into $M$ object-centric subtasks, (3) Fitting a DMP to each subtask, (4) Generating new rollouts for varied scene configurations by dynamically adapting DMPs at execution time (object pose, scene geometry, robot initial state), (5) Real-time rollout abortion if subtask constraints are violated.

This approach enables data augmentation far beyond the scale of the original demonstrations, supporting both static and dynamic (mid-execution scene perturbation) settings.

## 2. Demonstration Segmentation and Subtask Decomposition

DynaMimicGen assumes task structure decomposes into a fixed, ordered sequence of object-centric subtasks $S_1(o_1), ..., S_M(o_M)$. In simulation, subtask boundaries are detected through event signals (e.g., gripper closure for grasp completion, collision/contact metrics for placement), whereas in the real world, boundaries can be manually annotated or inferred via simple heuristics such as joint position thresholds.

Segmentation Algorithm:

```
Input: τ = {s_t, a_t : t=1…T}, subtask order [o₁,…,o_M]
Output: τ₁…τ_M
t₀ = 1
for i in 1…M:
    find t_i > t_{i-1} such that event_i(s_{t_i}, a_{t_i}) == true
    τ_i = τ[t_{i-1} : t_i]
```

This process produces trajectory segments each mapped to a single object manipulation primitive.

## 3. Dynamic Movement Primitives: Parameterization and Adaptation

For each subtask, DynaMimicGen fits a DMP to the demonstrated trajectory segment. The DMP formulation, based on transformation and canonical systems, facilitates trajectory modulation toward new goals and scene variations:

- **Canonical system (phase variable $x$):**
  $$
  \dot{x} = -\alpha_x x
  $$
- **Transformation system (Cartesian $y$):**
  $$
  \tau^2 \ddot{y} = K(g - y) - D \tau \dot{y} + K f(x)
  $$
- **Forcing function:**
  $$
  f(x) = x (g - y_0) \cdot \left[
    \frac{\sum_i \omega_i \psi_i(x)}{\sum_i \psi_i(x)}
  \right], \quad \psi_i(x) = \exp( -h_i (x - c_i)^2 )
  $$

Learning proceeds via Locally Weighted Regression, optimizing the match between demo forcing terms and basis-weighted activations.

*Adaptation for New Scenes*: At each rollout, DMPs are recomputed for current end-effector pose $y_0$ and a new goal $g$, derived by re-projecting the demonstrated end-effector-to-object transform into the new object pose. Temporal scaling $\tau$ may be adapted to relative distances, and the learned DMP weights $\omega_i$ are reused to preserve trajectory shape.

## 4. Real-Time Generation and On-the-Fly Adaptation

At execution, DynaMimicGen operates in real time, adapting to dynamic object poses at each control tick (20 Hz):

1. Sense the current object pose $\tilde T_W^{\mathrm{obj}}(t)$.
2. Update target $g(t) = \tilde T_W^{\mathrm{obj}}(t) \cdot \hat T^{\mathrm{target}}_{\mathrm{obj}}$.
3. Step the DMP integrator toward updated $g(t)$.
4. Emit Cartesian command $y(t)$ and accompanying gripper action.
5. Repeat until DMP phase variable $x \rightarrow 0$.

This mechanism allows the policy to "bend" its trajectory in response to perturbations, enabling robust imitation even under unpredictable changes.

## 5. Dataset Synthesis and Diversity

DynaMimicGen generates 1000 successful rollouts per task variant and demonstration count, spanning various scene layouts, object perturbations, and robot initializations. Example domains include:

- **Stack:** Block stacking in different workspace sizes (e.g., $0.16 \times 0.16$ m to $0.4 \times 0.4$ m).
- **Square:** Nut/peg insertion in boxes with varied size/orientation.
- **MugCleanup:** Drawer and mug placement tasks over multiple scene arrangements.

Dynamic scenarios introduce controlled object pose perturbations during execution. Robot configurations include simulated Sawyer/Franka arms and real-world Franka Panda executions.

## 6. Imitation Policy Training and Evaluation

Policies are trained via both behavior cloning (BC-RNN, as in RoboMimic) and Diffusion Policy (DP):

- **State/observation spaces:** Image-based (dual 84 × 84 RGB cameras) and low-dimensional (end-effector pose, gripper, ground-truth object pose).
- **Action space:** Absolute end-effector pose (normalized) and gripper.
- **Training regime:** BC is trained up to 600 epochs (LR $1 \times 10^{-3}$ or $1 \times 10^{-4}$), DP up to 2000 epochs. Early stopping is based on success rate.
- **Evaluation:** Maximum checkpointed success over 50 rollouts, averaged across 3 random seeds.

**Performance results:**

| Task       | Method     | D0 (%) | D1 (%) | D2 (%) |
|------------|------------|--------|--------|--------|
| Stack      | DP-D-MG    | 83.3   | 70.7   | –      |
|            | DP-MG      | 74.0   | 65.3   | –      |
|            | BC-D-MG    | 76.0   | 71.3   | –      |
|            | BC-MG      | 72.7   | 58.0   | –      |
| Square     | DP-D-MG    | 86.7   | 46.7   | 23.3   |
|            | DP-MG      | 75.3   | 24.7   | 12.0   |
|            | BC-D-MG    | 97.3   | 50.7   | 37.3   |
|            | BC-MG      | 88.7   | 44.0   | 34.0   |
| MugCleanup | DP-D-MG    | 90.0   | 60.7   | –      |
|            | DP-MG      | 61.3   | 38.0   | –      |
|            | BC-D-MG    | 79.3   | 58.7   | –      |
|            | BC-MG      | 65.3   | 31.3   | –      |

Data generation rates (DGR) and policy successes demonstrate that DynaMimicGen exceeds or matches prior baselines (MG) in static settings and is uniquely effective in dynamic environments.

## 7. Failure Modes, Limitations, and Extensions

Identified limitations include:

- Assumption of a known, fixed sequence of object-centric subtasks.
- Requirement for accurate, real-time ground-truth object poses; sensitivity to sensor noise or occlusions.
- Handling of only one reference object per subtask; lack of multi-object or explicit obstacle consideration.
- Fixed DMP durations can be limiting under late perturbations.
- Focus on single-arm setups; bimanual coordination not supported.
- Real-world policy generalization remains limited by dataset size and perception system calibration.
- Sim-to-real robustness and large-scale visual diversity require further domain randomization.

*Future directions* include multi-object DMP extensions, integration with learned visual perception modules for markerless operation, bimanual and multi-agent generalization, enhanced sim-to-real transfer (domain/dynamics randomization), and incorporation of foundation or language models for automatic subtask extraction.

## References

- "DynaMimicGen: A Data Generation Framework for Robot Learning of Dynamic Tasks" [2511.16223]

Source: https://www.emergentmind.com/topics/dynamimicgen