UniPrototype Framework: Skill Transfer

• UniPrototype is a framework for human-robot skill learning that leverages unified, compositional prototypes to enable efficient knowledge transfer from human demonstrations.
• The architecture employs a three-stage process—temporal encoding, compositional prototype discovery, and skill alignment—to robustly align human and robot motion features.
• Experimental results in simulation and real-world scenarios demonstrate significant improvements in cross-embodiment success rates and robust performance under varied conditions.

UniPrototype is a framework for human-robot skill learning that leverages a unified, compositional prototype representation to facilitate efficient knowledge transfer from human demonstrations to robotic embodiments. Addressing the persistent issue of data scarcity in robotic manipulation, UniPrototype enables shared motion primitives and compositional skill representations, supporting robust policy learning and cross-embodiment generalization. Its primary contributions are a compositional prototype discovery mechanism with soft multi-prototype assignments, an adaptive prototype selection strategy using assignment entropy, and the demonstration of effective human-to-robot manipulation knowledge transfer across both simulation and real-world settings (Hu et al., 27 Sep 2025).

1. Framework Architecture and Knowledge Transfer Pipeline

The UniPrototype pipeline is structured into three principal stages that process unpaired human demonstration videos and robot demonstration datasets:

• Stage 1: Temporal Skill Encoding
  • Each demonstration video is divided into overlapping clips $v_{ij}$ of length $L$.
  • A shared transformer-based encoder $f_{\text{temp}}$ maps each clip to a temporal embedding $z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$.
  • Data augmentations (cropping, color, geometric changes) enforce invariance to embodiment, improving the alignment of human and robot motion features.
• Stage 2: Compositional Prototype Discovery
  • A learnable prototype matrix $C \in \mathbb{R}^{d \times K}$ is maintained.
  • For a batch of embeddings $Z = [z_1, ..., z_B]$, similarity is computed as $S = C^\top Z$, and soft assignments are produced:

  $$Q_{i, k} = \frac{\exp(S_{k, i}/\tau)}{\sum_{k'} \exp(S_{k', i}/\tau)}$$

  where $\tau$ is a temperature parameter and row-wise normalization of $S^\top$ enables compositional representations (multiple co-active prototypes). - The encoder and prototypes are trained jointly using: - Prototype consistency loss $L$0 (contrastive over augmentations). - Temporal coherence loss $L$1 (time-contrastive). - The number of prototypes $L$2 is selected via entropy-based monitoring (see Section 3).

• Stage 3: Skill Alignment and Policy Learning

  • An attention-based Skill Alignment Module (SAM) aligns current robot observations $L$3 to the next prototype in the human-extracted sequence.
  • A diffusion policy $L$4 is trained, conditioned on compositional embedding $L$5, to produce robot actions via iterative denoising.
  • At inference, human demonstration encodings are aligned and rolled out by the diffusion policy to execute corresponding robotic behavior.

2. Compositional Prototype Discovery Mechanism

The compositional prototype module enables flexible, multi-prototype activation for each encoded skill segment:

• Soft Assignment
  • For batch $L$6 and prototype matrix $L$7:
  • Similarity: $L$8, $L$9.
  • Assignment: $f_{\text{temp}}$0 (row-normalized over $f_{\text{temp}}$1).
  • This mechanism contrasts with hard clustering (e.g., Sinkhorn), where only one prototype is active per embedding. Soft assignment allows hierarchical and blended skill representations.
• Losses
  • Prototype Consistency:

  $f_{\text{temp}}$2

  where $f_{\text{temp}}$3 and $f_{\text{temp}}$4 are assignments from two augmentations of the same clip. - Temporal Coherence:

  $f_{\text{temp}}$5

  enforcing temporal smoothness in skill representations.

3. Adaptive Prototype Selection via Entropy

To ensure scalability and alignment with task complexity, UniPrototype employs an entropy-based strategy for adaptive prototype selection:

• The average prototype activation is:

$f_{\text{temp}}$6

• Assignment entropy for $f_{\text{temp}}$7 prototypes:

$f_{\text{temp}}$8

• $f_{\text{temp}}$9 is increased until the entropy increment $z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$0 falls below threshold $z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$1, and the smallest $z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$2 achieving this is selected, avoiding unnecessary overparameterization.

Adaptive $z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$3 analysis in experiments showed task-dependent prototype counts: simple tasks ($z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$4–$z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$5), tool use ($z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$6–$z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$7), multi-step tasks ($z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$8–$z_{ij} = f_{\text{temp}}(v_{ij}) \in \mathbb{R}^d$9), and complex tasks ($C \in \mathbb{R}^{d \times K}$0–$C \in \mathbb{R}^{d \times K}$1), corresponding to higher entropies for more complex behaviors.

4. Training and Inference Workflow

The training process comprises prototype discovery and policy learning, as summarized below:

$C \in \mathbb{R}^{d \times K}$8

At test time, human demonstration encodings are mapped via $C \in \mathbb{R}^{d \times K}$2 and $C \in \mathbb{R}^{d \times K}$3, aligned with SAM, and executed using the trained diffusion policy.

5. Experimental Setups and Evaluation Metrics

Experiments were conducted in both simulated and real-world conditions:

Setting	Task Types	Metrics	Baselines
RLBench	100 manipulation tasks	Success rate (%)	GCD Policy, GCD+TCN, XSkill
Real-world	Table wiping, grasp/place, drawer, spatula flipping	Success rate (%) & robustness (clutter, lighting, position shifts)	GCD Policy, XSkill

Simulated manipulations covered tasks such as emptying dishwashers, closing boxes, and peg insertion, with evaluation at multiple robot execution speeds ($C \in \mathbb{R}^{d \times K}$4, $C \in \mathbb{R}^{d \times K}$5). Real-world tests used a Franka Emika Panda arm over varied object identities and spatial conditions, measuring both task success and robustness.

6. Quantitative and Qualitative Performance

UniPrototype demonstrated state-of-the-art transfer and execution robustness:

Simulation (RLBench) Cross-Embodiment Success Rates (%):

Method	Same-speed	Cross-speed×1.0	Cross-speed×2.0
GCD Policy	68.3±2.1	12.4±1.8	4.1±0.9
GCD+TCN	71.2±1.9	24.7±2.3	11.6±1.7
XSkill	84.6±1.5	78.2±1.8	52.3±2.4
UniPrototype	91.3±1.2	87.5±1.4	71.2±2.0

Real-World Success Rates (%):

Task	GCD Policy	XSkill	UniPrototype
Table Wiping	20.8±4.2	45.8±5.1	70.8±4.5
Cup Grasping	41.7±4.8	66.7±4.7	83.3±3.7
Drawer Retrieval	16.7±3.8	50.0±5.1	75.0±4.4
Tool Use (Spatula)	25.0±4.4	54.2±5.1	79.2±4.1
Average	26.1±4.3	54.2±5.0	77.1±4.2

Ablation studies confirm the importance of each contribution: replacing soft RowNorm with hard Sinkhorn assignment reduces cross-embodiment success by approximately 11 points; using a fixed $C \in \mathbb{R}^{d \times K}$6 decreases performance by 4.5 points; omitting temporal coherence or compositional alignment each degrades performance by 3–5 points.

Qualitative analyses (t-SNE, prototype-activation timelines) corroborate the effective compositional encoding of skill segments and the overlap of human-robot embedding trajectories, including during transition intervals (e.g., "lift + rotate" in pouring tasks).

7. Limitations and Prospects

Key limitations include increased computational and memory overhead due to the entropy-based $C \in \mathbb{R}^{d \times K}$7 search and multi-prototype assignment, relative to single-clustered models. Experiments are limited to semi-structured laboratory environments; empirical validation in unstructured, real-world scenarios remains unaddressed.

Proposed directions for future research are:

• Online refinement of the prototype vocabulary as new demonstrations become available.
• Human-interpretable semantic grounding of discovered motion prototypes.
• Extension of the framework to multi-agent coordination and deformable-object manipulation scenarios.
• Improved computational efficiency for the adaptive prototype selection process.

UniPrototype establishes a compositional, entropy-adapted prototype vocabulary that demonstrably bridges human and robotic skill domains, yielding marked improvements in sample efficiency and robustness across a range of manipulation tasks (Hu et al., 27 Sep 2025).