PALUM: Attention for Motion Retargeting

• PALUM is a transformer-based motion retargeting framework that leverages part-based attention to maintain motion semantics across varied skeleton configurations.
• It partitions the human skeleton into six semantic parts and applies localized spatial attention pooling along with temporal transformers for robust feature aggregation.
• Empirical results demonstrate that PALUM achieves lower mean per-joint errors in both intra- and cross-structural scenarios, setting a new performance standard.

PALUM (Part-based Attention Learning for Unified Motion Retargeting) is a transformer-based motion retargeting framework that introduces a principled part-based attention mechanism for robust transfer of human motion across skeletons with diverse joint layouts and topologies. PALUM addresses the core problem of maintaining motion semantics and realism when transferring data between character skeletons with varying joint counts, morphologies, and connectivity by learning skeleton-agnostic motion representations. It leverages a biologically motivated semantic partitioning of the skeleton and a rigorous cycle-consistency framework to achieve state-of-the-art results in both intra-structural and cross-structural motion retargeting scenarios (Liu et al., 12 Jan 2026).

1. Problem Formulation and Core Challenges

Motion retargeting seeks to map a motion sequence $\mathcal{M}_A$ from a source skeleton $\mathcal{S}_A$ to a corresponding motion $\mathcal{M}_B$ on a target skeleton $\mathcal{S}_B$. This problem becomes particularly challenging when $\mathcal{S}_A$ and $\mathcal{S}_B$ differ in joint count ($N_A\neq N_B$), bone connectivity, and proportions. Traditional techniques often rely on manual joint correspondences or hand-crafted features, limiting their flexibility and failing to generalize to skeletons with previously unseen structures. PALUM reforms this paradigm by:

• Constructing skeleton-agnostic representations that are independent of joint count and topology.
• Preserving semantic fidelity, including motion style and end-effector behaviors, irrespective of explicit joint mapping or heuristics.
• Overcoming the instability and generalization barriers observed in prior motion retargeting models.

2. Semantic Part-Based Attention Mechanism

PALUM exploits the hierarchical organization of the human skeleton by partitioning it into six overlapping semantic parts—torso (with shared spine and hip joints), left leg, right leg, left arm, right arm, and head (sharing neck/spine joints with torso). This partitioning enables localized attention within anatomical regions, such that joints within the same limb or part benefit from focused and contextually rich feature aggregation, while unnecessary cross-body attention is minimized. Each body part may share certain joints (e.g., the hips for legs and torso), a design discovered to improve model accuracy.

For each skeleton, joint features are embedded and grouped into these six semantic parts, leading to improved stability during model training and enhancing generalization to unseen skeletons with unfamiliar topologies or joint configurations.

3. Architecture: Encoding, Attention, and Decoding

PALUM's architecture is comprised of several coordinated neural modules:

• Encoder: For a motion sequence $\mathcal{M}_A$, at each frame, a 3D root position and a per-joint 6D rotation vector (following Zhou et al., 2020) compose the raw input. Features are linearly projected, and per-part, per-joint features are enhanced via three embeddings: a sinusoidal positional encoding, a frozen T5-based text embedding of joint names, and a T-pose positional encoding derived from the skeleton's root-relative offsets.
• Spatial Attention Pooling: Within each body part $i$, a set of $m$ learnable queries pools attention-weighted aggregates from $\mathcal{S}_A$0 joint representations:

$\mathcal{S}_A$1

The part features across all six partitions ($\mathcal{S}_A$2) are concatenated.

• Temporal Transformer Encoder: The pooled part tokens are processed by a Transformer, which applies multi-head self-attention jointly over time and semantic part tokens, producing the skeleton-agnostic representation $\mathcal{S}_A$3.
• Skeleton-Specific Decoder: For motion transfer to target skeleton $\mathcal{S}_A$4, a randomly initialized noise tensor is arranged into the six-part configuration of the target and enhanced using target-specific positional and name embeddings. A Transformer decoder—attending both within target tokens and across encoded features via cross-attention—outputs per-frame, per-joint features. These are mapped back to rotations and positions, and retargeted motions are produced via forward kinematics and root height normalization.

4. Cycle Consistency and Loss Framework

PALUM incorporates multiple loss terms to ensure robust learning:

• Reconstruction Loss ($\mathcal{S}_A$5): Enforces that the encoder-decoder pair can faithfully reproduce input motions on the same skeleton.
• Cycle Consistency Loss ($\mathcal{S}_A$6): After mapping A→B→A, enforces consistency of the latent feature representations to promote skeleton-agnostic invariance:

$\mathcal{S}_A$7

• Root Stability Loss ($\mathcal{S}_A$8): Penalizes drift in the global root’s position and orientation.

The full training objective is:

$\mathcal{S}_A$9

5. Training Protocol and Evaluation Design

PALUM is trained on Mixamo motion capture data at 30 fps, with 12 characters for training and 7 held out for testing (3512 sequences in total) and finger joints removed via name-based filtering. The evaluation considers four data splits: seen character/seen motion (sc+sm), seen character/unseen motion (sc+um), unseen character/seen motion (uc+sm), and unseen character/unseen motion (uc+um). Model hyperparameters include:

• AdamW optimizer ($\mathcal{M}_B$0, $\mathcal{M}_B$1, $\mathcal{M}_B$2, weight decay $\mathcal{M}_B$3),
• Batch size 16, motion window $\mathcal{M}_B$4,
• Embedding dimension $\mathcal{M}_B$5, $\mathcal{M}_B$6 queries per part, 8 attention heads,
• 6 encoder and 6 decoder Transformer layers.

6. Empirical Results: Quantitative, Qualitative, and Ablation

Motion fidelity is quantified by the normalized mean per-joint position error (MSE normalized by character height).

• For intra-structural retargeting, PALUM achieves 0.00272 mean MSE, surpassing R²ET (0.00487), PAN (0.00898), and MoMa (0.01100).
• For cross-structural transfer, PALUM's MSE is 0.00567; PAN: 0.01436, MoMa: 0.02340. R²ET is inapplicable to cross-structural cases.

Qualitatively, on Mixamo, SMPL, and MetaHuman skeletons, PALUM preserves coordinated limb motion and end-effector semantics (e.g., punch amplitude, foot strike timing) more successfully than baselines. In the most challenging scenario (unseen character plus unseen motion), PALUM produces natural trajectories with minimal wobble; PAN and MoMa display misaligned or drifting body parts.

Ablation experiments demonstrate:

Ablation Condition	Intra Error	Cross Error
w/o shared joints ("w/o share")	0.00377	0.00653
w/o positional/T-pose ("w/o pos")	-	0.00660
Joint masking (as in MoMa)	-	0.00693
No cycle loss ($\mathcal{M}_B$7)	-	+17%
$\mathcal{M}_B$8 reduced 4→2	+10%

Notably, shared joints between parts, inclusion of both positional and T-pose embeddings, and cycle consistency are critical to performance; reducing part query count or random masking impairs accuracy.

7. Limitations and Future Perspectives

PALUM's skeleton-agnostic capability enables generalization to unseen character topologies within the Mixamo-style domain. Limitations arise from exclusive reliance on T5-based joint name embeddings, which may become unreliable for skeletons with wildly divergent kinematic chain lengths (e.g., differences in spinal joint segmentation). The current strategy leaves end-effector positioning unconstrained, limiting retargeting accuracy for environment-coupled actions such as object manipulation.

Proposed directions include augmenting joint embeddings with explicit bone-length ratios or learned geometric features to resolve ambiguities arising from inconsistent chain lengths, extending semantic partitioning frameworks to non-humanoid morphologies, and deploying part-based attention architectures toward tasks such as motion generation from text or pose, editing, and in-betweening.

PALUM exemplifies a shift towards biologically inspired, attention-driven motion representation, establishing new performance standards for cross-skeleton motion retargeting under topological heterogeneity and providing a scalable foundation for broader, semantics-preserving animation synthesis (Liu et al., 12 Jan 2026).