Learning Reactive Human Motion Generation from Paired Interaction Data Using Transformer-Based Models

Abstract: Recent advances in deep learning have enabled the generation of videos from textual descriptions as well as the prediction of future sequences from input videos. Similarly, in human motion modeling, motions can be generated from text or predicted from a single person's motion sequence. However, these approaches primarily focus on single-agent motion generation. In contrast, this study addresses the problem of generating the motion of one person based on the motion of another in interaction scenarios, where the two motions are mutually dependent. We construct a dataset of paired action-reaction motion sequences extracted from boxing match videos and investigate the effectiveness of Transformer-based models for this task. Specifically, we implement and compare three models: a simple Transformer, iTransformer, and Crossformer. In addition, we introduce a person ID embedding to explicitly distinguish between individuals, enabling the model to maintain structural consistency and better capture interaction dynamics. Experimental results show that the simple Transformer can generate plausible interaction-aware motions without suffering from posture collapse, while iTransformer and Crossformer accumulate errors over time, leading to unstable motion generation. Furthermore, the proposed person ID embedding contributes to preventing structural collapse and improving motion consistency. These results highlight the importance of explicitly modeling individual identity in interaction-aware motion generation.

• The paper introduces a 210,926-frame paired boxing dataset and shows that a simple temporal Transformer generates more stable, human-like reactions than iTransformer and Crossformer models.
• Person ID embeddings improve motion quality, raising the simple Transformer’s offline human-likeness score from 4.0 to 4.7 while preserving interaction performance.
• Feature-wise architectures accumulate joint errors during autoregressive generation, whereas full-frame temporal modeling better preserves skeletal structure, though latency and subjective evaluation remain limitations.

Problem setting and motivation

Most human motion generation research addresses single-agent settings: text-to-motion synthesis or autoregressive motion prediction from one person's past trajectory. This paper by Soga and Takebayashi (Wakayama University) targets a comparatively underexplored task: generating the reaction motion of one individual conditioned on the action motion of another, in mutually dependent interaction scenarios. The authors select boxing as the domain because it demands rapid, physically coupled responses, and because existing interaction datasets such as SBU (~7K frames) are too small and homogeneous to support robust learning, while diffusion-based approaches such as ReMoS achieve high diversity but incur generation times of up to 50 seconds, precluding real-time use (2604.22164).

The paper positions itself against two prior reaction-generation systems. Interaction Transformer (InterFormer) uses parallel temporal and spatial self-attention with an Interaction Distance Module and Skeleton Adjacency Module, but was evaluated mainly on datasets where reactions are stereotyped (e.g., backward recoiling in SBU), leaving distance-preserving defensive patterns such as ducking unevaluated. ReMoS demonstrates that DDPMs yield higher motion diversity than Transformer baselines but at prohibitive inference cost. The present work instead asks whether a plain Transformer can produce plausible reactive motion at interactive rates.

Dataset construction

The training corpus is derived from the Kaggle Olympic Boxing Video Dataset: 21 match videos at 50 fps and 1920×1080 resolution. The pipeline proceeds in four stages:

• Segment extraction: non-match intervals (between matches, between rounds, judging segments, referee stoppages) are removed.
• 2D pose estimation: AlphaPose with FastPose/ResNet-152, YOLO-X detection, and OC-SORT tracking, producing COCO-17 keypoints. Because spectators and referees crossing the frame cause frequent ID switches, manual ID consolidation is applied as post-processing — an acknowledged manual bottleneck in the pipeline.
• Gap handling: missing frames are linearly interpolated only when gaps span at most three consecutive frames; sequences shorter than 30 consecutive frames after interpolation are discarded.
• 3D lifting: MotionBERT lifts COCO-17 sequences to Human3.6M-format 3D joints via an explicit joint mapping table (e.g., pelvis computed as the mean of left/right hips).

Preprocessing then retargets all poses to a fixed Humanoid skeleton (preserving joint directions while matching bone lengths), normalizes each frame relative to the pelvis translated to $(x,y,z)=(0,1,0)$, and rounds coordinates to six significant digits. The final dataset comprises 210,926 frames of paired action–reaction motion — roughly 30× the frame count of SBU. A caveat worth noting: the data derive from monocular video pose estimation rather than ground-truth mocap, so label noise from the AlphaPose/MotionBERT chain propagates into training.

Model architectures

Three Transformer variants are trained under identical hyperparameters (2 encoder layers, 1 decoder layer, $d_{model}=512$, 8 heads, FFN dimension 2048, dropout 0.1, batch size 32, learning rate $10^{-4}$, MSE loss):

• Simple Transformer: input motions $X$ (subject) and $Y$ (counterpart), each 30 frames of a 51-dimensional per-frame vector (17 joints × 3 coordinates), concatenated along the feature dimension; temporal self-attention over time steps; decoder conditions on 10 past frames plus ground truth to predict the next frame autoregressively.
• iTransformer: encoder-only; joints (features) serve as tokens and attention operates across the feature dimension, with temporal aggregation delegated to a learned linear projection.
• Crossformer: Dimension-Segment-Wise embedding with segment length 5 and Two-Stage Attention using 30 routers, reducing cross-dimension attention complexity from $O(D^2)$ to $O(FD)$.

A person ID embedding — a learnable vector added to the full input sequence per individual — is introduced to disambiguate which features belong to which person, encouraging intra-person feature interactions while preserving cross-person correspondence learning.

Evaluation methodology

Evaluation combines a real-time system (Azure Kinect DK capture → model inference → Unity visualization over TCP) with an offline evaluation on held-out test data, isolating latency effects. Four canonical boxing motions define the inputs: straight punch, hook, block, and ducking, with the expectation that offensive inputs elicit defensive reactions and vice versa. For the initial 30 frames, the counterpart input is initialized by mirroring the subject's fighting stance along the x- and z-axes. Quality is assessed via a 5-point Likert questionnaire with 17 participants covering temporal smoothness, interaction appropriateness, and human-likeness.

Results

The headline findings are stark. Summarizing mean scores:

Configuration	Smoothness	Interaction	Human-likeness
Simple Transformer w/o ID (offline)	4.4	3.8	4.0
Simple Transformer w/ ID (offline)	4.6	4.2	4.7
iTransformer w/o ID (real-time)	3.1	1.0	1.0
iTransformer w/ ID (real-time)	3.7	1.4	1.4
Crossformer w/o ID (real-time)	2.5	1.5	1.1
Crossformer w/ ID (real-time)	2.5	1.9	1.4

The simple Transformer with person ID embedding achieves near-ceiling offline scores (smoothness mode 5, human-likeness mean 4.7 with SD 0.5), whereas iTransformer without ID embedding receives unanimous "strongly disagree" ratings on both interaction and human-likeness in the real-time condition (mean 1.0, SD 0.0 across all 17 participants). Qualitatively, at 100 frames into offline generation, the simple Transformer produces a recognizable ducking avoidance response to a straight punch, while iTransformer and Crossformer outputs exhibit posture collapse severe enough that the figure is no longer clearly humanoid.

Person ID embedding improves scores across nearly all configurations, most notably human-likeness for the simple Transformer (3.2 → 3.9 real-time; 4.0 → 4.7 offline). Crucially, it does not degrade interaction scores, addressing the concern that identity conditioning might weaken cross-person correspondence learning.

Temporal constraints matter: offline scores exceed real-time scores almost uniformly. Crossformer shows the largest smoothness gain offline (2.5 → 3.6 without ID), attributed to its longer inference time; the simple Transformer gains most on interaction and human-likeness, plausibly due to frame-rate mismatch between training data and live Kinect capture.

Why feature-wise architectures collapse

The paper's central analytical claim is architectural: the simple Transformer updates all joints of a frame jointly as a single token, implicitly enforcing global pose consistency and preventing independent amplification of per-joint errors. iTransformer treats each joint as an independent token updated along time, weakening intra-frame joint constraints; Crossformer, despite its staged Cross-Time/Cross-Dimension attention, likewise never updates a full frame atomically. Both therefore accumulate joint-wise errors under autoregressive rollout until skeletal structure collapses. Crossformer degrades less severely than iTransformer because its Cross-Time Stage captures temporal dependencies via attention rather than a fixed linear projection, yielding more balanced error propagation. This constitutes a somewhat counterintuitive result: multivariate time-series Transformers designed for forecasting generality underperform a vanilla Transformer on structured human pose generation, suggesting that feature-wise dependency modeling alone is insufficient for stable interaction-aware motion synthesis.

Limitations and open questions

Several limitations are conceded or evident. First, evaluation is entirely subjective (n=17 Likert questionnaires); no quantitative metrics (MPJPE, FID-style diversity measures, or interaction-specific distances à la InterFormer's Interaction Distance Module) are reported, making cross-paper comparison impossible. Second, the simple Transformer "does not perfectly reproduce boxing-specific reactions" — interaction appropriateness plateaus around 4.2 even in the best configuration, indicating responsive but not always correct behavior. Third, the dataset derives from noisy monocular pose estimation with manual tracker-ID correction, and the real-time system suffers from inference latency and train/inference frame-rate mismatch, both of which measurably depress scores. Fourth, the person ID embedding analysis leaves open whether its benefit stems from disambiguation or from regularization effects. Finally, generalization beyond four canonical boxing motions and beyond the boxing domain remains untested.

Conclusion

This paper formulates paired action–reaction motion generation in boxing, contributes a 210,926-frame dataset built from broadcast video through a documented pose-extraction pipeline, and demonstrates that a plain temporal Transformer outperforms purpose-built multivariate forecasting architectures (iTransformer, Crossformer) for this task, avoiding the posture collapse that afflicts feature-wise tokenization under autoregressive generation. The proposed person ID embedding consistently improves structural consistency without harming interaction learning. The results argue that architectural simplicity and explicit identity modeling outweigh added complexity for stable interactive motion generation, though the absence of quantitative evaluation metrics and the residual gap in reaction fidelity leave the precise characterization of interaction quality an open problem.