Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Abstract: The world knowledge and reasoning capabilities of text-based LLMs are advancing rapidly, yet current approaches to human motion understanding, including motion question answering and captioning, have not fully exploited these capabilities. Existing LLM-based methods typically learn motion-language alignment through dedicated encoders that project motion features into the LLM's embedding space, remaining constrained by cross-modal representation and alignment. Inspired by biomechanical analysis, where joint angles and body-part kinematics have long served as a precise descriptive language for human movement, we propose \textbf{Structured Motion Description (SMD)}, a rule-based, deterministic approach that converts joint position sequences into structured natural language descriptions of joint angles, body part movements, and global trajectory. By representing motion as text, SMD enables LLMs to apply their pretrained knowledge of body parts, spatial directions, and movement semantics directly to motion reasoning, without requiring learned encoders or alignment modules. We show that this approach goes beyond state-of-the-art results on both motion question answering (66.7\% on BABEL-QA, 90.1\% on HuMMan-QA) and motion captioning (R@1 of 0.584, CIDEr of 53.16 on HumanML3D), surpassing all prior methods. SMD additionally offers practical benefits: the same text input works across different LLMs with only lightweight LoRA adaptation (validated on 8 LLMs from 6 model families), and its human-readable representation enables interpretable attention analysis over motion descriptions. Code, data, and pretrained LoRA adapters are available at https://yaozhang182.github.io/motion-smd/.

• The paper proposes a novel SMD pipeline that deterministically transforms 3D skeletal data into human-readable text, eliminating learned motion encoders.
• It leverages rule-based biomechanical calculations to encode joint angles and trajectories into hierarchical structured descriptions, ensuring clear interpretability.
• Experimental results show improved QA and captioning metrics across multiple datasets, demonstrating strong domain robustness and efficient LoRA adaptation.

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Introduction and Motivation

The paper "Encoder-Free Human Motion Understanding via Structured Motion Descriptions" (2604.21668) introduces a paradigm shift in how 3D human motion is interfaced with LLMs. Prior state-of-the-art approaches require a dedicated motion encoder to map skeletal data into the embedding space of LLMs, with complex multi-stage training pipelines and tight coupling to specific backbone architectures. In contrast, this work proposes a deterministic, rule-based transformation of motion sequences into human-interpretable structured text, named Structured Motion Descriptions (SMD), eliminating the need for learned motion encoders and cross-modal alignment modules.

Figure 1: Encoder-based methods require complex motion encoders and alignment; the proposed approach uses rule-based SMDs with only LoRA fine-tuning.

This transformation exploits LLMs’ inherent linguistic competence regarding body parts, spatial relations, and movement semantics by casting motion understanding directly as a language task. The resulting system is model-agnostic, facilitating flexible adaptation across diverse LLM architectures via a single LoRA fine-tuning stage.

Methodology

Structured Motion Description (SMD) Framework

The SMD pipeline converts raw 3D skeletal joint trajectories into highly structured natural-language descriptions. The process includes joint angle calculation following biomechanical standards, extraction of global trajectory via the pelvis, segmentation of temporal patterns, and assembly into hierarchical textual blocks. The description is composed of meta information (duration, frame count), a trajectory block covering forward/lateral/global movement and orientation, and a set of per-joint angle descriptors grouped anatomically.

Figure 2: Overview of the deterministic SMD pipeline: joint angles and trajectories are computed rule-based and compiled into a structured textual prompt for conditioning LLM-based reasoning or captioning.

Key properties:

• Determinism: The pipeline is non-learned and fully specified by biomechanical rules, making it robust to domain shifts and skeleton variations.
• Human-readability: Every token in the SMD text corresponds to a semantically interpretable anatomical feature, facilitating transparency in downstream LLM inference.
• Hierarchical organization: The text is grouped such that active/critical joints appear with fine granularity, and static joints are compactly summarized, resulting in both efficiency and clarity.

SMD size is modulated by selecting the $K$ most active joints, balancing comprehensiveness with computational cost (sequences range from $\sim$1,000 to $4,000$ tokens in their full 26-angle form).

Task Formulation and LLM Adaptation

Both Motion Question Answering (QA) and Captioning are cast as autoregressive text generation tasks. For each sample, SMD is formatted into a prompt; for QA, the prompt includes a question and candidate answers; for captioning, only the SMD is provided.

Figure 3: Prompt structures for motion QA and captioning using SMD, enabling task conditioning.

The only learning component is a low-rank adapter (LoRA) inserted into a frozen LLM backbone. No motion encoder or fusion module is used, and the same SMD prompt can be fed to any compliant LLM with minimal retraining.

Experimental Evaluation

Datasets and Metrics

Evaluation is conducted on BABEL-QA and HuMMan-QA for fine-grained motion question answering, and HumanML3D for natural-language motion captioning. Metrics include exact QA accuracy, text-motion retrieval (R-Precision, MM-Distance), and standard captioning benchmarks (BLEU, ROUGE-L, CIDEr, BERTScore).

Key Results:

• Motion QA: SMD outperforms prior methods, achieving 66.7% on BABEL-QA and 90.1% on HuMMan-QA, a significant improvement over the previous best (60.1%/75.2% by IMoRe) and a strong demonstration of domain robustness (encoder-based baselines degrade severely when crossing data domains).
• Motion Captioning: On HumanML3D, SMD yields higher R@1 (0.584), CIDEr (53.16), and BERTScore (45.58) than all baselines, showing a marked increase in both retrieval-alignment and semantic similarity.

Numerical Highlights and Ablations

The superiority of SMD is especially evident when compared directly to encoder-based paradigms using the same LLM backbone. For example, the MotionGPT3-Qwen baseline achieves only 50.1% QA accuracy and lower captioning metrics despite matching model capacity, highlighting the impact of the representation rather than the model.

SMD also exhibits strong zero-shot properties: unadapted LLMs can already extract morphological and kinematic features from SMD, with QA accuracy significantly above chance and reasonable (if verbose) captions. However, LoRA fine-tuning is necessary to teach the model action-level semantics and generate concise descriptions.

Ablation studies further reveal:

• Granularity: Including only top-3 active joints maximizes QA accuracy; full-body SMDs improve captioning scores.
• Trajectory Coding: Absolute (world-frame) trajectory information slightly boosts retrieval/captioning, but SMD is effective even if trajectory info is omitted, due to joint-angle context.

Robustness, Portability, and Efficiency

SMD is validated on 8 LLMs from 6 families, ranging 3B–14B parameters; all perform consistently well. Swapping LLM backbones requires only LoRA retraining ($\sim$40M parameters, $\sim$8 GPU-hours). This dramatically simplifies pipeline reuse and model upgrading compared to encoder-based methods that require retraining a cross-modal projection stack.

Figure 4: Qualitative zero-shot captioning: even unadapted LLMs can analyze SMD, but concise action-aware captions require LoRA-based task adaptation.

Figure 5: Attention visualization during captioning: the LLM focuses on the SMD segments relevant to the described action—trajectory and cyclic joints for walking, and arm joints for waving.

Computation-wise, the main tradeoff is inference latency: LLMs process longer prompt texts than dense-token encodings. For top-3 SMDs, inference is slower by $\sim$10x relative to VAE-token representations, but with a corresponding gain in interpretability and robustness.

Model Interpretability

A key advantage of SMD is that it makes model decision-making directly interrogable. Attention heatmaps show that LLMs focus on relevant anatomical descriptors when answering questions or generating captions.

Figure 6: SMD computation relies on hierarchical biomechanical coordinate frames: the pelvis establishes the root body axes, from which all joint angles are measured, enabling accurate anatomical descriptions.

Figure 7: Visualization of a complete all-26 SMD for a waving motion: dynamic segments and body part headers are clearly marked, highlighting SMD's transparency.

Theoretical and Practical Implications

The SMD approach exposes a new interface between structured visual data and text-based reasoning in foundation models. By leveraging deterministic, semantically grounded representations, this method sidesteps generalization failures that plague cross-modal alignment. It also makes prompt engineering and interpretability tractable, which is especially valuable in applications demanding explainability, such as healthcare or robotics.

Practically, the rule-based design enhances domain transfer: SMD is robust to both sensor format and acquisition protocol, since anatomical rules are universal. The unified interface streamlines LLM upgrades and reduces data/time costs for adapting or scaling the system.

Theoretically, this work suggests that for modalities with rich, low-dimensional structure and interpretable domain-specific descriptors, adapting the data to language (rather than adapting models to data) can yield more general and robust systems. This could inspire parallel developments in other structured non-text modalities such as music, program analysis, or biomechanics, provided that a high-fidelity textual abstraction can be derived.

Conclusion

This paper demonstrates that structured, deterministic, and human-interpretable textual representations of human motion trajectories enable LLMs to match or exceed the performance of specialized encoder-based architectures on motion QA and captioning tasks. SMD removes the necessity for learned motion encoders, drastically lowers adaptation complexity, and provides valuable transparency for model interpretation and debugging.

Future work should address inference latency (possibly via input compression or blockwise processing), augment SMD for dense or high-DOF skeletons (e.g., hands/fingers), and explore learned SMD-style abstraction layers. Extending SMD to generative tasks (motion synthesis/editing) and integrating domain-specific priors for more nuanced biomechanical phenomena represents further high-impact research directions.