MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos (2512.10881v1)

Abstract: Motion capture now underpins content creation far beyond digital humans, yet most existing pipelines remain species- or template-specific. We formalize this gap as Category-Agnostic Motion Capture (CAMoCap): given a monocular video and an arbitrary rigged 3D asset as a prompt, the goal is to reconstruct a rotation-based animation such as BVH that directly drives the specific asset. We present MoCapAnything, a reference-guided, factorized framework that first predicts 3D joint trajectories and then recovers asset-specific rotations via constraint-aware inverse kinematics. The system contains three learnable modules and a lightweight IK stage: (1) a Reference Prompt Encoder that extracts per-joint queries from the asset's skeleton, mesh, and rendered images; (2) a Video Feature Extractor that computes dense visual descriptors and reconstructs a coarse 4D deforming mesh to bridge the gap between video and joint space; and (3) a Unified Motion Decoder that fuses these cues to produce temporally coherent trajectories. We also curate Truebones Zoo with 1038 motion clips, each providing a standardized skeleton-mesh-render triad. Experiments on both in-domain benchmarks and in-the-wild videos show that MoCapAnything delivers high-quality skeletal animations and exhibits meaningful cross-species retargeting across heterogeneous rigs, enabling scalable, prompt-driven 3D motion capture for arbitrary assets. Project page: https://animotionlab.github.io/MoCapAnything/

• The paper presents a prompt-based, category-agnostic method that reconstructs 3D joint trajectories from monocular videos using arbitrary rigged assets.
• It leverages a multi-modal architecture including a Reference Prompt Encoder, DINOv2 feature extractor, and an IK optimization layer to produce temporally coherent rotations.
• Quantitative tests on the Truebones Zoo benchmark show a marked reduction in CD-Skeleton error, demonstrating robust performance across diverse and unseen skeletons.

MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos

Introduction and Motivation

The "MoCapAnything" framework (2512.10881) introduces a fundamentally novel approach to monocular motion capture, extending beyond fixed-species paradigms to enable prompt-based, category-agnostic 3D motion capture (CAMoCap). Departing from human- or specific-animal-centric models such as SMPL or SMAL, this method leverages an arbitrary rigged 3D asset as a reference prompt and reconstructs temporally-coherent, rotation-based animations directly in the asset’s skeleton space from monocular RGB sequences. This formulation addresses the limitations of prior work—namely brittle angle regression across heterogeneous topologies, the lack of robust retargeting protocols for arbitrary assets, and poor generalization in in-the-wild, non-template scenarios.

Framework Architecture

The core architecture comprises three learnable modules and an optimization-based Inverse Kinematics (IK) fitting layer, as illustrated in Figure 1.

Figure 1: Architecture of MoCapAnything delineating the multi-modal Reference Prompt Encoder, mesh/video feature extraction, Unified Motion Decoder, and asset-specific IK layer.

Reference Prompt Encoder

This encoder fuses the mesh geometry, skeletal topology, and rendered appearances of the reference asset into per-joint queries. Graph-based multi-head attention mechanisms—augmented by kinematic biases—afford explicit structure-aware reasoning over arbitrary skeletons.

Video Feature Extractor

A frozen DINOv2 backbone extracts dense visual descriptors per frame and reconstructs a coarse 4D deforming mesh from the monocular video, providing geometry-aware cues that bridge the modality gap between high-dimensional RGB tokens and sparse joint distributions.

Unified Motion Decoder

Jointly attending to reference queries and multimodal extracted features, the decoder produces temporally smooth 3D joint trajectories. Its integration of intraframe graph attention, temporal cross-attention to visual and geometric streams, and per-joint temporal modeling enables robust handling of occlusions, rapid motions, and topology diversity.

IK Fitting

Rather than direct angle regression, asset-specific joint rotations are reconstructed via a constrained, temporally-stable IK optimization—enforcing hierarchy, bone lengths, and joint limits while suppressing under-constrained bone-axis twist.

Dataset and Evaluation Protocol

The Truebones Zoo benchmark is introduced, providing 1,038 motion clips spanning a broad taxonomy of animal and synthetic rigs. Each clip includes a standardized skeleton-mesh-rendered video tuple, enabling both in-domain assessments and rigorous generalization protocols: "Seen" (ample training samples), "Rare" (limited data), and "Unseen" (never observed species).

Experimental Analysis

Quantitative Results

MoCapAnything demonstrates a substantial reduction in structural motion error compared to GenZoo—the current SOTA for animal mocap—when evaluated with CD-Skeleton metrics (see Figure 2).

Figure 2: Qualitative and structural error comparison with GenZoo, highlighting improved plausibility and generalization across non-quadruped skeletons.

On the complete Truebones Zoo test set, CD-Skeleton error is decreased from 0.4580 (GenZoo) to 0.2549. Notably, the framework maintains robust performance for non-quadruped and unseen categories, a persistent challenge for template-bound models.

Ablation studies across input modalities and architectural components indicate the critical value of mesh and graph-attention paths, especially for rare/unseen splits—diminishing MPJPE and MPJVE markedly when retained.

Qualitative Analysis

Generalization to heterogeneous species and out-of-domain (Objaverse) skeletons is illustrated in Figure 3 and Figure 4.

Figure 3: Cross-species retargeting on the Truebones Zoo test set, demonstrating stable motions across humans, reptiles, quadrupeds, and non-animal rigs.

Figure 4: In-the-wild generalization results on internet animal videos, preserving plausible skeletal motion and retargeting ability under challenging, unconstrained inputs.

The framework supports unconstrained retargeting and synthesizes semantically plausible animations when transferring motion across fundamentally different morphologies (see Figures 5 and 6).

Figure 5: Generalization to diverse real-world species and scenarios beyond controlled test data.

Figure 6: Unconstrained cross-species retargeting examples, transferring chicken/flamingo motions to raptor/jaguar skeletons.

Technical Innovations

• Factorized Architecture: Decomposes motion recovery into trajectory prediction and rotation fitting, stabilizing learning and improving transferability.
• Multimodal Integration: Mesh cues bridge dense RGB features and sparse joint spaces, enhancing geometric groundedness and supporting arbitrary skeletal topologies.
• Graph Attention: Structure-aware message passing through kinematic trees generalizes across unknown morphologies.
• Prompt-Based Retargeting: Leverages mesh, skeleton, and appearance prompts to transfer motion between domains, with no explicit cross-species training objective.
• Efficient IK Optimization: Combines geometric initialization, warm-starting, and twist regularization for temporally coherent, anatomically plausible rotations under broad constraints.

Implications and Future Directions

Practically, MoCapAnything signals a shift toward unified, scalable mocap solutions for content creation, animation, and virtual production—circumventing the need for species-specific parametric models or template engineering. Its modular, promptable design anticipates workflows spanning games, VTubers, artistic retargeting, and digital asset crowds.

Theoretically, this architecture suggests that explicit structural modeling—via graph-based attention and mesh-guided cues—substantially improves domain transfer in motion tasks, underscoring the limitations of direct regression or vision-only methods in category-agnostic settings.

Anticipated research avenues include:

• End-to-end, contact- and physics-aware IK pipelines;
• Video-only geometry priors eliminating dependence on 4D reconstructor quality;
• World-grounded trajectory inference for robust physical realism;
• Multimodal and text-only prompting for asset description and retargeting;
• Multi-character motion interaction and crowd synthesis via arbitrary rig coordination.

Conclusion

MoCapAnything inaugurates prompt-based, category-agnostic motion capture from monocular video, substantiated by robust quantitative and qualitative analyses across heterogeneous skeletal domains. The architecture's modular factorization and explicit multimodal integration confer state-of-the-art performance in CAMoCap, yielding stable, high-fidelity animations and unconstrained retargeting ability. By dissolving species and template boundaries, this framework represents a foundational advance toward general-purpose motion reconstruction and retargeting systems—poised for broad impact in scalable animation and AI-driven asset workflows.