Motion Foundation Models

Updated 10 July 2026

Motion foundation models are defined by transferable motion priors that capture kinematics, temporal dynamics, geometry, and control rather than static appearance.
They utilize diverse representations—discrete tokenization, latent continuous spaces, and geometric parameterization—to enable tasks like action recognition, camera pose estimation, and motion planning.
Their advanced methodologies, including masked reconstruction and self-supervised pretraining, improve performance across video, wearable, and robotics benchmarks while addressing privacy challenges.

Motion foundation models are foundation-model approaches in which the transferable prior is organized around movement in space and time rather than static appearance alone. Across recent work, the term covers pretrained video models evaluated for motion understanding, human-motion backbones built from discrete vocabularies or generative priors, wearable and mobility sequence models, behavioral priors for physics-based control, and hybrid robotic stacks that use visual or language foundation models to inform motion planning and execution (Lee et al., 2024). The common objective is to learn, adapt, or exploit reusable structure over kinematics, temporal order, geometry, dynamics, and control so that downstream tasks such as action recognition, optical flow, camera-pose estimation, trajectory completion, anonymization, robot instruction following, and 4D scene reconstruction can be solved with less task-specific supervision or hand engineering (Xu et al., 7 Jul 2026).

1. Scope and problem setting

The literature does not present a single canonical architecture for motion foundation models. Instead, it defines a family of systems in which motion is the principal transferable object. In video understanding, motion is treated as a core capability complementary to appearance, with benchmarks explicitly separating motion-heavy datasets such as Something-Something-v2 and Diving-48 from appearance-heavy datasets such as Kinetics-400 and Moments-in-Time (Lee et al., 2024). In human motion generation and control, motion is modeled as a reusable latent prior for synthesis, editing, inpainting, tracking, or physics-based deployment (Baharani et al., 8 Feb 2025). In robotics, foundation models frequently supply semantic perception or language grounding while classical planners, automata, or control policies remain responsible for correctness and execution (Quartey et al., 2024).

This scope is unusually broad because “motion” appears at multiple levels of abstraction: pose evolution, ego-vehicle kinematics, optical flow, camera motion, block motion estimation, wearable inertial traces, and GPS-like trajectories. The field therefore spans not only human-motion generation but also dynamic-scene modeling, robot task-and-motion planning, mobility modeling, privacy-preserving motion release, and embodied evaluation.

Domain	Representative papers	Motion object
Video and visual reasoning	TWLV-I (Lee et al., 2024), FoundationMotion (Gan et al., 11 Dec 2025), EgoDyn-Bench (Schäfer et al., 22 Apr 2026)	Temporal dynamics in video
Human motion and control	MoFM (Baharani et al., 8 Feb 2025), UMO (Cong et al., 16 Mar 2026), BFMTrack (Rupf et al., 23 Jun 2026), FreeMotion (Zhang et al., 2024)	Body motion, latent control, synthesis
Robotics and embodiment	Layered framework (Yang et al., 2023), LIMP (Quartey et al., 2024)	Task-conditioned motion planning
Geometry, privacy, mobility, sensing	Uni4D (Yao et al., 27 Mar 2025), Endo-FASt3r (Zeinoddin et al., 10 Mar 2025), FlowSeek (Poggi et al., 5 Sep 2025), Pantomime (Hanisch et al., 13 Jan 2025), Inertia-1 (Xu et al., 7 Jul 2026), trajectory tutorial (Merten et al., 25 Nov 2025)	3D motion, anonymized motion, wearable or trajectory sequences

2. Representations and priors

A defining feature of the area is representational diversity. MoFM represents human motion not as raw RGB or sparse joint coordinates alone, but as spatio-temporal pose heatmaps $U \in \mathbb{R}^{J \times F \times H \times W}$ , partitioned into “Thermal Cubes” and discretized into a learned motion dictionary, MotionBook, with vocabulary size $8192$ (Baharani et al., 8 Feb 2025). This design places motion tokenization at the center of the model: the discrete vocabulary becomes a reusable basis for masked pretraining and downstream transfer.

Other works define motion priors in latent continuous spaces. Pantomime fits arbitrary motion into the SMPL model

$M(r,\Phi,\Theta,\beta),$

encodes the sequence with pretrained motion priors such as VPoser or HuMoR, perturbs the latent code, and decodes back to motion. Its anonymization step is explicitly

$a_t = \gamma p + z_t$

for static noise and

$a_t = \gamma p_t + z_t$

for variable noise, with static noise reported as superior because it preserves temporal coherence (Hanisch et al., 13 Jan 2025). The important point is that the foundation model is used as a plausibility manifold, not as a recognizer.

Behavioral Foundation Models adopt yet another representation. In BFMTrack, a policy $\pi(\mathbf{s},\mathbf{z})$ is conditioned on a latent $\mathbf{z}$ on the unit hypersphere, with a backward map $B(\mathbf{s})$ and forward map $F(\mathbf{s},\mathbf{a},\mathbf{z})$ . Motion tracking then becomes optimization over an entire latent trajectory $\{\mathbf{z}_t\}_{t=1}^T$ , rather than prompting the controller with a single stationary latent (Rupf et al., 23 Jun 2026). This reframes motion as a time-indexed control prior.

In vision-centric estimation, motion may be parameterized geometrically. FlowSeek derives optical-flow priors from inverse depth and low-dimensional motion bases corresponding to camera translations and rotations, replacing brute-force flow learning with a recurrent estimator conditioned on structured motion subspaces (Poggi et al., 5 Sep 2025). EgoDyn-Bench, by contrast, abstracts ego-motion into a deterministic oracle over hidden state sequences

$8192$0

mapping continuous kinematics to discrete semantic labels such as turn direction, braking intensity, or stop-and-go (Schäfer et al., 22 Apr 2026). The trajectory tutorial extends this logic to mobility data, treating sequences of $8192$1 observations as the motion tokens of a GPT-style spatiotemporal model, with delta encoding used for more stable prediction (Merten et al., 25 Nov 2025).

Taken together, these works suggest that motion foundation models are distinguished less by one shared architecture than by a shared commitment to reusable motion priors, whether those priors are discrete, latent, geometric, symbolic, or kinematic.

3. Learning paradigms, adaptation, and scaling

The training objectives in this area are heterogeneous. Masked reconstruction remains central in several strands. MoFM learns a discrete Variational Encoder-Decoder and then performs BERT-style masked prediction over motion tokens, with keypoint-aware masking preferred over naive blockwise masking because it forces reconstruction of motion-bearing regions rather than empty background (Baharani et al., 8 Feb 2025). TWLV-I also treats motion understanding as a first-class pretraining target and argues that fair comparison of video foundation models requires standardized evaluation across motion and appearance, using linear probing, attentive probing, KNN, and downstream temporal tasks (Lee et al., 2024).

Large-scale self-supervised exploration is especially explicit in wearable sensing. Inertia-1 studies 10 pretraining approaches across 15 datasets and 3 task families, using a corpus of 18.2M hours of accelerometer data from 115,450 people. Its main empirical conclusion is that self-supervised pretraining broadly helps, but no single objective dominates across all tasks, and data scale matters more consistently than model size (Xu et al., 7 Jul 2026). The paper’s practical recipe favors triaxial accelerometer input, wrist placement, 20 Hz sampling, 30-second windows, time-domain modeling, and moderate model size.

Other systems focus on adaptation rather than from-scratch pretraining. Endo-FASt3r adapts foundation models for both monocular depth and relative pose estimation in endoscopic self-supervised learning, introducing Reloc3rX for stable pose regression in SSL and DoMoRA for higher-rank adaptation. On SCARED it reports AbsRel $8192$2, ATE-T1 $8192$3, ATE-T2 $8192$4, and summarizes the gains as about $8192$5 improvement in pose and about $8192$6 improvement in depth over prior work (Zeinoddin et al., 10 Mar 2025). UMO fine-tunes a pretrained DiT-based text-to-motion model into a unified in-context generator through three frame-level meta-operations—preserve, generate, and edit—with temporal fusion adding only $8192$7M parameters (Cong et al., 16 Mar 2026).

Data curation itself has also become a foundation-model problem. FoundationMotion addresses the scarcity of fine-grained motion supervision by automatically converting raw videos into motion-centric captions and QA pairs through detection, tracking, and LLM generation. The resulting dataset contains about 46.7K videos and 467K caption/QA pairs, averaging 10.04 questions per video and 1.671 questions per second (Gan et al., 11 Dec 2025). This moves the field toward scalable motion supervision rather than narrow hand annotation.

4. Embodied planning, control, and instruction following

A major branch of the literature uses foundation models as semantic modules inside larger motion-planning stacks rather than as end-to-end controllers. The layered robotics framework of “Perception $8192$8 Cognition $8192$9 Planning $M(r,\Phi,\Theta,\beta),$ 0 Execution $M(r,\Phi,\Theta,\beta),$ 1 Learning” places visual foundation models primarily in the perception layer, where RGB-D observations are converted into object detection, scene understanding, and spatial relationship modeling. The reported results include $M(r,\Phi,\Theta,\beta),$ 2 object recognition and scene understanding accuracy, $M(r,\Phi,\Theta,\beta),$ 3 to $M(r,\Phi,\Theta,\beta),$ 4 task success across manipulation tasks, efficient path finding in cluttered environments $M(r,\Phi,\Theta,\beta),$ 5 of the time, and $M(r,\Phi,\Theta,\beta),$ 6 improvement in path planning compared with traditional methods (Yang et al., 2023). The contribution is architectural: the planner itself remains based on established methods.

LIMP makes the same modular commitment more formal. Natural-language instructions are translated into LTL,

$M(r,\Phi,\Theta,\beta),$ 7

then grounded in a Referent Semantic Map through VLM detections, Composible Referent Descriptors, and spatial predicates, before motion is synthesized by a task-and-motion stack using automata and FMT*. The abstract reports evaluation on 150 instructions across five real-world environments, with a $M(r,\Phi,\Theta,\beta),$ 8 success rate on complex spatiotemporal instructions versus $M(r,\Phi,\Theta,\beta),$ 9 for baselines; the detailed summary also reports $a_t = \gamma p + z_t$ 0 success on object-goal navigation and $a_t = \gamma p + z_t$ 1 on mobile manipulation for the best prompting variant (Quartey et al., 2024). Its central claim is “correct-by-construction” behavior because planning is constrained by the specification itself.

FreeMotion shows a different hybrid pattern. It uses a multimodal LLM, specifically GPT-4V, as a keyframe designer and keyframe animator, then fills motion between keyframes via interpolation and environment-aware motion tracking, all without motion-capture data (Zhang et al., 2024). BFMTrack extends physics-based control in another direction: by optimizing entire latent sequences with temporally correlated noise, it supports dense tracking, sparse keyframing, and direct deployment onto a humanoid robot (Rupf et al., 23 Jun 2026). Across these systems, foundation models contribute semantics, decomposition, and priors, while classical optimization, formal methods, or physics-based control supply robustness.

5. Visual motion understanding, geometry, and benchmarking

Several papers argue that motion understanding is limited less by model scale than by the availability of motion-specific supervision or physically aligned representations. FoundationMotion provides the clearest example: auto-labeled motion data improves NVILA-Video-15B by $a_t = \gamma p + z_t$ 2 on MotionBench, $a_t = \gamma p + z_t$ 3 on AV-Car, $a_t = \gamma p + z_t$ 4 on Daily, and $a_t = \gamma p + z_t$ 5 on Robotics, and enables a mid-sized model to outperform Gemini-2.5 Flash and Qwen2.5-VL-72B on some motion benchmarks (Gan et al., 11 Dec 2025). TWLV-I reaches the same conclusion from the evaluation side, reporting average top-1 linear probing accuracy of $a_t = \gamma p + z_t$ 6 for ViT-L and showing that existing video foundation models often trade off motion understanding against appearance understanding rather than excelling at both (Lee et al., 2024).

A second recurring finding is that geometric priors remain indispensable. Uni4D combines CoTracker3, UniDepthV2, RAM, GPT-4o, Grounding-SAM, and DEVA in a three-stage optimization pipeline for camera poses, static geometry, and dynamic 3D trajectories, all without retraining or fine-tuning. Its temporal optimization improves the reported depth consistency score from $a_t = \gamma p + z_t$ 7 for UniDepth to $a_t = \gamma p + z_t$ 8 for Uni4D (Yao et al., 27 Mar 2025). FlowSeek similarly injects depth-foundation-model outputs and motion bases into a RAFT-style optical-flow estimator, trains on a single RTX 3090 GPU, and reports about $a_t = \gamma p + z_t$ 9 relative improvement over SEA-RAFT on Sintel Final and about $a_t = \gamma p_t + z_t$ 0 on KITTI, while remaining compact (Poggi et al., 5 Sep 2025). FAST-ME applies the same principle to block motion estimation, combining SAD with ViT-, SAM-, and CLIP-derived semantic attention; it reports over $a_t = \gamma p_t + z_t$ 1 reduction relative to Full Search, with Foreman results of PSNR $a_t = \gamma p_t + z_t$ 2 and semantic coverage score $a_t = \gamma p_t + z_t$ 3 (Panagidi et al., 22 May 2026).

Benchmarking also reveals unresolved weaknesses. EgoDyn-Bench audits 20+ models and identifies a “Perception Bottleneck”: foundation models can often reason over explicit motion text but fail to align ego-motion concepts with visual evidence. On the geometrically answerable subset, a Visual Odometry baseline reaches balanced accuracy $a_t = \gamma p_t + z_t$ 4, exceeding GPT-5.1 at $a_t = \gamma p_t + z_t$ 5 and Qwen3-VL-8B at $a_t = \gamma p_t + z_t$ 6; yet when explicit kinematic time series are supplied, Qwen3-VL-8B rises from $a_t = \gamma p_t + z_t$ 7 vision-only balanced accuracy to $a_t = \gamma p_t + z_t$ 8 (Schäfer et al., 22 Apr 2026). This indicates that motion logic may be present but not visually grounded.

6. Privacy, limitations, and future directions

Motion foundation models are not only generative or perceptual tools; they also expose privacy and governance issues. Pantomime begins from the observation that human motion is a behavioral biometric and uses foundation motion models as anonymizing priors. On CeTI-Locomotion and Horst-DB it reduces identification accuracy to $a_t = \gamma p_t + z_t$ 9 while preserving naturalness better than direct corruption, with VPoser-based anonymization giving the best privacy-utility balance in the reported experiments (Hanisch et al., 13 Jan 2025). This extends the scope of motion foundation models from recognition and control to privacy-preserving release.

The field’s limitations are equally consistent across domains. Some systems remain architectural and descriptive rather than formally specified; the layered robotics framework explicitly provides no formal mathematical framework, no detailed ablations by layer, and no standardized benchmark suite (Yang et al., 2023). Others are computationally heavy: Pantomime requires about 1.5 weeks for CeTI-Locomotion on a single RTX 3090, making live streaming unsuitable in its current implementation (Hanisch et al., 13 Jan 2025). Endo-FASt3r inherits the usual monocular SSL issues of scale ambiguity and sensitivity to photometric assumptions (Zeinoddin et al., 10 Mar 2025). UMO is limited to whole-body control and language conditioning, without part-level operations or audio inputs (Cong et al., 16 Mar 2026). EgoDyn-Bench shows that large multimodal models still underutilize visual motion cues even when their language-side physical logic is relatively strong (Schäfer et al., 22 Apr 2026).

Future directions follow directly from these bottlenecks. Robotics papers emphasize adaptation to new tasks, more complex and dynamic real-world scenes, and multi-agent environments (Yang et al., 2023). Surgical motion modeling points toward video foundation models for improved temporal stability (Zeinoddin et al., 10 Mar 2025). Wearable sensing argues for data-first scaling, broader modality and placement diversity, and evaluation beyond human activity recognition into gait and disease prediction (Xu et al., 7 Jul 2026). Motion-generation work points toward richer in-context interfaces, better geometric control, and broader multimodal conditioning (Cong et al., 16 Mar 2026). A plausible implication is that the next stage of the field will be less about naming a single “motion foundation model” family than about establishing interoperable motion priors—tokenized, latent, geometric, and symbolic—that can be grounded across perception, reasoning, privacy, and control.