---
title: Motion Foundation Models
url: https://www.emergentmind.com/topics/motion-foundation-models
type: topic
---

# Motion Foundation Models

Motion foundation models are foundation-model approaches in which the transferable prior is organized around movement in space and time rather than static appearance alone. Across recent work, the term covers pretrained video models evaluated for motion understanding, human-motion backbones built from discrete vocabularies or generative priors, wearable and mobility sequence models, behavioral priors for physics-based control, and hybrid robotic stacks that use visual or language foundation models to inform motion planning and execution [2408.11318]. The common objective is to learn, adapt, or exploit reusable structure over kinematics, temporal order, geometry, dynamics, and control so that downstream tasks such as action recognition, optical flow, camera-pose estimation, trajectory completion, anonymization, robot instruction following, and 4D scene reconstruction can be solved with less task-specific supervision or hand engineering [2607.06617].

## 1. Scope and problem setting

The literature does not present a single canonical architecture for motion foundation models. Instead, it defines a family of systems in which motion is the principal transferable object. In video understanding, motion is treated as a core capability complementary to appearance, with benchmarks explicitly separating motion-heavy datasets such as Something-Something-v2 and Diving-48 from appearance-heavy datasets such as Kinetics-400 and Moments-in-Time [2408.11318]. In human motion generation and control, motion is modeled as a reusable latent prior for synthesis, editing, inpainting, tracking, or physics-based deployment [2502.05432]. In robotics, foundation models frequently supply semantic perception or language grounding while classical planners, automata, or control policies remain responsible for correctness and execution [2402.11498].

This scope is unusually broad because “motion” appears at multiple levels of abstraction: pose evolution, ego-vehicle kinematics, optical flow, camera motion, block motion estimation, wearable inertial traces, and GPS-like trajectories. The field therefore spans not only human-motion generation but also dynamic-scene modeling, robot task-and-motion planning, mobility modeling, privacy-preserving motion release, and embodied evaluation.

| Domain | Representative papers | Motion object |
|---|---|---|
| Video and visual reasoning | TWLV-I [2408.11318], FoundationMotion [2512.10927], EgoDyn-Bench [2604.22851] | Temporal dynamics in video |
| Human motion and control | MoFM [2502.05432], UMO [2603.15975], BFMTrack [2606.25056], FreeMotion [2406.10740] | Body motion, latent control, synthesis |
| Robotics and embodiment | Layered framework [2309.11244], LIMP [2402.11498] | Task-conditioned motion planning |
| Geometry, privacy, mobility, sensing | Uni4D [2503.21761], Endo-FASt3r [2503.07204], FlowSeek [2509.05297], Pantomime [2501.07149], Inertia-1 [2607.06617], trajectory tutorial [2511.20610] | 3D motion, anonymized motion, wearable or trajectory sequences |

## 2. Representations and priors

A defining feature of the area is representational diversity. MoFM represents human motion not as raw RGB or sparse joint coordinates alone, but as spatio-temporal pose heatmaps \(U \in \mathbb{R}^{J \times F \times H \times W}\), partitioned into “Thermal Cubes” and discretized into a learned motion dictionary, MotionBook, with vocabulary size \(8192\) [2502.05432]. This design places motion tokenization at the center of the model: the discrete vocabulary becomes a reusable basis for masked pretraining and downstream transfer.

Other works define motion priors in latent continuous spaces. Pantomime fits arbitrary motion into the SMPL model
\[
M(r,\Phi,\Theta,\beta),
\]
encodes the sequence with pretrained motion priors such as VPoser or HuMoR, perturbs the latent code, and decodes back to motion. Its anonymization step is explicitly
\[
a_t = \gamma p + z_t
\]
for static noise and
\[
a_t = \gamma p_t + z_t
\]
for variable noise, with static noise reported as superior because it preserves temporal coherence [2501.07149]. The important point is that the foundation model is used as a plausibility manifold, not as a recognizer.

Behavioral Foundation Models adopt yet another representation. In BFMTrack, a policy \(\pi(\mathbf{s},\mathbf{z})\) is conditioned on a latent \(\mathbf{z}\) on the unit hypersphere, with a backward map \(B(\mathbf{s})\) and forward map \(F(\mathbf{s},\mathbf{a},\mathbf{z})\). Motion tracking then becomes optimization over an entire latent trajectory \(\{\mathbf{z}_t\}_{t=1}^T\), rather than prompting the controller with a single stationary latent [2606.25056]. This reframes motion as a time-indexed control prior.

In vision-centric estimation, motion may be parameterized geometrically. FlowSeek derives optical-flow priors from inverse depth and low-dimensional motion bases corresponding to camera translations and rotations, replacing brute-force flow learning with a recurrent estimator conditioned on structured motion subspaces [2509.05297]. EgoDyn-Bench, by contrast, abstracts ego-motion into a deterministic oracle over hidden state sequences
\[
s_t = [v_t,a_t,j_t,\omega_t,\theta_t]^\top,
\]
mapping continuous kinematics to discrete semantic labels such as turn direction, braking intensity, or stop-and-go [2604.22851]. The trajectory tutorial extends this logic to mobility data, treating sequences of \((\text{latitude}, \text{longitude}, t)\) observations as the motion tokens of a GPT-style spatiotemporal model, with delta encoding used for more stable prediction [2511.20610].

Taken together, these works suggest that motion foundation models are distinguished less by one shared architecture than by a shared commitment to reusable motion priors, whether those priors are discrete, latent, geometric, symbolic, or kinematic.

## 3. Learning paradigms, adaptation, and scaling

The training objectives in this area are heterogeneous. Masked reconstruction remains central in several strands. MoFM learns a discrete Variational Encoder-Decoder and then performs BERT-style masked prediction over motion tokens, with keypoint-aware masking preferred over naive blockwise masking because it forces reconstruction of motion-bearing regions rather than empty background [2502.05432]. TWLV-I also treats motion understanding as a first-class pretraining target and argues that fair comparison of video foundation models requires standardized evaluation across motion and appearance, using linear probing, attentive probing, KNN, and downstream temporal tasks [2408.11318].

Large-scale self-supervised exploration is especially explicit in wearable sensing. Inertia-1 studies 10 pretraining approaches across 15 datasets and 3 task families, using a corpus of 18.2M hours of accelerometer data from 115,450 people. Its main empirical conclusion is that self-supervised pretraining broadly helps, but no single objective dominates across all tasks, and data scale matters more consistently than model size [2607.06617]. The paper’s practical recipe favors triaxial accelerometer input, wrist placement, 20 Hz sampling, 30-second windows, time-domain modeling, and moderate model size.

Other systems focus on adaptation rather than from-scratch pretraining. Endo-FASt3r adapts foundation models for both monocular depth and relative pose estimation in endoscopic self-supervised learning, introducing Reloc3rX for stable pose regression in SSL and DoMoRA for higher-rank adaptation. On SCARED it reports AbsRel \(0.051\), ATE-T1 \(0.0702\), ATE-T2 \(0.0438\), and summarizes the gains as about \(10\%\) improvement in pose and about \(2\%\) improvement in depth over prior work [2503.07204]. UMO fine-tunes a pretrained DiT-based text-to-motion model into a unified in-context generator through three frame-level meta-operations—preserve, generate, and edit—with temporal fusion adding only \(0.207\)M parameters [2603.15975].

Data curation itself has also become a foundation-model problem. FoundationMotion addresses the scarcity of fine-grained motion supervision by automatically converting raw videos into motion-centric captions and QA pairs through detection, tracking, and LLM generation. The resulting dataset contains about 46.7K videos and 467K caption/QA pairs, averaging 10.04 questions per video and 1.671 questions per second [2512.10927]. This moves the field toward scalable motion supervision rather than narrow hand annotation.

## 4. Embodied planning, control, and instruction following

A major branch of the literature uses foundation models as semantic modules inside larger motion-planning stacks rather than as end-to-end controllers. The layered robotics framework of “Perception \(\rightarrow\) Cognition \(\rightarrow\) Planning \(\rightarrow\) Execution \(\rightarrow\) Learning” places visual foundation models primarily in the perception layer, where RGB-D observations are converted into object detection, scene understanding, and spatial relationship modeling. The reported results include \(92\%\) object recognition and scene understanding accuracy, \(85\%\) to \(95\%\) task success across manipulation tasks, efficient path finding in cluttered environments \(88\%\) of the time, and \(15\%\) improvement in path planning compared with traditional methods [2309.11244]. The contribution is architectural: the planner itself remains based on established methods.

LIMP makes the same modular commitment more formal. Natural-language instructions are translated into LTL,
\[
\varphi := p \mid \neg\varphi_1 \mid \varphi_1 \wedge \varphi_2 \mid \mathcal{X}\varphi \mid \varphi_1 \mathcal{U} \varphi_2,
\]
then grounded in a Referent Semantic Map through VLM detections, Composible Referent Descriptors, and spatial predicates, before motion is synthesized by a task-and-motion stack using automata and FMT\*. The abstract reports evaluation on 150 instructions across five real-world environments, with a \(79\%\) success rate on complex spatiotemporal instructions versus \(38\%\) for baselines; the detailed summary also reports \(90\%\) success on object-goal navigation and \(71\%\) on mobile manipulation for the best prompting variant [2402.11498]. Its central claim is “correct-by-construction” behavior because planning is constrained by the specification itself.

FreeMotion shows a different hybrid pattern. It uses a multimodal large language model, specifically GPT-4V, as a keyframe designer and keyframe animator, then fills motion between keyframes via interpolation and environment-aware motion tracking, all without motion-capture data [2406.10740]. BFMTrack extends physics-based control in another direction: by optimizing entire latent sequences with temporally correlated noise, it supports dense tracking, sparse keyframing, and direct deployment onto a humanoid robot [2606.25056]. Across these systems, foundation models contribute semantics, decomposition, and priors, while classical optimization, formal methods, or physics-based control supply robustness.

## 5. Visual motion understanding, geometry, and benchmarking

Several papers argue that motion understanding is limited less by model scale than by the availability of motion-specific supervision or physically aligned representations. FoundationMotion provides the clearest example: auto-labeled motion data improves NVILA-Video-15B by \(+1.0\) on MotionBench, \(+7.1\) on AV-Car, \(+2.4\) on Daily, and \(+14.9\) on Robotics, and enables a mid-sized model to outperform Gemini-2.5 Flash and Qwen2.5-VL-72B on some motion benchmarks [2512.10927]. TWLV-I reaches the same conclusion from the evaluation side, reporting average top-1 linear probing accuracy of \(48.61\) for ViT-L and showing that existing video foundation models often trade off motion understanding against appearance understanding rather than excelling at both [2408.11318].

A second recurring finding is that geometric priors remain indispensable. Uni4D combines CoTracker3, UniDepthV2, RAM, GPT-4o, Grounding-SAM, and DEVA in a three-stage optimization pipeline for camera poses, static geometry, and dynamic 3D trajectories, all without retraining or fine-tuning. Its temporal optimization improves the reported depth consistency score from \(0.109\) for UniDepth to \(0.043\) for Uni4D [2503.21761]. FlowSeek similarly injects depth-foundation-model outputs and motion bases into a RAFT-style optical-flow estimator, trains on a single RTX 3090 GPU, and reports about \(10\%\) relative improvement over SEA-RAFT on Sintel Final and about \(15\%\) on KITTI, while remaining compact [2509.05297]. FAST-ME applies the same principle to block motion estimation, combining SAD with ViT-, SAM-, and CLIP-derived semantic attention; it reports over \(98\%\) reduction relative to Full Search, with Foreman results of PSNR \(26.12\) and semantic coverage score \(69.7\) [2605.23428].

Benchmarking also reveals unresolved weaknesses. EgoDyn-Bench audits 20+ models and identifies a “Perception Bottleneck”: foundation models can often reason over explicit motion text but fail to align ego-motion concepts with visual evidence. On the geometrically answerable subset, a Visual Odometry baseline reaches balanced accuracy \(63.8\%\), exceeding GPT-5.1 at \(55.1\%\) and Qwen3-VL-8B at \(52.3\%\); yet when explicit kinematic time series are supplied, Qwen3-VL-8B rises from \(38.9\%\) vision-only balanced accuracy to \(62.2\%\) [2604.22851]. This indicates that motion logic may be present but not visually grounded.

## 6. Privacy, limitations, and future directions

Motion foundation models are not only generative or perceptual tools; they also expose privacy and governance issues. Pantomime begins from the observation that human motion is a behavioral biometric and uses foundation motion models as anonymizing priors. On CeTI-Locomotion and Horst-DB it reduces identification accuracy to \(10\%\) while preserving naturalness better than direct corruption, with VPoser-based anonymization giving the best privacy-utility balance in the reported experiments [2501.07149]. This extends the scope of motion foundation models from recognition and control to privacy-preserving release.

The field’s limitations are equally consistent across domains. Some systems remain architectural and descriptive rather than formally specified; the layered robotics framework explicitly provides no formal mathematical framework, no detailed ablations by layer, and no standardized benchmark suite [2309.11244]. Others are computationally heavy: Pantomime requires about 1.5 weeks for CeTI-Locomotion on a single RTX 3090, making live streaming unsuitable in its current implementation [2501.07149]. Endo-FASt3r inherits the usual monocular SSL issues of scale ambiguity and sensitivity to photometric assumptions [2503.07204]. UMO is limited to whole-body control and language conditioning, without part-level operations or audio inputs [2603.15975]. EgoDyn-Bench shows that large multimodal models still underutilize visual motion cues even when their language-side physical logic is relatively strong [2604.22851].

Future directions follow directly from these bottlenecks. Robotics papers emphasize adaptation to new tasks, more complex and dynamic real-world scenes, and multi-agent environments [2309.11244]. Surgical motion modeling points toward video foundation models for improved temporal stability [2503.07204]. Wearable sensing argues for data-first scaling, broader modality and placement diversity, and evaluation beyond human activity recognition into gait and disease prediction [2607.06617]. Motion-generation work points toward richer in-context interfaces, better geometric control, and broader multimodal conditioning [2603.15975]. A plausible implication is that the next stage of the field will be less about naming a single “motion foundation model” family than about establishing interoperable motion priors—tokenized, latent, geometric, and symbolic—that can be grounded across perception, reasoning, privacy, and control.

Source: https://www.emergentmind.com/topics/motion-foundation-models