Motion State Encoder: Design & Applications

• Motion State Encoder is a design principle that converts raw observations and trajectories into compact, task-ready representations.
• It encompasses diverse architectures including alignment-based, state-space, hierarchical, and token-centric models for motion prediction and control.
• Recent empirical studies show improved performance in robotics, inertial navigation, human motion generation, and BEV forecasting through optimized state encoding.

Motion State Encoder is a literature-level term for modules that map observations, trajectories, or candidate motions into compact representations that preserve motion-relevant structure for downstream prediction, scoring, control, or reconstruction. Across recent arXiv work, the term has been used for image-conditioned robot motion reasoning, event-based optical flow, inertial odometry, trajectory forecasting, skeletal completion, multi-object tracking, BEV motion prediction, video segmentation, and compact robot state learning; in each case, the encoder serves as the interface between raw signals and a representation of motion dynamics or motion-conditioned state (Oba et al., 2023, Raju et al., 14 Apr 2025, Nguyen et al., 2024, Hu et al., 2024).

1. Conceptual scope

Across the literature, the phrase does not denote a single canonical architecture. In image-conditioned manipulation, the encoder integrates image features sampled along a projected hand trajectory with pose features so that an Energy-Based Model evaluates whether a motion is achievable in the observed scene (Oba et al., 2023). In event-based optical flow, the Motion State Encoder is a state-space feature encoder that processes event-frame triplets with a global spatial receptive field and linear-time complexity, then feeds recurrent refinement modules (Raju et al., 14 Apr 2025). In robotic imitation learning, it is a selective state-space model that compresses a 16-dimensional sequence of joint angles and torques into a 4-dimensional latent state for prediction and control (Tsuji, 2024). In unsupervised representation learning from pixels, it is an encoder that outputs a low-dimensional position state and derives velocity by finite differences, yielding a structured state $s_t=[p_t,v_t]$ (Jonschkowski et al., 2017).

The term is equally broad in sequence modeling. In inertial odometry, the encoder maps acceleration and angular velocity windows into latent motion features later decoded into velocity segments (Nguyen et al., 2024). In trajectory forecasting, it can mean a hierarchical encoder over location, velocity, and acceleration streams (Xue et al., 2021). In skeleton completion, it denotes a transformer that treats each joint-time pair as a token and reconstructs missing motion under dual masking (Jiang et al., 2022). In diffusion-based human motion generation, it denotes the tokenization path that converts absolute global joint coordinates or their learned latent into Transformer-ready motion tokens (Meng et al., 26 May 2025).

Domain	Encoder input	Encoded motion state
Image-conditioned robot motion	Image and motion trajectory	Cross-domain consistency representation for EBM/DMO
Event-based optical flow	Event-frame triplets	State-space spatiotemporal features and propagated motion state
Robotic imitation learning	Joint angles and torques	4D latent state $h_t$
Inertial navigation	Acceleration and angular velocity	Multimodal temporal latent for velocity estimation
Multi-object tracking	Historical box states and deltas	Flow feature $F$ or learned filter embeddings
Motion generation	Absolute joint coordinates	Token sequence for diffusion decoding

A common misconception is that a Motion State Encoder must be a recurrent latent-memory module. The literature includes recurrent encoders, such as ConvLSTM and Mamba-based systems, but also non-recurrent aligned encoders, masked autoencoders, variational encoders, and token compressors (Romaguera et al., 2020, Tsuji, 2024, Jiang et al., 2022, Liu et al., 6 Oct 2025). Another misconception is that the state must be low-dimensional. Some works use highly compressed latent states, such as 4D Mamba latents or two-token StaMo states, whereas others preserve dense spatial fields or long token sequences to retain geometric structure (Tsuji, 2024, Liu et al., 6 Oct 2025, Meng et al., 26 May 2025).

2. State definitions and representation choices

A defining property of these encoders is the explicit choice of what counts as “state.” In image-conditioned manipulation, each motion sequence is a set of $T+1$ time-stamped pose vectors $p_t$, where each pose is 11-dimensional: 3D hand position $x_t \in \mathbb{R}^3$, 6D continuous orientation $r_t \in \mathbb{R}^6$, gripper status $s_t \in [0,1]$, and a scalar time stamp (Oba et al., 2023). In HMNet for vehicle forecasting, the state is decomposed into positions $s_t=[x_t,y_t]$, first-order differences yielding velocity, and second-order differences yielding acceleration; this decomposition is treated as a proxy for moving trends and driving intentions (Xue et al., 2021). LTMSformer extends such state design to higher-order neighbor attributes by explicitly forming motion-state vectors with relative positions, acceleration, jerk, and heading (Yan et al., 7 Jul 2025).

In inertial settings, the state is not a pose trajectory but a multimodal temporal sequence. iMoT treats the IMU stream as $A=\{A_a,A_g\}\in\mathbb{R}^{2D\times T}$ with $h_t$0 channels for acceleration and angular velocity, and it predicts velocity segments $h_t$1 directly rather than performing classical double integration (Nguyen et al., 2024). In PVEs, the state is deliberately factorized into position and velocity, with velocity defined by finite differences,

$h_t$2

and acceleration used only as an auxiliary quantity for priors rather than as part of the concatenated state (Jonschkowski et al., 2017).

Other domains encode state as structured spatial tensors rather than vectors. D-MAE defines the motion state of a person as a window of 3D skeletal joints $h_t$3 together with a missing-joint indicator $h_t$4 (Jiang et al., 2022). LEGO-Motion defines a per-cell BEV state with future displacement, dynamic/static state, and semantic class, which can be summarized as $h_t$5 (Qian et al., 10 Mar 2025). PriorMotion likewise treats motion state as cell-wise future displacement fields, current cell classification, and static probability maps in BEV space (Qian et al., 2024). In TrackSSM and SIKNet for tracking, the state is built around box-centric parameters such as $h_t$6 or Kalman-style XYAH/XYWH state vectors with velocity terms (Hu et al., 2024, Song et al., 14 Sep 2025).

Human motion generation introduces yet another representation issue: coordinate frame. ACMDM explicitly argues for absolute global joint coordinates $h_t$7 rather than pelvis-relative or frame-relative encodings, and uses this absolute state as the input to a diffusion backbone (Meng et al., 26 May 2025). By contrast, the image-conditioned manipulation encoder projects 3D hand positions into the image plane with a pinhole camera model, using the projected coordinates as the alignment variable that ties motion state to image evidence (Oba et al., 2023). This suggests that representation choice is inseparable from the observation model: some encoders are fundamentally kinematic, some are camera-aligned, and some are occupancy- or token-centric.

3. Architectural families

One major family consists of alignment-based encoders that combine motion with another modality. The spatially-aligned temporal encoding of image-conditioned robot motion uses a UNet-shaped CNN to produce dense image features, samples those features at the projected hand locations by bilinear interpolation, encodes each pose with an MLP, concatenates image and pose features at each time step, and processes the resulting sequence with a Transformer before mean pooling and an MLP head (Oba et al., 2023). Motion-state alignment for video semantic segmentation uses a different dual structure: a motion alignment branch with a decoupled Transformer for dynamic semantics and a state alignment branch with a stage Transformer for static semantics, followed by semantic assignment between region descriptors and pixel descriptors (Su et al., 2023). Both architectures treat motion encoding as alignment rather than mere compression.

A second family is built around state-space models. P-SSE formulates feature encoding as a linear time-invariant state-space system with discretized recurrence

$h_t$8

then strengthens stability through a perturb-then-diagonalize modification of the dynamics matrix and applies the model to 2D feature scans with global receptive field and linear complexity (Raju et al., 14 Apr 2025). Mamba-based robotic imitation learning similarly combines an input linear compressor, convolutional layers, and a selective SSM block, using a 4-dimensional latent state as an autoencoder bottleneck (Tsuji, 2024). TrackSSM uses a stacked Mamba encoder over historical trajectory embeddings to produce a single flow feature $h_t$9, while SIKNet uses a two-step Semantic-Independent Encoder: a 1D convolution over homogeneous-semantic elements followed by a fully connected layer over heterogeneous-semantic elements before covariance tracking in a KalmanNet-style filter (Hu et al., 2024, Song et al., 14 Sep 2025).

A third family is hierarchical or recurrent. HMNet uses three separate streams—location, velocity, and acceleration—each with an input embedding MLP and an LSTM encoder, and fuses their outputs with motional social pooling (Xue et al., 2021). The free-breathing liver-scan predictor uses a three-scale convolutional encoder and a ConvLSTM at each scale; the hidden maps act as motion states that summarize respiratory phase and local deformation patterns and can be extrapolated open-loop to future frames (Romaguera et al., 2020). These designs emphasize temporal state accumulation rather than token-level global attention.

A fourth family is token-centric. D-MAE constructs one token per joint-time pair and augments the 3D coordinates with skeletal-structural and temporal Fourier encodings, using dual masking across joints and time (Jiang et al., 2022). iMoT tokenizes inertial signals by treating each variate’s full temporal sequence as a token, then augments the encoder with Progressive Series Decoupler, Adaptive Positional Encoding, and Adaptive Spatial Sync (Nguyen et al., 2024). ACMDM treats motion as a 2D grid over time and joints, patchifies only along the spatial joint dimension, and feeds the resulting tokens into a DiT-style Transformer (Meng et al., 26 May 2025). StaMo compresses a frozen DINOv2 feature map into exactly two tokens and uses those tokens to condition a pre-trained Diffusion Transformer decoder (Liu et al., 6 Oct 2025).

4. Probabilistic modeling, optimization, and motion reasoning

Motion State Encoders often sit inside larger probabilistic systems rather than serving as standalone feature extractors. In image-conditioned motion prediction, the aligned encoder feeds an Energy-Based Model with conditional density

$F$0

so that lower energy corresponds to higher task achievability (Oba et al., 2023). The same aligned representation is reused by Deep Motion Optimizer, which learns a direct refinement map

$F$1

from VAE-generated pseudo-initial motions to task-achieving motions, thereby avoiding the step-size and noise tuning associated with gradient-based sampling (Oba et al., 2023).

Variational and generative formulations are also common. The CVAE+DMP framework first inverts a second-order DMP-style dynamic system to recover a force sequence from a demonstration, encodes that force sequence with a 1D-CNN-based encoder into a latent variable $F$2, and decodes a task-conditioned force that is integrated through the same dynamic system toward a goal state (Xu et al., 2024). PriorMotion uses a Raster-Vector prior Encoder to fuse raster motion fields and instance-level vector priors, then places the resulting prior-enhanced representation into a Gaussian latent space through a Spatio-Temporal prior Generator with KL regularization and a spatial GRU decoder (Qian et al., 2024). StaMo trains a diffusion autoencoder in which a two-token encoder conditions a pre-trained Diffusion Transformer through cross-attention, and the difference between the two tokens, $F$3, is treated as an emergent latent action (Liu et al., 6 Oct 2025).

Cross-modal retrieval and uncertainty modeling appear prominently in inertial navigation. iMoT uses encoder-side PSD, APE, and ASC to produce modality-aligned inertial tokens, then introduces learnable query motion particles in the decoder and a Dynamic Scoring Mechanism that learns how to aggregate aligned particles into a velocity estimate (Nguyen et al., 2024). This design makes the encoder part of a broader latent-state inference process: the encoder produces the motion context, and the decoder resolves uncertainty over motion modes.

In tracking and motion estimation, the encoder often parameterizes a transition model rather than reconstructing motion directly. TrackSSM converts historical box states into a flow feature $F$4 that drives a time-varying state-space transition in Flow-SSM and supports cascaded decoding with Step-by-Step Linear pseudo-labels (Hu et al., 2024). SIKNet places its Semantic-Independent Encoder in front of learned predictive and innovation covariance estimators so that the Kalman gain is data-dependent and robust to parameter mismatch and non-stationary motion (Song et al., 14 Sep 2025). In both cases, motion-state encoding is tightly coupled to state transition estimation.

5. Applications and empirical behavior

Empirical results show that Motion State Encoders are not confined to one performance regime or one application family. In event-based optical flow, P-SSE achieves EPE 0.680 and AE 2.560 on DSEC-Flow, improving 8.48% over TMA and 9.33% over BFlow; on MVSEC zero-shot evaluation it attains mean EPE 0.52, improving 11.86% over the prior best reported results in that protocol (Raju et al., 14 Apr 2025). In robotic imitation learning, the Mamba encoder with a short input window $F$5 attains online success rates of 20/20 on Cup Placing and 18/20 on Case Loading, exceeding both a short-window Transformer and a full-sequence Transformer despite higher RMSE on some offline prediction metrics (Tsuji, 2024).

In human motion generation, ACMDM reports FID 0.058 and R-Precision Top-1 0.522 for ACMDM-XL-PS2 on HumanML3D, while its ControlNet-style controllable generation setup achieves average inference time per sample of 2.51 s versus 81.00 s for OmniControl, along with substantially lower trajectory and location errors (Meng et al., 26 May 2025). In compact robot state learning, StaMo reports a 14.3% improvement on LIBERO, 30% real-world task success improvement, and 10.4% gains in policy co-training, while preserving near-baseline inference speed at 4.02 Hz versus 4.16 Hz for OpenVLA (Liu et al., 6 Oct 2025).

Trajectory prediction and BEV motion forecasting show similar gains from explicit motion-state design. LTMSformer’s MSE alone reduces validation minADE from 0.687 to 0.673, minFDE from 1.030 to 1.001, and MR from 0.103 to 0.098 relative to the HiVT-64 base configuration (Yan et al., 7 Jul 2025). PriorMotion reports approximately 15.24% improvement in accuracy for fast-moving objects, a 3.59% increase in generalization, a reduction of 0.0163 in motion stability, and a 31.52% reduction in prediction errors in distant regions (Qian et al., 2024). LEGO-Motion improves mean displacement error across static, slow, and fast groups for multiple BEV backbones and reduces motion stability variance from 0.0267 to 0.0098 with an STPN backbone (Qian et al., 10 Mar 2025).

Medical, clinical, and motion-capture applications illustrate that the concept also transfers beyond standard robotics benchmarks. The recurrent multi-scale encoder-decoder for free-breathing liver MRI achieves a vessel-position error of 2.07 ± 2.95 mm for next-frame prediction and outperforms both PCA and a non-recurrent encoder-decoder across horizons up to five future frames (Romaguera et al., 2020). In Parkinson’s gait analysis, the benchmarked motion encoders vary substantially, but fine-tuned PoseFormerV2 reaches weighted F1 0.62 under leave-one-subject-out cross-validation and shows significant ON/OFF medication sensitivity with $F$6 (Adeli et al., 2024). D-MAE improves motion-capture robustness under severe occlusion, reaching Shelf PCP 97.4 and BU-Mocap PCP 97.7% with fine-tuning (Jiang et al., 2022).

These results suggest a consistent empirical pattern: when the encoded state reflects domain-specific motion structure—projected trajectories, event dynamics, inertial decomposition, hierarchical kinematics, instance-consistent BEV priors, or compact latent actions—downstream optimization or prediction benefits measurably.

6. Limitations, assumptions, and recurring design tensions

Despite their breadth, Motion State Encoders share several recurring limitations. Alignment-based encoders frequently assume accurate geometry or synchronization. The image-conditioned manipulation encoder requires known camera intrinsics and a fixed camera, has no explicit occlusion handling, and is sensitive to large pose errors in the projected trajectory (Oba et al., 2023). Event-based and inertial encoders can struggle under extreme sparsity, bursty dynamics, modality phase mismatch, or sensor saturation; P-SSE notes difficulties in sustained event-free regions and rapid event bursts, while iMoT notes that extreme motion or sensor saturation can degrade decomposition and alignment (Raju et al., 14 Apr 2025, Nguyen et al., 2024).

A second tension is between compactness and sufficiency. Mamba imitation learning shows that a 4D latent can suffice for targeted robotic tasks, but the same paper notes that performance may vary on tasks requiring more complex long-horizon reasoning and richer context (Tsuji, 2024). StaMo’s two-token representation is efficient and interpretable, but single-image ambiguity remains: some motions cannot be inferred from one frame alone, particularly when hidden contacts or object properties matter (Liu et al., 6 Oct 2025). PVEs show the opposite failure mode: derivatives obtained by finite differences amplify noise, and tasks with rapid, high-acceleration motion violate the slowness and inertia priors used for unsupervised state learning (Jonschkowski et al., 2017).

A third limitation concerns physical structure. Several works improve physical plausibility implicitly rather than through explicit constraints. HMNet obtains physically compliant prediction through hierarchical supervision on acceleration and velocity but does not impose explicit acceleration or jerk bounds (Xue et al., 2021). LEGO-Motion and PriorMotion improve motion stability and instance consistency without explicit kinematic or collision penalties (Qian et al., 10 Mar 2025, Qian et al., 2024). This suggests that “motion state encoding” often substitutes representational bias for hard physical modeling.

Generalization remains a major issue. Multiple papers are trained and evaluated under fixed viewpoints, simulation settings, specific robot morphologies, or curated benchmark protocols. The image-conditioned manipulation model is evaluated in RLBench with a Franka Panda and a fixed camera (Oba et al., 2023). The Parkinson’s benchmark notes a domain gap between healthy-motion pretraining and pathological gait, and labels are available per participant and medication state rather than per walk (Adeli et al., 2024). The liver-scan predictor is restricted to in-plane motion and inherits any registration error present in its label generation pipeline (Romaguera et al., 2020). These are not incidental weaknesses; they indicate that the meaning of “motion state” is often only partially transferable outside the data-generation assumptions under which it was learned.

Taken together, the literature presents Motion State Encoder not as a singular module but as a design principle: isolate motion-relevant state in a form that downstream models can optimize, score, decode, or control. The principal open question is therefore not whether such encoders are useful, but which state factorization, alignment mechanism, and inductive bias best preserve the motion structure required by a given task.