Dense Motion Tokenization

• Dense motion tokenization is a method that transforms continuous motion signals into fine-grained, structured token grids over space and time.
• It employs vector quantization and codebook strategies to encode diverse motion features, balancing reconstruction fidelity with semantic granularity.
• The technique underpins applications in unsupervised discovery, human motion animation, and video understanding by enabling precise motion control and conditioning.

Dense motion tokenization denotes a family of representation-learning strategies that convert motion-bearing signals into structured token sequences or token grids at fine spatio-temporal granularity. The literature does not use the term in a single canonical sense. In one line of work, motion shapes a dense latent appearance codebook for unsupervised object discovery; in another, raw $3$D human motion is quantized on a frame$\times$joint lattice; elsewhere, dense $2$D or $3$D spatio-temporal heatmaps are compressed into local latent code indices; high-FPS video understanding uses patchwise residual tokenization conditioned on change; and some systems represent camera trajectories and human poses as continuous spatio-temporal control tokens rather than discrete codebook entries (Bao et al., 2023, Ding et al., 15 May 2025, Maldonado et al., 23 Sep 2025, Zhang et al., 17 Sep 2025, Li et al., 11 Apr 2025). A recurrent theme is that “dense” usually means that tokenization is local over time, space, body structure, or latent grids, rather than collapsing an entire motion sequence into one global symbol. At the same time, the literature also draws an explicit boundary between tokenization and continuous motion embedding: FlowFeat, for example, is a pixel-dense embedding of motion profiles, but “does not perform explicit tokenization” in the codebook sense (Araslanov et al., 10 Nov 2025).

1. Meanings of density in current usage

In the literature represented here, dense motion tokenization spans several distinct token units. In "Object Discovery from Motion-Guided Tokens" (Bao et al., 2023), the eventual tokens are dense, spatially localized quantized mid-level features on a latent $2$D grid, while motion enters through external sparse instance-level motion segmentation masks supervising slot attention. In "MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation" (Ding et al., 15 May 2025), density means a structured token field spanning the whole sequence over temporal positions and body joints. In "Adversarially-Refined VQ-GAN with Dense Motion Tokenization for Spatio-Temporal Heatmaps" (Maldonado et al., 23 Sep 2025), density refers to local latent cells over downsampled spatio-temporal heatmap volumes. In "Dense Video Understanding with Gated Residual Tokenization" (Zhang et al., 17 Sep 2025), dense tokenization means preserving all frames from a high-FPS video while tokenizing only changed patches and then merging semantically redundant scene content. In "TokenMotion" (Li et al., 11 Apr 2025), density refers to spatio-temporal tokens aligned to the latent video grid for both camera trajectories and human poses.

This diversity is not terminological noise; it reflects different choices about what the atomic motion-bearing unit should be. A token may correspond to a latent spatial position, a temporally downsampled frame-joint slot, a local heatmap patch in a $T\times H\times W$ lattice, a residual video patch, or a fused camera-pose latent patch. This suggests that dense motion tokenization is better treated as an umbrella concept for fine-grained motion-structured tokenization than as one fixed algorithmic template.

Setting	Token unit	Representative paper
Motion-guided object discovery	Latent $2$D grid token	(Bao et al., 2023)
Human $3$D motion animation	Frame$\times$joint token	(Ding et al., 15 May 2025)
Heatmap compression	Local space-time latent cell	(Maldonado et al., 23 Sep 2025)
High-FPS video understanding	Changed image patch / residual token	(Zhang et al., 17 Sep 2025)
Human-centric video generation	Latent spatio-temporal control token	(Li et al., 11 Apr 2025)

A related distinction concerns whether motion itself is tokenized directly. MoTok is explicit that it does not tokenize flow vectors or trajectories directly; it learns a dense codebook over visual mid-level latent features, with motion used as unsupervised guidance so that the resulting tokens become object-aware (Bao et al., 2023). By contrast, MTVCrafter directly quantizes normalized $3$D joint trajectories, and the spatio-temporal heatmap VQ-GAN directly tokenizes dense heatmap motion representations (Ding et al., 15 May 2025, Maldonado et al., 23 Sep 2025).

2. Quantization mechanisms and codebook design

The dominant discrete formulation is vector quantization on a structured latent grid. MoTok defines a VQ-VAE codebook

$\times$0

computes a dense latent map $\times$1, and performs nearest-neighbor assignment at each spatial position: $\times$2 Its central coupling is then to force a slot-decoded dense map $\times$3 to align with the quantized latent map $\times$4, making the token space jointly shaped by motion-guided slot learning and vector-quantized reconstruction (Bao et al., 2023).

MTVCrafter implements a $\times$5D motion VQ-VAE, 4DMoT, on the $\times$6 lattice of frames and joints. The encoder uses $\times$7D convolutions over frame and joint axes, temporally downsamples by $\times$8, and produces one latent vector per temporally downsampled frame position and joint. Quantization uses a codebook

$\times$9

with one discrete token per latent slot $2$0. The implementation states $2$1, EMA decay $2$2, and unused-code resetting every 20 steps. Because only frames are downsampled, the latent grid has size roughly $2$3, so tokenization remains dense over body structure even after temporal compression (Ding et al., 15 May 2025).

The heatmap VQ-GAN line uses the same core idea at a different signal level. Encoder outputs $2$4 are mapped to the nearest codebook embedding $2$5, producing a token map over a reduced spatio-temporal lattice. The paper makes the compression especially concrete by reporting three latent sizes for an original $2$6 sequence: $2$7 These correspond to compression factors of $2$8, $2$9, and $3$0, respectively. The same study reports an important modality effect: for $3$1D motion at F16, a 128-token vocabulary is optimal, whereas for $3$2D motion at F16, 1024 tokens are needed for high-fidelity reconstruction (Maldonado et al., 23 Sep 2025).

Not all quantizers use hard nearest-neighbor VQ. GeoMotionGPT adopts a decoder-only quantizer with differentiable Gumbel-Softmax: $3$3 and adds both codebook-utilization entropy regularization and orthogonality regularization on the codebook Gram matrix. Its codebook size is $3$4, code dimensionality is $3$5, and the paper uses temporally downsampled motion features rather than frame-level quantization (Ye et al., 12 Jan 2026).

A separate design axis concerns single-codebook versus multi-codebook or multi-scale tokenization. LG-Tok retains MoSa-style multi-scale residual quantization, producing $3$6 scales with total token counts of 104, 160, or 236 depending on the variant, while MotionLLaMA’s HoMi Tokenizer explicitly seeks RVQ-like reconstruction with a single codebook of size 2048 and code dimension 1536, aided by separate hand/body encoders and frequency-aware gating (Yan et al., 9 Feb 2026, Ling et al., 2024).

3. Organization of token spaces

Dense motion tokenization is rarely a flat sequence model. The most informative systems organize tokens according to explicit structure in space, time, anatomy, or task.

MoTok begins with object-centric slot learning but reconstructs a dense feature map rather than a slot list. From a dense spatiotemporal feature map $3$7, it computes $3$8 slots and corresponding attention masks, then decodes them back to a dense spatial map $3$9. The key point is that the tokens are not object IDs. They are latent grid tokens produced downstream of motion-guided grouping, so the architecture combines object-centric factorization with dense spatial tokenization (Bao et al., 2023).

MTVCrafter is structurally different: one token corresponds to one latent slot at one temporally downsampled frame position and one joint position. Because the encoder operates jointly over frame and joint axes using stacked $2$0D convolutions, dilated convolutions, and a sliding window of size $2$1, each token is local in index but contextual in content. The resulting representation is dense not merely in time but over the full spatio-temporal body lattice (Ding et al., 15 May 2025).

MoGeFlow advances a third arrangement: each frame is represented by a structured set of group-specific tokens rather than a single global token. It inherits a frozen PartVQ tokenizer with $2$2 data-derived joint groups, each with its own codebook of size 128. The frame state is

$2$3

The paper’s central claim is that these codebooks exhibit measurable, non-random, decoder-causal geometry. Across the six groups on held-out HumanML3D, the mean Spearman correlation between embedding distance and local motion-prototype distance is $2$4, while a shuffled control drops to $2$5. Near, mid, and far code replacements induce monotonically larger decoded motion changes, which the paper interprets as evidence that local neighborhoods in codebook space are physically meaningful (Fang et al., 10 Jun 2026).

LMR proposes a dual-granularity alternative in which one latent manifold is explicitly coarse and semantic, and the other remains kinematically dense. The reasoning branch uses temporal length $2$6, while the execution branch remains at length $2$7 in the main formulation. Generation is factorized as

$2$8

so planning tokens are generated first and dense execution tokens second. The paper’s tokenizer ablation makes the motivation explicit: execution at $2$9 resolution without reasoning improves reconstruction but hurts direct generation, whereas execution at $T\times H\times W$0 with reasoning at $T\times H\times W$1 gives the best joint trade-off (Qian et al., 30 Dec 2025).

LG-Tok addresses a related problem from a different angle. Rather than making the motion stream denser, it argues that dense token sequences can become semantically inefficient. Its Transformer tokenizer therefore uses language during tokenization and detokenization so that motion tokens become more semantically compact. This is a different interpretation of token organization: not denser temporal factorization, but denser semantic content per token (Yan et al., 9 Feb 2026).

4. Conditioning, control, and multimodal interfaces

Dense motion tokens frequently serve as conditioning signals rather than end tasks in themselves. MTVCrafter couples 4DMoT to MV-DiT, a motion-aware video diffusion transformer built on CogVideoX-5B-T2V. Motion tokens are flattened and used as the context sequence for dedicated $T\times H\times W$2D motion-attention layers, with $T\times H\times W$3D positional encoding

$T\times H\times W$4

The motion attention uses visual tokens as queries and motion tokens as keys and values, and the tokenizer is pretrained and frozen during full-system training. In this configuration, dense motion tokenization becomes a control interface for open-world human image animation rather than merely a compression layer (Ding et al., 15 May 2025).

TokenMotion presents a continuous analogue. Camera trajectories are converted into dense Plücker ray tensors

$T\times H\times W$5

human motion is represented with DWPose cues, and both are patchified with $T\times H\times W$6D spatial and $T\times H\times W$7D causal temporal convolutions into latent-aligned spatio-temporal control tokens. These two token streams are processed separately and then fused by a human-aware dynamic mask before cross-attention into a DiT backbone. The fused representation is therefore dense, local, and interaction-aware, but continuous rather than codebook-quantized (Li et al., 11 Apr 2025).

GRT approaches conditioning from the perspective of efficient dense video understanding. It defines residual frames

$T\times H\times W$8

where changed patches are identified by patchwise SSIM thresholding. Key frames are tokenized densely; subsequent P-frames contribute only changed-patch tokens. A second stage then merges semantically redundant scenes by collapsing redundant key tokens while preserving dynamic P-frame tokens. The result is a residual token stream designed for high-FPS video understanding rather than motion generation (Zhang et al., 17 Sep 2025).

Motion tokenization also functions as an interface to LLMs. MotionLLaMA’s HoMi Tokenizer produces discrete motion tokens from normalized global joint coordinates and feeds them into an autoregressive multimodal LLM together with text and audio tokens, using a single codebook shared across single-person and dual-person motion. GeoMotionGPT similarly tokenizes motion for LLM-based understanding, but insists that the motion codebook and the LLM motion embeddings should share a unified geometric basis via sparse projection and orthogonality regularization (Ling et al., 2024, Ye et al., 12 Jan 2026).

These uses show that dense motion tokenization is not only about compression. It also defines how motion becomes queryable, conditionable, and compositional inside diffusion transformers, autoregressive priors, and multimodal LLMs.

5. Empirical patterns and design trade-offs

Several stable empirical patterns recur across the literature. First, dense tokenization quality depends strongly on the decoder or downstream interface. In MoTok, linear and CNN decoders perform poorly relative to attention-based decoders; on MOVi-E in VQ-space and without motion, performance improves from 14.6 for Linear to 16.3 for Linear-CNN to 40.1 for Transformer to 52.4 for Perceiver. The same study reports that motion guidance and dense tokenization are complementary rather than substitutable: on MOVi-E, MoTok rises from 52.4 without motion to 63.8 with motion; on TRI-PD, it reaches 60.6 with GT motion and 55.0 with estimated motion; on KITTI, it reaches 64.4. The paper also reports token cluster purity improving from 40.1 for vanilla VQ-VAE to 65.7 for MoTok, with chance at 38.5, and GPU memory of 10.9 GB for MoTok versus 66.7 GB for STEVE on TRI-PD (Bao et al., 2023).

Second, discrete motion tokens can improve generation quality even when a continuous autoencoder is available. MTVCrafter reports TikTok scores of FVD 140.60, FID-VID 6.98, PSNR 19.37, SSIM 0.784, LPIPS 0.217, and FID 19.46. Its tokenizer ablation is direct: replacing 4DMoT with a continuous autoencoder worsens FID-VID from 6.98 to 9.79, with declines in SSIM and LPIPS as well. The appendix further reports codebook utilization of 69.7% active codes at inference, only 2.9% frequent, and 30.3% underutilized (Ding et al., 15 May 2025).

Third, vocabulary size is modality-dependent rather than arbitrary. The spatio-temporal heatmap VQ-GAN reports that at F16 compression, $T\times H\times W$9D motion performs best with a compact 128-token vocabulary, with SSIM $2$0, PSNR $2$1, $2$2, T-Std $2$3, and Q-loss $2$4, whereas $2$5D motion requires 1024 tokens for SSIM $2$6. Its strongest headline comparison is between the best $2$7D VQ-GAN and $2$8D dVAE at F8 compression: SSIM improves from $2$9 to $3$0, a 9.31% improvement, while T-Std drops from $3$1 to $3$2, a 37.1% reduction (Maldonado et al., 23 Sep 2025).

Fourth, support coverage matters as much as architecture. Beyond MoCap argues that a motion tokenizer trained only on narrow MoCap support learns a restricted motion vocabulary. With temporal downsampling factor $3$3, hidden/code dimension 512, and codebook expanded to 2048, the study finds that the best OOD tokenizer uses a real:synthetic ratio of $3$4, giving HumanML3D reconstruction FID 0.076, Top-1 0.504, Motion-X++ overall MPJPE 252.4, and codebook utilization 76.4%. The same work reports that larger codebooks become under-utilized beyond $3$5, while $3$6 retains about 76% active codes (Yan et al., 25 Jun 2026).

Fifth, more tokens do not monotonically improve downstream generation. LG-Tok makes this explicit with a token-count study: 104 tokens gives reconstruction rFID $3$7 and generation gFID $3$8; 160 tokens gives rFID $3$9 and gFID $\times$0; 236 tokens gives rFID $\times$1 but gFID worsens to $\times$2. The paper interprets this as a semantic efficiency problem. Its language-guided tokenizer improves Motion-X perplexity from 146.5 to 103.1 and generation FID from 0.257 to 0.088, while LG-Tok-mini uses only 104 total tokens yet remains highly competitive (Yan et al., 9 Feb 2026).

Finally, planning and execution benefit from different temporal resolutions. LMR reports that execution at $\times$3 with no reasoning yields rFID 0.044, MPJPE 16.608, but gFID 0.242; the dual-granularity setting with execution at $\times$4 and reasoning at $\times$5 preserves the same rFID 0.044 and MPJPE 16.608 while improving R1 to 0.474, ACC to 0.538, and gFID to 0.040 (Qian et al., 30 Dec 2025).

6. Misconceptions, limitations, and open directions

A central misconception is that dense motion tokenization always means direct discretization of motion fields. MoTok is the clearest counterexample: motion does not enter as optical flow inside the reconstruction loss, and the method is better characterized as motion-guided dense appearance tokenization than motion tokenization per se (Bao et al., 2023). A second misconception is that any dense motion-informed representation is a tokenizer. FlowFeat explicitly rejects that characterization: it learns a continuous dense feature map $\times$6, with $\times$7, by distilling optical-flow-consistent motion profiles into single-image features, but it does not use vector quantization, discrete codebooks, or learned motion vocabularies (Araslanov et al., 10 Nov 2025).

Another recurrent point is that “dense” does not necessarily mean frame-level at the original sampling rate. MTVCrafter downsamples time by $\times$8 while keeping density over joints; GeoMotionGPT tokenizes temporally downsampled motion features rather than original frames; MotionLLaMA uses a downsampling rate of 4 in HoMi; LMR explicitly advocates dual rates rather than uniform high-frequency tokenization (Ding et al., 15 May 2025, Ye et al., 12 Jan 2026, Ling et al., 2024, Qian et al., 30 Dec 2025). This suggests that density is best understood relative to the structure being preserved, not as literal one-token-per-frame encoding.

The limitations reported by the papers are likewise heterogeneous. MoTok notes object/background ambiguity, slot drift over long sequences, degraded temporal consistency in later frames, and challenges in real-world crowded scenes (Bao et al., 2023). MTVCrafter is human-specific because it relies on SMPL with 24-body-joint structure, depends on the quality of $\times$9D pose extraction, and explicitly notes poor precise hand control (Ding et al., 15 May 2025). The heatmap VQ-GAN degrades at aggressive F32 compression and does not analyze discriminator structure in detail (Maldonado et al., 23 Sep 2025). MoGeFlow acknowledges that terminal projection back to valid codes may introduce projection error or ambiguity and points to future work on better projection mechanisms (Fang et al., 10 Jun 2026). GRT leaves several implementation details under-specified, including scene-boundary construction and threshold values for $3$0 and $3$1 (Zhang et al., 17 Sep 2025).

There are also unresolved questions about what makes a tokenizer “good.” GeoMotionGPT argues for geometry-aware codebooks and LLM embeddings; Beyond MoCap argues that support coverage is the primary bottleneck; LG-Tok argues that semantic compression, not token density alone, is the critical variable; and LMR argues that a single latent stream cannot jointly optimize planning semantics and execution fidelity (Ye et al., 12 Jan 2026, Yan et al., 25 Jun 2026, Yan et al., 9 Feb 2026, Qian et al., 30 Dec 2025). These positions are not mutually exclusive, but they imply different optimization targets: codebook geometry, codebook utilization, semantic efficiency, or hierarchical factorization.

A plausible implication is that dense motion tokenization is converging toward a design space with three orthogonal axes: the carrier being tokenized (appearance latent, joints, heatmaps, residual patches, or fused control fields), the structure imposed on tokens (flat, lattice-based, part-factorized, multi-scale, or dual-granularity), and the role of the tokens downstream (reconstruction, conditioning, reasoning, or LLM interfacing). The most reusable ideas across the surveyed work are joint optimization of structured quantized latents with motion-aware grouping, alignment of token layout to the underlying spatial or anatomical lattice, explicit management of codebook diversity and utilization, and architectures that preserve motion structure after flattening through positional encodings or geometry-aware priors (Bao et al., 2023, Ding et al., 15 May 2025, Fang et al., 10 Jun 2026).