OmniTokenizer: Unified Visual Tokenizer for Media

• OmniTokenizer is a transformer-based unified visual tokenizer that jointly processes images and videos using shared parameters.
• It employs a spatial-temporal decoupled architecture with a progressive training strategy to enhance reconstruction performance across benchmarks.
• The method supports both discrete and continuous tokenization via OmniTokenizer-VQVAE and OmniTokenizer-VAE, catering to autoregressive and diffusion models.

Searching arXiv for OmniTokenizer and closely related tokenizer work.

OmniTokenizer is a transformer-based tokenizer for joint image and video tokenization that maps visual data into a compact latent space for generative modeling. It is presented as a unified framework with shared parameters that can handle both static images and dynamic videos, addressing the prior separation between image-only and video-only tokenizers. The method is instantiated in two forms: OmniTokenizer-VQVAE, which produces discrete tokens for language-model-based generation, and OmniTokenizer-VAE, which produces continuous latents for latent diffusion models. Its central technical claims are a spatial-temporal decoupled architecture, a progressive image-to-image-video training strategy, and empirical state-of-the-art reconstruction performance on both image and video benchmarks (Wang et al., 2024).

1. Problem formulation and motivation

OmniTokenizer is motivated by the observation that visual tokenization serves as the representational bottleneck of visual generative models. In the formulation used by the paper, a tokenizer functions as a translator that maps high-dimensional visual signals into a compact latent representation that downstream generators can model efficiently. For language-model-style generation this typically means discrete tokens, whereas diffusion-style generation typically uses a continuous latent space (Wang et al., 2024).

The paper argues that prior tokenizers were largely specialized either for images or for videos. Image tokenizers were optimized for spatial structure, while video tokenizers had to additionally model temporal dynamics. This specialization limited parameter sharing, prevented direct exploitation of complementary supervision from image and video data, and reduced practical flexibility when both image and video generation were required. A further motivation was data imbalance: image data are abundant and diverse, whereas video data are comparatively scarce, so a video tokenizer trained only on video data does not benefit from large-scale spatial supervision available from image corpora (Wang et al., 2024).

A unified tokenizer is therefore presented as desirable for four reasons stated in the source: better representation learning through joint appearance and motion supervision, mitigation of video data scarcity, a shared visual vocabulary or latent space, and better practical scalability. The paper frames OmniTokenizer as the first tokenizer to handle both images and videos within one unified framework and one shared set of weights (Wang et al., 2024).

This framing later became a reference point in subsequent work. For example, VFRTok describes OmniTokenizer as a representative prior tokenizer that compresses video on a fixed frame grid and associates it with the “Frame-Proportional Information Assumption,” under which token count scales linearly with the number of frames (Zhong et al., 17 May 2025). This later comparison is significant because it locates OmniTokenizer historically as a canonical fixed-grid visual tokenizer rather than merely one implementation among many.

2. Model architecture and latent interface

OmniTokenizer uses a transformer-based architecture with explicit spatial-temporal decoupling. The overall model consists of patch embedding layers, a space-time transformer encoder with separate spatial and temporal blocks, a latent bottleneck, a symmetric decoder, and final linear projections back to pixel space (Wang et al., 2024).

The input is defined as

$$x \in \mathbb{R}^{(1+T) \times H \times W \times 3},$$

where $1+T$ is the number of frames, with $T=0$ for an image. A key design choice is that the first frame and the remaining frames are handled separately. The first frame,

$x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$

is split into 2D non-overlapping patches of size $p \times p$, while the remaining frames,

$x_{1:T},$

are split into 3D non-overlapping patches of size $t \times p \times p$. The resulting embeddings are

$$e_0 \in \mathbb{R}^{L_1 \times c}, \qquad e_{1:T} \in \mathbb{R}^{L_2 \times c},$$

with

$$L_1 = \frac{H}{p} \times \frac{W}{p}, \qquad L_2 = \frac{T}{t} \times \frac{H}{p} \times \frac{W}{p}.$$

These are concatenated to form the full space-time token sequence (Wang et al., 2024).

The encoder is factorized into separate spatial and temporal attention paths. Spatial modeling uses window attention from Swin Transformer, with the paper emphasizing local aggregation capacity and efficiency at higher resolutions. Temporal modeling uses causal attention so that a token can attend only to current and previous times, which is justified both for motion modeling and for compatibility with autoregressive visual generation. In the reported implementation, the tokenizer uses 4 spatial layers based on window attention, 4 temporal layers based on causal attention, a window size of 8, a hidden dimension of 512, and a latent dimension of 8 (Wang et al., 2024).

The latent bottleneck has two variants. OmniTokenizer-VQVAE uses vector quantization against a learned codebook, yielding discrete visual tokens. OmniTokenizer-VAE is obtained by KL fine-tuning the VQVAE model into a continuous latent tokenizer suitable for diffusion. For image tokenization on $256 \times 256$ inputs, the latent shape is $1+T$0, implying 1024 discrete tokens when flattened. The VQ configuration uses a codebook size of 8192 in the image experiments, while the VAE variant is marked as having codebook size $1+T$1 (Wang et al., 2024).

The decoder is described as symmetric with the encoder and ends with two linear projection layers without activation, mirroring the separate treatment of the first frame and subsequent temporal chunks (Wang et al., 2024).

3. Discrete and continuous tokenization schemes

OmniTokenizer explicitly supports both discrete and continuous tokenization. This duality is central to its positioning because it aims to serve both major visual generation paradigms with one tokenizer family (Wang et al., 2024).

In OmniTokenizer-VQVAE, the encoder outputs are quantized using a learned codebook. The discrete latent grid is then flattened in raster order and passed to an autoregressive transformer for next-token prediction. The vector-quantization objective is

$1+T$2

where $1+T$3 is the encoder, $1+T$4 is the patch embedding sequence, $1+T$5 is the selected codebook vector, and $1+T$6 is the stop-gradient operator. The implementation sets $1+T$7 and $1+T$8 (Wang et al., 2024).

In OmniTokenizer-VAE, the discrete bottleneck is replaced by a Gaussian latent formulation regularized by

$1+T$9

where $T=0$0 is the learned posterior and $T=0$1 is a Gaussian prior. The paper sets $T=0$2. This conversion is presented as a KL fine-tuning step after VQ training and is used to produce continuous latent maps for diffusion models (Wang et al., 2024).

Beyond $T=0$3 or $T=0$4, both training regimes also employ $T=0$5 reconstruction loss $T=0$6 and GAN loss $T=0$7. The paper does not provide a single combined objective formula for all losses, but it states that these losses are used during both VQ training and KL fine-tuning (Wang et al., 2024).

The downstream autoregressive language-model objective over flattened code indices is written as

$T=0$8

where $T=0$9 is the flattened sequence of codebook indices and $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$0 is optional conditioning such as a class label (Wang et al., 2024).

A later paper, VFRTok, uses OmniTokenizer as a comparison point for fixed-grid video tokenization. In that framing, OmniTokenizer is reported in Table 1 as using 4096 tokens with latent dimension 8 under a standard $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$1 compression regime for a 16-frame, $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$2 video. VFRTok’s critique is that such tokenization retains a fixed temporal compression rate, so token count still grows linearly with frame count and hence with FPS for fixed-duration clips (Zhong et al., 17 May 2025). This suggests that OmniTokenizer’s latent interface is compact and effective for reconstruction, but still tied to a frame-indexed temporal grid.

4. Progressive training strategy

A defining feature of OmniTokenizer is its two-stage progressive training strategy. The paper argues that naive joint training is insufficient and that curriculum design is necessary to realize genuine image-video synergy (Wang et al., 2024).

In stage 1, the tokenizer is trained on image data only at fixed resolution $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$3 for 500K iterations. The stated purpose is to learn strong spatial encoding, appearance, texture, and structure before introducing temporal complexity. In stage 2, the same tokenizer weights are trained jointly on image and video data for another 500K iterations. During this stage, the spatial resolution is randomly sampled from 128, 192, 256, 320, and 384, and the video sequence length is 17 frames. The paper states that image and video data are forwarded iteratively during this joint stage (Wang et al., 2024).

This curriculum is empirically important. The ablations reported in the source show that video-only training is clearly worse than joint training on video benchmarks, and that naive joint fixed-resolution training helps videos but hurts image reconstruction. By contrast, the proposed progressive image-first and then joint multi-resolution training gives the best image performance and the best video performance simultaneously. In the summarized findings, image-only fixed-resolution training gives ImageNet rFID 1.28, joint fixed-resolution training worsens this to 1.35, and the proposed joint multi-resolution progressive strategy improves it to 1.11 (Wang et al., 2024).

The reported training details are specific. Optimization uses Adam with $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$4 and $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$5, with learning rate warmup to $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$6 followed by cosine decay to 0. Data augmentation is random horizontal flip only. Training is reported to require 8 NVIDIA A100 GPUs for 2 weeks (Wang et al., 2024).

The significance of this training strategy extends beyond the original paper. Later unified-tokenizer work such as UniFlow frames the central difficulty as the conflict between semantic abstraction for understanding and low-level fidelity for generation, and proposes a different resolution via layer-wise adaptive self-distillation and a patch-wise pixel flow decoder (Yue et al., 12 Oct 2025). Relative to that later design space, OmniTokenizer represents an earlier unified-tokenizer paradigm in which joint image-video learning and architectural factorization are the primary mechanisms for synergy.

5. Empirical performance

OmniTokenizer reports strong reconstruction results on both image and video datasets. On image reconstruction, Table 1 in the source reports the following results at latent shape $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$7: on CelebA-HQ, ViT-VQGAN obtains rFID 4.66 while OmniTokenizer-VQVAE achieves 1.93; on FFHQ, ViT-VQGAN obtains 3.13 while OmniTokenizer-VQVAE achieves 1.91; on ImageNet, DALL-E obtains 32.01, VQGAN 1.49, ViT-VQGAN 1.28, OmniTokenizer-VQVAE 1.11, and OmniTokenizer-VAE 0.69 (Wang et al., 2024).

The headline image result emphasized by the paper is 1.11 reconstruction FID on ImageNet for the VQVAE tokenizer, which the abstract states beats the previous state of the art by 13%. The VAE variant further improves to 0.69 rFID (Wang et al., 2024).

On video reconstruction, Table 2 reports UCF-101 and Moments-in-Time rFVD. The source lists MaskGIT at 240 on UCF, VQGAN at 299 on UCF and 306 on MiT, ViT-VQGAN at 167 on MiT, CViViT at 66 on MiT, TATS at 162 on UCF, MAGVIT at 58 on UCF, OmniTokenizer-VQVAE at 42 on UCF and 20 on MiT, and OmniTokenizer-VAE at 23 on UCF and 13 on MiT (Wang et al., 2024). The abstract highlights 42 reconstruction FVD on UCF-101, described as beating the previous state of the art by 26% (Wang et al., 2024).

A broader ablation table across video datasets reports that the best setting, “Ours-Joint (Multi),” achieves ImageNet 256 rFID 1.11, K600 128 rFVD 84.38, K600 256 rFVD 25.97, UCF 128 rFVD 107.80, UCF 256 rFVD 42.35, MiT 128 rFVD 59.47, MiT 256 rFVD 19.87, SSV2 128 rFVD 84.78, and SSV2 256 rFVD 20.30 (Wang et al., 2024).

The paper also reports downstream generative results. For class-conditional ImageNet autoregressive generation, a 227M AR model with OmniTokenizer yields 10.13 FID and 94.5 IS, compared with 18.65 FID and 80.4 IS for VQGAN; a 650M AR model with OmniTokenizer yields 7.45 FID and 146.7 IS, compared with 8.81 FID and 110.8 IS for ViT-VQGAN (Wang et al., 2024).

For video generation and frame prediction, the reported table shows Phenaki at 36.4 FVD on K600, MAGVIT at 76 on UCF and 9.9 on K600, MAGVITv2 at 58 on UCF and 4.3 on K600, TATS at 332 on UCF, and OmniTokenizer-based AR models at 314 UCF / 34.2 K600 for 227M parameters and 191 UCF / 32.9 K600 for 650M parameters (Wang et al., 2024). The prose states that OmniTokenizer-based autoregressive video generation beats previous SOTA autoregressive methods like TATS on UCF-101, although the table itself does not place it ahead of MAGVITv2 (Wang et al., 2024).

For diffusion-based generation, the image results with DiT are reported as DiT-XL/2 at 9.62 FID and 121.50 IS, DiT-XL/2-CFG at 2.27 FID and 278.24 IS, Ours-DiT-XL/2 at 12.25 FID and 109.94 IS, and Ours-DiT-XL/2-CFG at 3.48 FID and 244.23 IS (Wang et al., 2024). These numbers show compatibility with diffusion transformers but do not surpass the strongest DiT baseline in ImageNet FID. On UCF-101 with Latte, however, the paper reports Latte at latent compression $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$8 with FVD 478.0, while Ours-Latte uses latent compression $x_0 \in \mathbb{R}^{1 \times H \times W \times 3},$9 and achieves FVD 209.2 (Wang et al., 2024).

6. Significance, later reinterpretations, and limitations

OmniTokenizer’s main significance lies in demonstrating that joint image-video tokenization can be both feasible and beneficial. Its contribution is not only that one model can accept both image and video inputs, but that joint training with shared parameters can improve tokenization quality across both modalities when the architecture and curriculum are appropriately designed (Wang et al., 2024).

The architecture and efficiency ablation further supports this interpretation. The paper compares a full space-time JointAttn model, a decoupled plain-attention model, and the proposed design with spatial window attention and temporal causal attention. Reported results are: VQGAN at 167 iGFLOPs and $p \times p$0 vGFLOPs with rFID 1.49; JointAttn at 72 iGFLOPs and 358 vGFLOPs with rFID 1.89; DePlainAttn at 72 iGFLOPs and 299 vGFLOPs with rFID 1.33; and the proposed OmniTokenizer design at 51 iGFLOPs and 262 vGFLOPs with rFID 1.28 (Wang et al., 2024). This places the decoupled architecture as both more efficient and more accurate than the compared attention baselines.

At the same time, later work has clarified some of OmniTokenizer’s limitations. VFRTok explicitly treats OmniTokenizer as a representative fixed-frame-grid tokenizer whose latent token count still scales linearly with frame count. The later critique is that such tokenization reflects a “Frame-Proportional Information Assumption,” whereas VFRTok proposes a “Duration-Proportional Information Assumption” and variable frame-rate encoding/decoding instead (Zhong et al., 17 May 2025). In that later comparison, OmniTokenizer is reported to require 4096 tokens for a 16-frame, $p \times p$1 clip and 10240 tokens at 60 FPS, with downstream DiT cost rising from 5.82 TFLOPs to 22.67 TFLOPs (Zhong et al., 17 May 2025). This does not invalidate OmniTokenizer’s original achievements; rather, it identifies a scaling issue that became more salient as video generation moved toward higher frame rates.

The original paper itself states several caveats. Some architectural details are under-specified, including the exact temporal patch size $p \times p$2, the full adversarial loss form, the discriminator architecture, and codebook-usage statistics beyond brief discussion of factorized codes and $p \times p$3-normalized codes following ViT-VQGAN. Diffusion image generation is competitive but not uniformly state of the art, and training remains computationally substantial at 8 A100 GPUs for 2 weeks (Wang et al., 2024).

A broader misconception is to treat every later “Omni-” system as a tokenizer in the same sense. The literature cited here draws sharper distinctions. OmniDrop is described not as an “omni-modal tokenizer” but as a decoder-side, layer-wise token pruning framework that operates downstream of tokenization inside an omni-modal LLM decoder (Park et al., 14 May 2026). OmniVLM similarly does not introduce a module named “OmniTokenizer”; its closest analogue is a projection-stage visual token compression mechanism reducing SigLIP outputs from 729 tokens to 81 (Chen et al., 2024). These distinctions help situate OmniTokenizer precisely: it is a front-end visual tokenizer for image and video generation, not a generic token-pruning framework or a lightweight projector-stage compressor.

In the longer trajectory of unified tokenization research, OmniTokenizer occupies an early and influential position. UniFlow later addresses the understanding-versus-generation trade-off with a continuous unified tokenizer built from a pretrained visual encoder, layer-wise adaptive self-distillation, and a patch-wise pixel flow decoder (Yue et al., 12 Oct 2025), while UniWeTok approaches unified visual tokenization through binary discrete codes, Pre-Post Distillation, and a Generative-Aware Prior (Zhuang et al., 15 Feb 2026). Relative to these later systems, OmniTokenizer is best understood as establishing the viability of a single shared tokenizer for images and videos, and as showing that image-video co-training can be a productive learning problem rather than a compromise (Wang et al., 2024).