AToken: A Unified Tokenizer for Vision (2509.14476v2)

Abstract: We present AToken, the first unified visual tokenizer that achieves both high-fidelity reconstruction and semantic understanding across images, videos, and 3D assets. Unlike existing tokenizers that specialize in either reconstruction or understanding for single modalities, AToken encodes these diverse visual inputs into a shared 4D latent space, unifying both tasks and modalities in a single framework. Specifically, we introduce a pure transformer architecture with 4D rotary position embeddings to process visual inputs of arbitrary resolutions and temporal durations. To ensure stable training, we introduce an adversarial-free training objective that combines perceptual and Gram matrix losses, achieving state-of-the-art reconstruction quality. By employing a progressive training curriculum, AToken gradually expands from single images, videos, and 3D, and supports both continuous and discrete latent tokens. AToken achieves 0.21 rFID with 82.2% ImageNet accuracy for images, 3.01 rFVD with 40.2% MSRVTT retrieval for videos, and 28.28 PSNR with 90.9% classification accuracy for 3D.. In downstream applications, AToken enables both visual generation tasks (e.g., image generation with continuous and discrete tokens, text-to-video generation, image-to-3D synthesis) and understanding tasks (e.g., multimodal LLMs), achieving competitive performance across all benchmarks. These results shed light on the next-generation multimodal AI systems built upon unified visual tokenization.

• The paper introduces a unified tokenizer that processes images, videos, and 3D assets through a shared 4D latent space and transformer architecture.
• It employs an adversarial-free training strategy using perceptual and Gram matrix losses to ensure stable, high-fidelity reconstructions.
• A progressive multimodal curriculum and scalability analysis reveal that cross-modal training can enhance single-modality performance while supporting efficient generation and understanding.

AToken: A Unified Tokenizer for Vision

Motivation and Problem Statement

The fragmentation of visual tokenization across modalities and tasks has impeded the development of general-purpose vision models analogous to the success of unified tokenization in LLMing. Existing visual tokenizers are typically specialized for either high-fidelity reconstruction or semantic understanding, and are limited to single modalities (images, videos, or 3D assets). This specialization restricts transfer learning, scalability, and the ability to build truly multimodal AI systems. The AToken framework addresses these limitations by introducing a unified tokenizer capable of both reconstruction and understanding across images, videos, and 3D assets, leveraging a shared 4D latent space and a pure transformer architecture.

Figure 1: Illustration of AToken on different visual modalities, showing shared 4D latent space for high-fidelity reconstructions and strong semantic understanding.

Unified 4D Latent Representation

AToken's central innovation is the sparse 4D latent representation, which unifies all visual modalities. Images are encoded as 2D slices, videos as temporal stacks, and 3D assets as surface voxels, all within a single 4D coordinate system $(t, x, y, z)$. This representation enables a single transformer encoder to process arbitrary resolutions and durations without architectural changes. The use of 4D Rotary Position Embeddings (RoPE) provides relative position awareness across all axes, supporting efficient scaling and joint modeling.

Figure 2: Overview of AToken’s unified space-time patchification and encoding into sparse 4D latents, supporting both reconstruction and understanding.

For 3D assets, the pipeline extends Trellis-SLAT by rendering multi-view images and aggregating patch features into voxel space, enabling seamless integration with image and video modalities.

Figure 3: 3D tokenization pipeline, extending Trellis-SLAT for multimodal unification via direct tokenization of RGB patches and viewpoint-based aggregation.

Transformer Architecture and Training Stability

AToken employs a pure transformer architecture for both encoder and decoder, processing sparse feature-position pairs. The encoder is initialized from SigLIP2 and extended to 4D via space-time patch embedding and 4D RoPE. The decoder is trained from scratch for modality-specific reconstruction, including Gaussian splatting for 3D assets.

A key contribution is the adversarial-free training objective, which combines perceptual and Gram matrix losses. This approach circumvents the instability of GAN-based training in transformer tokenizers, as demonstrated by the rapid mode collapse and degraded rFID observed with adversarial objectives.

Figure 4: GAN training instability in transformer-based tokenizers, motivating adversarial-free optimization.

Gram matrix loss directly optimizes feature covariance, which accounts for $\approx86.6\%$ of rFID error, yielding stable and superior reconstruction quality.

Progressive Multimodal Curriculum

AToken is trained via a four-stage curriculum:

1. Image Foundation: Pretrained SigLIP2 encoder, image reconstruction added.
2. Video Dynamics: Temporal modeling, expanded latent dimensions, KV-caching for efficient video encoding.
3. 3D Geometry: Active voxels, Gaussian splatting, joint optimization across modalities.
4. Discrete Tokenization: FSQ quantization for compatibility with discrete generative models.

This curriculum enables stable learning and reveals that multimodal training can enhance single-modality performance, contradicting the common assumption of catastrophic forgetting.

Figure 5: Progressive training curriculum, showing staged addition of modalities and capabilities.

KV-caching in video encoding eliminates redundant computation and maintains temporal coherence.

Figure 6: Video encoding and decoding process with KV-caching for efficient temporal modeling.

Empirical Results and Scaling Analysis

AToken achieves competitive or state-of-the-art results across all modalities:

• Images: 0.21 rFID, 82.2% ImageNet accuracy.
• Videos: 3.01 rFVD, 32.6% MSRVTT retrieval.
• 3D: 28.19 PSNR, 90.9% classification accuracy.

Scaling analysis demonstrates that sufficient model capacity is critical for successful multimodal tokenization. The So400m model maintains or improves performance across all stages, while smaller models degrade when extending beyond single-modality training.

Figure 7: Architectural scaling comparison, highlighting the necessity of large model capacity for multimodal tokenization.

Representation Structure and Compression Trade-offs

T-SNE visualizations reveal that dense features exhibit clear semantic clustering, but dimensional reduction to 48-dim latents leads to more mixed class distributions. Despite this, semantic performance remains strong, suggesting that explicit clustering in low-dimensional spaces is not strictly necessary for high performance in large models.

Figure 8: Learned representations across training stages, showing semantic clustering and the effects of dimensionality reduction.

Qualitative Reconstruction and Generation

AToken demonstrates superior reconstruction quality at higher compression ratios, excelling in preservation of high-frequency textures, fine details, and text elements. Video and 3D reconstructions maintain temporal and color consistency, respectively.

Figure 9: Qualitative comparison of image reconstruction across tokenization methods, highlighting AToken’s fidelity at high compression.

Figure 10: ImageNet generation samples using continuous tokens, demonstrating high-fidelity synthesis.

Downstream Applications

AToken serves as a universal visual foundation for multimodal LLMs, visual generation (image, video, 3D), and understanding tasks. Integration into SlowFast-LLaVA-1.5 yields competitive performance on image and video understanding benchmarks, outperforming specialized vision encoders at multiple model scales. In generative tasks, AToken supports both continuous and discrete token-based synthesis, matching or surpassing specialized tokenizers in gFID and perceptual metrics.

Implications and Future Directions

AToken’s unified approach to visual tokenization enables scalable, efficient, and versatile multimodal AI systems. The adversarial-free training strategy and progressive curriculum provide a blueprint for stable optimization in large transformer-based models. The empirical finding that multimodal training can enhance single-modality performance challenges prevailing assumptions and suggests new directions for cross-modal transfer learning.

The framework opens avenues for further research in:

• Scaling unified tokenizers to even larger model sizes and more modalities (e.g., audio, action).
• Optimizing generative models for high-dimensional latent spaces.
• Investigating semantic preservation under aggressive compression and quantization.
• Building comprehensive omnimodels for end-to-end multimodal understanding and generation.

Conclusion

AToken establishes a unified visual tokenization paradigm, achieving high-fidelity reconstruction and semantic understanding across images, videos, and 3D assets within a single transformer framework. The combination of sparse 4D representation, adversarial-free training, and progressive multimodal curriculum enables competitive performance and efficient scaling. These results demonstrate the feasibility and advantages of unified tokenization for vision, laying the foundation for next-generation multimodal AI systems.