ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Abstract: A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained LLM and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that multimodal training with visual quantized representations largely improves efficiency, yielding up to 20\%-70\% acceleration with different base LLMs and training recipes.

• The paper introduces a two-stage paradigm that aligns visual features with language models, enabling text-aligned discrete representations at any resolution.
• It utilizes progressive quantization with FSQ and RoPE to compress native-resolution images, achieving up to 70% training speed-ups and significant storage savings.
• Empirical results on tasks like VQA, OCR, and chart recognition demonstrate that ViQ matches or surpasses continuous encoders in semantic and reconstruction quality.

Unified Multimodal Discrete Representation: Analysis of ViQ

Motivation and Context

Large Multimodal LLMs (MLLMs) require vision encoders capable of providing visual representations that are both semantically rich and suitable for downstream multimodal tasks. The prevalent continuous encoders, such as CLIP and SigLIP derivatives, introduce representational heterogeneity between visual and language modalities, with high computational demands and limited flexibility regarding input resolution. Discrete visual tokenization, inspired by advances in vector quantization and discrete latent spaces, promises to unify multimodal representations. However, prior discrete encoders exhibit pronounced loss in either semantic fidelity or fine-grained visual detail, constraining their utility for real-world and complex multimodal scenarios.

Architecture and Training Paradigm

ViQ introduces a two-stage paradigm for text-aligned visual quantized representations supporting arbitrary resolutions. The first stage performs text-aligned pre-training using vision-language paired data, aligning the visual encoder's semantic space with pretrained LLMs. Native-resolution adaptation is accomplished by replacing fixed positional embeddings with scalable ones and utilizing dynamic token packing strategies. Self-distillation regularizes the semantic token features to maintain consistency across resolutions.

Figure 1: Stage 1 aligns semantic space using language supervision; Stage 2 progressively compresses visual features into discrete codes.

The second stage employs progressive quantization, first bottlenecking the continuous features (dimension $C$ to an intermediate $D$ then $d$; for instance, SigLIP2-g $1536\to128\to6$), then applying $L_\infty$-norm regularization to facilitate proximity to quantization anchors. Finite Scalar Quantization (FSQ) discretizes features, accompanied by multi-head attention which expands patch-level tokens, and 2D Rotary Position Embedding (RoPE) ensures spatial resolution invariance. Low-level supervision by pretrained autoencoders (e.g., Qwen-Image) is incorporated through latent regression losses on the VAE latent, not pixels, enhancing reconstruction stability.

Benefits in Efficiency and Compression

ViQ achieves multimodal alignment and discrete visual tokens with strong semantic and reconstruction properties, while substantially improving training efficiency.

Figure 2: Training efficiency comparison: ViQ outperforms continuous encoders (e.g., SigLIP2-g) in both 4k and 16k SFT settings.

The quantized pipeline enables offline image encoding, eliminating costly visual feature extraction during VLM training. Speed-ups range from 20% to 70%, with acceleration particularly pronounced for smaller LLMs and longer sequence lengths. Storage efficiency is also stark: ViQ compresses native-resolution images into small integer token sequences, achieving $\sim1/96$ the raw image size with little perceptual loss, outperforming aggressive JPEG compression at equivalent bitrates.

Figure 3: High-compression-ratio visual tokenization—ViQ reconstructs images at their original resolution from discrete codes without significant artefacts.

Quantitative Evaluation: Semantic and Reconstruction Quality

ViQ is evaluated across diverse multimodal benchmarks: VQA, document and chart recognition, OCR, and infographic understanding. On aggregated scores for Qwen2.5-1.5B and Qwen2.5-7B LLMs, ViQ achieves 57.2 and 63.9 respectively, outperforming previous state-of-the-art quantized models and matching leading continuous encoders despite aggressive quantization. ViQ is especially strong on detail-centric tasks (DocVQA, TextVQA, ChartQA), evidencing the preservation of low-level information after discretization. The residual gap on certain benchmarks (e.g., OCRBench) is attributed to the limitation of discrete encoding regarding high-frequency pixel information.

For reconstruction, ViQ yields an SSIM of 0.66 and rFID of 0.62—second only to UniTok in discrete tokenizers, and competitive with continuous tokenizers such as Open-MAGVIT2 and Qwen-Image, while maintaining at least parity in PSNR with QLIP-B and MUSE-VL. The ablation studies confirm that bottlenecked latent compression, $L_\infty$ normalization, large FSQ codebooks ($\sim64k$), and RoPE positional encoding are critical for both semantic and reconstruction gains.

Figure 4: ViQ quantized codes preserve both high-level semantics (for multimodal tasks) and low-level features (for image reconstruction), outperforming comparable continuous encoders.

Figure 5: Reconstruction quality visualizations by ViQ at any resolution, demonstrating retention of detail and native aspect ratio fidelity.

Implications and Future Directions

ViQ represents a rigorous step toward unified, text-aligned discrete multimodal representation that enables memory-efficient training, fast inference, and high-resolution visual processing. The discrete codes facilitate easy integration with autoregressive LLMs and support scalable storage and retrieval scenarios, paving the way for flexible multimodal interfaces, edge deployments, and generative applications. Residual limitations in detail preservation point toward directions including multi-scale quantization, residual encoding, and specialized domain adaptation.

Theoretical implications include potential for improved interpretability in token-level visual representations, streamlined cross-modal token routing, and unified generative/understanding pipelines leveraging discrete latent spaces. Practically, ViQ unlocks efficient large-scale multimodal SFT, rapid online interaction, and viable deployment on resource-limited hardware.

Conclusion

ViQ establishes a competitive framework for text-aligned, quantized visual representations at arbitrary resolutions, balancing high-level semantics with low-level reconstruction using a progressive two-stage training regime. It surpasses prior quantized models in both multimodal task performance and efficiency, and matches the leading continuous encoders. The work demonstrates the viability of unifying language and vision via discrete tokens and delineates practical, scalable approaches for future multimodal models (2606.27313).