TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation (2505.05422v1)

Abstract: Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP.

TokLIP: Advancement in Multimodal Comprehension and Generation

The paper introduces TokLIP, a visual tokenizer developed to address limitations in multimodal comprehension and generation models by enhancing semantic quality in visual tokenization. Existing token-based models, such as Chameleon and Emu3, have provided scalable frameworks but are hindered by significant computational demands and insufficient semantic comprehension. TokLIP provides an innovative approach by semanticizing vector-quantized (VQ) tokens, integrating high-level CLIP semantics, and maintaining efficient end-to-end multimodal autoregressive training.

TokLIP combines a discrete VQGAN tokenizer with a Vision Transformer (ViT)-based encoder using causal attention. This combination enables the capture of high-level continuous semantics from low-level discrete VQ tokens, contrasting with previous methods like VILA-U that attempt to discretize CLIP features at a high level. This novel integration allows VQ tokenizers to be directly used without needing tailored quantization operations, effectively disentangling objectives for comprehension and generation.

Empirical Results:

The empirical analysis demonstrates TokLIP's superior data efficiency and semantic comprehension. Remarkably, TokLIP requires less than 20% of the pretraining data used by VILA-U while outperforming existing tokenizers in zero-shot ImageNet classification. This efficiency is highlighted by TokLIP's ability to achieve comparable performance using only 3% of the training data required by SynerGen-VL, in alignment with LLMs for comprehension tasks. TokLIP also shows enhancements in image generation capabilities, confirming that high-level semantic features complement low-level VQ token embeddings.

Discussion on TokLIP Framework:

The core innovation in TokLIP lies in its semanticization strategy. By transforming low-level VQ tokens into continuous high-level features, TokLIP sidesteps the conflicts posed by joint reconstruction and semantic supervision objectives. This approach proves beneficial over existing discrete-to-continuous methods that struggle with quantization information loss. TokLIP's framework is scalable, allowing incorporation of ongoing advancements in tokenizers and semantic vision encoders.

Moreover, TokLIP's design is highly extensible. It can be improved with superior VQGANs or semantic encoders and can be further integrated with diffusion techniques to enhance visual generation.

Conclusion and Implications:

In conclusion, TokLIP presents a promising direction in multimodal model development, effectively harmonizing comprehension and generation tasks. By leveraging sequential multimodal token prediction, TokLIP offers potential advancements in multimodal applications, providing efficient learning and integration of comprehensive semantic understanding. The findings suggest that TokLIP could play a critical role in future developments of AI, enabling more robust, efficient, and semantically aware multimodal models.

This paper contributes significantly to the progression of multimodal AI through a strategic integration of semantics in token-based models, anticipating continued developments and applications in both academic and practical domains.