Factorized Visual Tokenization and Generation (2411.16681v2)

Abstract: Visual tokenizers are fundamental to image generation. They convert visual data into discrete tokens, enabling transformer-based models to excel at image generation. Despite their success, VQ-based tokenizers like VQGAN face significant limitations due to constrained vocabulary sizes. Simply expanding the codebook often leads to training instability and diminishing performance gains, making scalability a critical challenge. In this work, we introduce Factorized Quantization (FQ), a novel approach that revitalizes VQ-based tokenizers by decomposing a large codebook into multiple independent sub-codebooks. This factorization reduces the lookup complexity of large codebooks, enabling more efficient and scalable visual tokenization. To ensure each sub-codebook captures distinct and complementary information, we propose a disentanglement regularization that explicitly reduces redundancy, promoting diversity across the sub-codebooks. Furthermore, we integrate representation learning into the training process, leveraging pretrained vision models like CLIP and DINO to infuse semantic richness into the learned representations. This design ensures our tokenizer captures diverse semantic levels, leading to more expressive and disentangled representations. Experiments show that the proposed FQGAN model substantially improves the reconstruction quality of visual tokenizers, achieving state-of-the-art performance. We further demonstrate that this tokenizer can be effectively adapted into auto-regressive image generation. https://showlab.github.io/FQGAN

Overview of Factorized Visual Tokenization and Generation

The paper "Factorized Visual Tokenization and Generation" presents an innovative approach to address the limitations of traditional Vector Quantization (VQ)-based visual tokenizers used in image generation systems. These limitations primarily stem from scalability issues encountered when codebook sizes are increased, leading to training instability and diminishing performance returns. The authors introduce Factorized Quantization (FQ), which decomposes large codebooks into multiple independent sub-codebooks, offering a scalable and efficient method for visual tokenization.

Key Contributions

The paper introduces several key components to enhance VQ-based tokenization:

1. Factorized Quantization Design: The proposed method decomposes a traditionally large codebook into several smaller, independent sub-codebooks. This decomposition results in reduced lookup complexity and promotes diverse representations.
2. Disentanglement Regularization: To ensure that each sub-codebook captures distinct and complementary information, the paper proposes a disentanglement regularization technique. This mechanism minimizes redundancy and promotes diversity across sub-codebooks, encouraging each to focus on different visual aspects such as spatial structure, texture, and color.
3. Integration of Representation Learning: Leveraging pretrained vision models like CLIP and DINO, the paper incorporates representation learning into the training process. This integration infuses semantic richness into the representations learned by each sub-codebook, leading to more expressive and semantically meaningful visual tokenization.
4. Improvement in State-of-the-Art Performance: The proposed FQGAN model demonstrates significant improvements in reconstruction quality over existing visual tokenizers. It achieves state-of-the-art results in terms of reconstruction FID on the ImageNet dataset, surpassing both traditional VQ and Lookup-Free Quantization (LFQ) methods.
5. Adaptation to Auto-regressive Image Generation: The paper further adapts the proposed tokenizer for use in auto-regressive image generation tasks, enhancing image generation quality by producing richer and more expressive token representations.

Implications and Future Directions

This research introduces several theoretical and practical implications for the field of visual tokenization and generation:

• Scalability and Efficiency: The factorized approach provides a pathway to efficiently managing large codebooks, addressing a key scalability issue in VQ-based tokenization methods. This has potential implications for developing more computationally efficient and scalable visual generation systems.
• Enhanced Semantic Representation: By integrating representation learning, the factorized tokenization method can capture semantically rich features, suggesting a direction towards multimodal models that excel in both visual understanding and generation tasks.
• Transferability of Improvements: The substantial improvements in reconstruction quality achieved with the proposed method suggest that enhancing tokenization can also lead to better performance in downstream tasks such as auto-regressive generation.

Future research could explore extending the factorization approach to more sub-codebooks, probing the semantic capabilities of the tokenizers in multimodal understanding tasks, and further bridging the gap between VQ and LFQ approaches in terms of downstream generation performance.

In conclusion, the paper provides substantial contributions to the field, offering a novel and effective solution to the traditional limitations of VQ-based visual tokenization and setting the stage for future advancements in efficient, scalable image generation models.