CAT: Content-Adaptive Image Tokenization (2501.03120v1)

Abstract: Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages LLMs to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.

• The paper presents a novel dynamic token allocation method that tailors the number of tokens to an image’s content complexity using LLM-based captions.
• It leverages a nested VAE architecture for adaptive compression, significantly boosting both image reconstruction quality and inference throughput by 18.5%.
• CAT outperforms fixed-ratio models, achieving lower FID scores on benchmarks such as ImageNet, CelebA, and ChartQA.

Overview of "CAT: Content-Adaptive Image Tokenization"

The publication introduces Content-Adaptive Tokenizer (CAT), a novel approach for image tokenization that dynamically alters the number of tokens based on the image's content complexity. The primary goal of CAT is to address the inefficiencies found in existing image tokenizers that use a fixed number of tokens, irrespective of an image’s varying content complexity. By using a content-adaptive approach, CAT optimizes both the quality of image reconstruction and computational efficiency.

Key Contributions

1. Dynamic Token Allocation: CAT allocates varying numbers of tokens to images based on their content complexity, which is predicted using captions processed by LLMs. This is in contrast to fixed-ratio tokenizers that do not optimize for the inherent complexity of images.
2. Caption-Based Evaluation System: The authors developed a system leveraging LLMs to assess the complexity of images through captions. This system predicts the appropriate compression ratio by considering aspects critical to human perception, such as the presence of text or human faces.
3. Training Framework: CAT is trained on images with diverse compression ratios, allowing it to effectively reconstruct images using varying token lengths.
4. Robust Performance: The approach demonstrates robust performance across datasets, improving the Fréchet Inception Distance (FID) score over traditional fixed-ratio baselines for ImageNet generation. Notably, CAT also enhances inference throughput by 18.5%.
5. Nested VAE Architecture: The paper introduces a nested Variational Autoencoder (VAE) architecture that facilitates multilevel compression within a single model, supporting adaptive token allocation effectively.

Methodology and Implementation

• Complexity Scoring: The paper employs an LLM to evaluate image complexity based on a text description, which includes image captions and additional context about visual elements significant to human perception.
• Adaptive Compression: The nested VAE architecture enables adaptive compression by routing intermediate features to a middle block in the VAE, which produces latent features of varying spatial dimensions.
• Training and Evaluation: The model was trained on a substantial dataset of licensed images, demonstrating improved reconstruction and generation capabilities over baselines that do not adapt token counts to content complexity.

Results and Implications

CAT outperformed fixed compression ratio models in both reconstruction and generation tasks, especially on images containing perceptually significant details like text or faces. It achieved a significant reduction in rFID on datasets such as CelebA and ChartQA relative to fixed-ratio baselines. This dynamic approach shows potential for improving resource efficiency and reducing computational costs in the processing of complex visual data.

Future Directions

The paper suggests several areas for further exploration:

• Extending CAT to handle discrete tokenizers and combine with quantization techniques.
• Adapting CAT to other modalities beyond images, such as video processing, where temporal elements can introduce additional complexity.
• Investigating more varied downstream tasks and integrating CAT into broader multimodal frameworks to enhance model versatility and performance across diverse applications.

In conclusion, the proposed CAT framework introduces significant advancements in image tokenization by effectively balancing computational efficiency and image quality through content-adaptive processing. Such approaches could be pivotal as the need for efficient and high-quality image processing grows in various complex AI applications.