Subobject-level Image Tokenization (2402.14327v2)

Abstract: Transformer-based vision models typically tokenize images into fixed-size square patches as input units, which lacks the adaptability to image content and overlooks the inherent pixel grouping structure. Inspired by the subword tokenization widely adopted in LLMs, we propose an image tokenizer at a subobject level, where the subobjects are represented by semantically meaningful image segments obtained by segmentation models (e.g., segment anything models). To implement a learning system based on subobject tokenization, we first introduced a Direct Segment Anything Model (DirectSAM) that efficiently produces comprehensive segmentation of subobjects, then embed subobjects into compact latent vectors and fed them into a LLM for vision language learning. Empirical results demonstrated that our subobject-level tokenization significantly facilitates efficient learning of translating images into object and attribute descriptions compared to the traditional patch-level tokenization. Codes and models are open-sourced at https://github.com/ChenDelong1999/subobjects.

• The paper introduces a novel subobject-level tokenization method that segments images into semantically grouped parts, similar to subword tokenization in NLP.
• The methodology employs a Sequence-to-Sequence AutoEncoder (SeqAE) to compress irregular image segments into robust embedding vectors.
• Integrating subobject embeddings with a large vision-language model significantly boosts efficiency and accuracy on benchmarks like SA-1B and CLEVR.

Subobject-level Image Tokenization: Enhancing Vision-LLMs

This paper addresses a critical concern in the current paradigm of vision-LLMs that utilize transformer architectures. Traditional methodologies tokenize images into fixed-size square patches without adapting to image content, consequently ignoring the inherent pixel grouping structure. In response, the authors propose a novel approach to image tokenization at a subobject level, reminiscent of subword tokenization in NLP. The method aims to leverage semantically meaningful image segments, or “subobjects,” achieved through segmentation models, for improved efficiency and accuracy in vision-language tasks.

Key Innovations

• Subobject-Level Tokenization: The central proposition is to tokenize images into subobjects, akin to subword tokenization in text, thus bridging the gap between pixel-level and object-level representations. This approach is informed by advancements in image segmentation, particularly models like the Segment Anything Model (SAM). Subobject tokenization addresses inefficiencies in the prevalent patch-based methods, which are analogous to ineffective character-level tokenizations in NLP.
• Sequence-to-Sequence AutoEncoder (SeqAE): To facilitate the transformation of subobject segments into compact representations, the authors introduce SeqAE. This model compresses segments of varying shapes into embedding vectors, maintaining a rich representation of visual data without unnecessary downsampling. The SeqAE framework enables the handling of irregular segment sizes more efficiently than conventional techniques.
• Large Vision LLM (LVLM) Integration: The paper describes an LVLM architecture that incorporates these subobject embeddings, integrating them with a LLM. The subobject tokens are treated similarly to textual subword tokens, with additional positional embeddings to account for their two-dimensional nature.

Empirical Results

The authors substantiate their claims through empirical evaluations on datasets such as SA-1B and CLEVR. The SeqAE model is pre-trained on the SA-1B dataset for robust subobject embeddings, while the LVLM is assessed on CLEVR for image captioning tasks. The results are compelling: subobject-level tokenization significantly expedites the learning process and enhances the model's accuracies in identifying object attributes and counts.

Implications and Future Prospects

From a practical standpoint, subobject-level tokenization presents an opportunity to enhance the efficiency of vision-LLMs significantly. It aligns with the increasing demand for systems that can effectively process visual information with semantic understanding, a crucial aspect of intelligent systems. Theoretically, it opens new research avenues in tokenization strategies that consider the semantic granularity of inputs, potentially applicable beyond vision tasks. Future work may explore expanding the subobject tokenization approach to varied domains and integrating it with increasingly sophisticated LLMs to achieve higher levels of contextual understanding and generation in multimodal AI systems.

The paper offers a scholarly contribution to the vision-LLMing domain by challenging the entrenched methodologies and presenting a robust alternative that aligns well with contemporary LLM practices. This work adds a significant layer of interpretability and efficiency, potentially setting a new standard for how image data is structured and processed in advanced AI systems.