MaskInversion: Localized Embeddings via Optimization of Explainability Maps

Abstract: Vision-language foundation models such as CLIP have achieved tremendous results in global vision-language alignment, but still show some limitations in creating representations for specific image regions. % To address this problem, we propose MaskInversion, a method that leverages the feature representations of pre-trained foundation models, such as CLIP, to generate a context-aware embedding for a query image region specified by a mask at test time. MaskInversion starts with initializing an embedding token and compares its explainability map, derived from the foundation model, to the query mask. The embedding token is then subsequently refined to approximate the query region by minimizing the discrepancy between its explainability map and the query mask. During this process, only the embedding vector is updated, while the underlying foundation model is kept frozen allowing to use MaskInversion with any pre-trained model. As deriving the explainability map involves computing its gradient, which can be expensive, we propose a gradient decomposition strategy that simplifies this computation. The learned region representation can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation. We evaluate the proposed method on all those tasks on several datasets such as PascalVOC, MSCOCO, RefCOCO, and OpenImagesV7 and show its capabilities compared to other SOTA approaches.

• The paper demonstrates MaskInversion’s ability to iteratively refine localized embeddings by matching explainability maps with query masks.
• It employs gradient decomposition to reduce computational overhead while adapting pre-trained vision-language models without retraining.
• Evaluations reveal significant performance gains in referring and class retrieval tasks, achieving 56.1% and 93.5% Acc@1 on benchmark datasets.

MaskInversion: Localized Embeddings via Optimization of Explainability Maps

The paper "MaskInversion: Localized Embeddings via Optimization of Explainability Maps" by Bousselham et al. presents a novel methodology for generating localized embeddings for specific regions within an image using pre-trained vision-LLMs, such as CLIP. This paper examines the challenges posed by existing vision-LLMs in creating precise embeddings for localized image regions and introduces MaskInversion as a solution that leverages explainability maps for optimizing these embeddings.

Methodology

Initialization

MaskInversion initiates by employing the output of a vision-LLM, initialized from the global [CLS] token, to create an initial localized embedding for a query region specified by a mask. The contextual relevance of this token is then refined iteratively.

Optimization Process

The refinement process involves comparing the explainability map, derived from the model, with the query mask. By minimizing the discrepancy between the explainability map and the mask through iterative gradient descent, the localized embedding is optimized. This process ensures that only the embedding vector is updated while the foundation model remains unchanged, allowing MaskInversion to be used with any pre-trained model.

Gradient Decomposition

To address the computational overhead of generating explainability maps, the authors propose a gradient decomposition strategy. This reduces the need for multiple gradient evaluations, thereby enhancing computational efficiency, particularly when multiple masks are involved.

Contributions

1. Localized Embedding Token: MaskInversion learns a localized embedding token at test time that encapsulates the region characteristics within the query mask. This token can replace any application-based on the same backbone model.
2. Computational Efficiency: The methodology incorporates gradient decomposition to expedite the computation for multiple query masks within the same image.
3. Versatile Applications: The resultant embeddings can be applied across various region-based downstream tasks, demonstrating improved performance over existing methods.

Experiments and Results

Referring Expression Retrieval

The paper evaluates MaskInversion on referring expression retrieval tasks using datasets such as PhraseCut, RefCOCO, and RefCOCO+. The results reveal that MaskInversion surpasses other state-of-the-art (SOTA) methods, achieving high accuracy and demonstrating effective localization of embeddings. For instance, on RefCOCO, MaskInversion achieves an Acc@1 of 56.1% using ViT-B/16, significantly outperforming other methods such as AlphaCLIP and FGVP.

Class Retrieval

In the zero-shot class retrieval setting, MaskInversion also emerges as a strong performer. Utilizing datasets like PascalVOC, PascalContext, and MSCOCO, the method consistently outperforms existing techniques. Notably, for PascalVOC, MaskInversion achieves an Acc@1 of 93.5% with ViT-H/14, highlighting its capability in handling fine-grained class distinctions without additional fine-tuning.

Localized Captioning

The ability of MaskInversion to focus on specific image regions is further demonstrated through localized captioning tasks. By using CLIPCap, MaskInversion facilitates the generation of accurate captions tailored to the masked regions, outperforming the baseline CLIP by a significant margin.

Implications and Future Work

The practical and theoretical implications of MaskInversion are profound. The method opens avenues for improved region-based understanding and generation in vision-LLMs, eliminating the need for extensive model retraining or fine-tuning. The approach can be further extended to various applications, including fine-grained image generation and localized decision making.

Looking ahead, future developments could explore:

• Integration with other explainability methods: Further enhancement of the embedding optimization process using different or more advanced explainability techniques.
• Scalability and real-time applications: Further refinement to enable real-time applications and scalability for larger, more complex datasets.
• Cross-domain adaptability: Testing and adapting MaskInversion across different domains and types of visual content, including medical imaging and remote sensing.

Conclusion

The paper by Bousselham et al. presents a robust and innovative approach to localized embedding generation using pre-trained vision-LLMs. Through MaskInversion, it is possible to generate highly accurate and context-aware embeddings for specific image regions without extensive model modification. The method offers significant advancements in computational efficiency and practical applicability, setting a new benchmark in the field of vision-language representation learning.