LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2103.08784v2)

Abstract: Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computation cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by pre-training on three novel learning objectives, extracting feature indexes offline, and employing instant dot-product matching with further re-ranking, which significantly speeds up retrieval process. In fact, LightningDOT achieves new state of the art across multiple ITR benchmarks such as Flickr30k, COCO and Multi30K, outperforming existing pre-trained models that consume 1000x magnitude of computational hours. Code and pre-training checkpoints are available at https://github.com/intersun/LightningDOT.

Overview of LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval

The paper "LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval" addresses the inefficiency challenges within multidimensional visual-and-language (V+L) models and proposes a novel approach for the Image-Text Retrieval (ITR) task. The existing V+L models, while powerful in terms of capabilities, suffer from significant computational costs due to the use of cross-modal attention mechanisms in Transformer architectures. These limitations impede their application in real-time environments where quick responses are beneficial.

LightningDOT introduces a methodology to dramatically enhance the inference speed of image-text retrieval by circumventing the computationally expensive cross-modal attention. Its primary innovation is the implementation of dot-product matching between pre-trained feature embeddings of images and text, supplemented by a re-ranking process to refine results. This strategy results in acceleration factors of up to thousands, as compared to traditional models, while maintaining or even improving the retrieval accuracy.

Methodological Advancements

The paper presents several technical innovations:

1. Efficient Cross-Modal Embedding: LightningDOT avoids the intensive cross-modal attention mechanisms by utilizing separate Transformer encoders for image and text, specifically pre-trained to extract high-quality embeddings. The pre-training involves three custom-designed tasks—Visual-embedding fused Masked LLMing (VMLM), Semantic-embedding fused Masked Region Modeling (SMRM), and a Cross-modal Retrieval Objective (CMR). This approach fosters efficient and effective learning of cross-modal representations without the need for cross-attention layers.
2. Offline Feature Extraction: The model supports offline computation of image embeddings, thereby allowing these representations to be stored as indexes. When a retrieval request is processed, only the text embedding is computed online, ensuring rapid assessments and comparisons against the pre-computed image indexes.
3. Re-ranking Mechanism: Despite bypassing cross-attention during initial retrievals, LightningDOT incorporates a re-ranking mechanism utilizing a more computationally demanding model for top-retrieved candidates. This stage ensures that the final retrieval fidelity is competitive with state-of-the-art approaches.

Empirical Evaluation

The comprehensive experimentation conducted on the ITR benchmarks, including Flickr30K and COCO datasets, elucidates LightningDOT's performance. The model not only achieves state-of-the-art accuracy but also provides a striking acceleration in retrieval operations—surpassing models like UNITER by up to 1900 times in speed. Additionally, in scenarios involving larger candidate pools, LightningDOT is reportedly 23,000 times faster, highlighting its practical applicability in real-world tasks demanding efficiency and scalability.

Implications and Future Directions

LightningDOT suggests a significant shift towards making V+L models more practical for real-time applications by bolstering retrieval efficiency without sacrificing accuracy. The proposed method effectively balances the trade-off between computational complexity and model performance. Moreover, this work opens avenues for further optimization and combination with other techniques, such as compression methods or alternative learning paradigms, to explore additional gains in efficiency and real-world applicability.

Looking to the future, the deployment of LightningDOT into multimedia, search engines, and other interactive AI systems can substantially advance the capability of AI to process and retrieve information across modalities in real time. Furthermore, the integration of more nuanced re-ranking techniques or increased incorporation of unlabeled data could further enhance the retrieval process, setting a new benchmark for efficiency and accuracy in the field of V+L multimodal learning.