ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2004.12832v2)

Abstract: Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep LLMs (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity. By delaying and yet retaining this fine-granular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations offline, considerably speeding up query processing. Beyond reducing the cost of re-ranking the documents retrieved by a traditional model, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from a large document collection. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query.

Contextualized Late Interaction over BERT for Efficient and Effective Passage Search

Introduction

The surge in applying deep LLMs (LMs) to document ranking tasks has notably enhanced the Natural Language Understanding (NLU) capabilities within Information Retrieval (IR) systems. However, the computational demands of these models, particularly when using BERT for evaluating query-document pairs, introduce significant challenges in terms of latency and resource consumption. To address these issues, the paper introduces ColBERT, a novel ranking model leveraging late interaction over BERT to offer a balance between efficiency and retrieval effectiveness.

ColBERT Model Architecture

ColBERT innovates by encoding queries and documents independently into sets of contextual embeddings using BERT, followed by a late interaction mechanism. This approach significantly differs from traditional models where embeddings for the query and the document are either combined early or require exhaustive pairwise interaction, leading to increased computational costs.

• Query and Document Encoders: ColBERT employs separate encoders for queries and documents, appending special tokens to differentiate between the two. Post-encoding, embeddings corresponding to punctuation in documents are filtered out, reducing unnecessary computational overhead.
• Late Interaction: The core of ColBERT's efficiency lies in its late interaction mechanism where the relevance of a document with respect to a query is computed using a summation of maximum similarity (MaxSim) scores across the query's and document's embeddings. This late interaction allows for the pre-computation of document embeddings, significantly speeding up the retrieval process.
• Efficient Indexing and Retrieval: ColBERT extends its efficiency to indexing and retrieval, allowing documents' representations to be pre-computed and stored. This enables leveraging vector-similarity indices for fast top-k retrieval from large document collections.

Experimental Evaluation

ColBERT is extensively evaluated on two recent passage search datasets, demonstrating that it maintains competitive effectiveness with existing BERT-based models while significantly reducing computational costs.

• Execution Speed and Resource Consumption: ColBERT achieves a remarkable improvement in execution speed, performing two orders of magnitude faster and requiring four orders of magnitude fewer FLOPs per query compared to existing BERT-based models.
• Effectiveness: Despite its efficiency, ColBERT does not compromise on the effectiveness of retrieval, outperforming all non-BERT baselines and providing competitive results against BERT-based models.
• Practical Indexing and Retrieval Overheads: The model's design facilitates practical indexing processes and manageable space footprints, further underscoring its applicability to real-world IR systems.

Implications and Future Directions

The introduction of ColBERT provides a promising avenue for integrating deep LLMs into IR systems without incurring prohibitive computational costs. Its efficient and effective design presents a significant step forward in addressing the latency challenges associated with deep LMs like BERT in document ranking tasks. Future work may explore the extension of the ColBERT architecture to other forms of interaction and the application of the model to additional IR tasks beyond passage search.

In summary, ColBERT elegantly reconciles the trade-off between efficiency and effectiveness prevalent in current deep LM-based IR systems, offering a scalable solution for incorporating advanced NLU capabilities into real-time search engines.