DESIRE-ME: Domain-Enhanced Supervised Information REtrieval using Mixture-of-Experts (2403.13468v1)

Abstract: Open-domain question answering requires retrieval systems able to cope with the diverse and varied nature of questions, providing accurate answers across a broad spectrum of query types and topics. To deal with such topic heterogeneity through a unique model, we propose DESIRE-ME, a neural information retrieval model that leverages the Mixture-of-Experts framework to combine multiple specialized neural models. We rely on Wikipedia data to train an effective neural gating mechanism that classifies the incoming query and that weighs the predictions of the different domain-specific experts correspondingly. This allows DESIRE-ME to specialize adaptively in multiple domains. Through extensive experiments on publicly available datasets, we show that our proposal can effectively generalize domain-enhanced neural models. DESIRE-ME excels in handling open-domain questions adaptively, boosting by up to 12% in NDCG@10 and 22% in P@1, the underlying state-of-the-art dense retrieval model.

• The paper introduces DESIRE-ME, a modular neural retrieval model that dynamically specializes in multiple domains using a supervised Mixture-of-Experts framework.
• It employs a gating mechanism to classify queries by domain, achieving up to 12% improvement in NDCG@10 and 22% in P@1 over state-of-the-art baselines.
• Extensive experiments on diverse datasets demonstrate DESIRE-ME’s robust generalization in zero-shot scenarios and its practical applicability in open-domain Q&A.

DESIRE-ME: Enhancing Open-Domain Question Answering with Domain-Specific Expertise

Introduction to DESIRE-ME

The field of open-domain question answering (Q&A) presents formidable challenges due to the diverse and broad array of topics questions can span. To address this heterogeneous landscape, we introduce DESIRE-ME, a neural information retrieval model that employs a Mixture-of-Experts (MoE) framework. The core innovation of DESIRE-ME lies in its ability to adaptively specialize in multiple domains through a neural gating mechanism that classifies the domain of a query and weights the contribution of domain-specific experts accordingly. This approach enables DESIRE-ME to excel in handling open-domain questions, significantly improving upon traditional dense retrieval models.

Key Contributions

• Modular Framework: DESIRE-ME is designed as a modular extension to existing dense retrieval systems, leveraging domain specialization to enhance the effectiveness of the retrieval process.
• Supervised Gating Method: A novel supervised gating mechanism facilitates the detection of query topics, allowing for more precise domain contextualization by dynamically weighing the experts' contributions.
• Experimental Validation: Through rigorous experimentation using Wikipedia-based datasets, DESIRE-ME demonstrates substantial improvements in performance metrics, notably achieving up to 12% increase in NDCG@10 and 22% in P@1 over state-of-the-art baselines.
• Generalization Capability: DESIRE-ME's architecture enables it to perform effectively in zero-shot scenarios on datasets with similar characteristics, highlighting its potential for wide applicability in the field of open-domain Q&A.

DESIRE-ME Architecture

DESIRE-ME's architecture is crafted around the core principles of the Mixture-of-Experts framework. The model is structured to include:

• Query and Document Encoders: Retained from the underlying dense retrieval model, facilitating the semantic understanding of queries and documents.
• MoE Module: A specialized component that interjects domain-specific knowledge into the query representation, comprising a gating function, multiple specializers (experts), and a pooling module.
• Gating Function: Utilizes a multi-label domain classifier to predict domain relevancy, employing a sigmoid function to allow non-exclusive domain associations for queries.
• Specializers: Each tailored to optimize the query representation with respect to a specific domain, enhancing the retrieval's precision.
• Pooling Module: Aggregates the outputs of the specializers based on the gating function's weights, culminating in a refined query representation for retrieval.

Experimental Insights

The experimental analysis of DESIRE-ME underscores its efficacy across several public datasets (NaturalQuestion, HotpotQA, FEVER), showcasing consistent improvements in key retrieval metrics. These results confirm the hypothesis that domain-specific specialization significantly benefits the retrieval process in open-domain Q&A. Additionally, the experiments reveal DESIRE-ME's adeptness at generalizing to similar datasets in zero-shot scenarios, an essential characteristic for practical application in diverse information retrieval environments.

Concluding Thoughts

DESIRE-ME marks a significant step forward in enhancing neural information retrieval models for open-domain Q&A through domain specialization. By judiciously combining the expertise of domain-specialized components under a supervised gating mechanism, DESIRE-ME not only sets new benchmarks in retrieval performance but also opens new avenues for future research into domain-enhanced retrieval models. Looking ahead, focusing on optimization strategies for the neural architectures of the specializers and gating mechanism presents a promising direction. Additionally, exploring methods for automated domain labeling and integration of DESIRE-ME with broader datasets would be instrumental in further expanding the model's applicability and performance.