LAReQA: Language-agnostic answer retrieval from a multilingual pool

Abstract: We present LAReQA, a challenging new benchmark for language-agnostic answer retrieval from a multilingual candidate pool. Unlike previous cross-lingual tasks, LAReQA tests for "strong" cross-lingual alignment, requiring semantically related cross-language pairs to be closer in representation space than unrelated same-language pairs. Building on multilingual BERT (mBERT), we study different strategies for achieving strong alignment. We find that augmenting training data via machine translation is effective, and improves significantly over using mBERT out-of-the-box. Interestingly, the embedding baseline that performs the best on LAReQA falls short of competing baselines on zero-shot variants of our task that only target "weak" alignment. This finding underscores our claim that languageagnostic retrieval is a substantively new kind of cross-lingual evaluation.

• The paper presents a novel benchmark that evaluates strong cross-lingual alignment in retrieving answers from a multilingual pool.
• It compares mBERT-based dual encoder models, highlighting the promising performance of the X-Y model in reducing language bias.
• Results indicate that translation-based methods still outperform pure embeddings, emphasizing the need for improved multilingual training strategies.

Overview of LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool

The paper "LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool" introduces a novel benchmark designed to assess language-agnostic retrieval capabilities within multilingual contexts. This research delineates substantial differences from existing cross-lingual evaluations by emphasizing the necessity for "strong" cross-lingual alignment, setting a new frontier in evaluating multilingual embeddings.

Introduction and Motivation

The advent of self-supervised multilingual models like multilingual BERT (mBERT) and XLM-R has shown promise in cross-lingual transfer without explicit alignment objectives. These models suggest the possibility of language-independent representations. However, the potential for genuinely strong language-agnostic embeddings remains underexplored. The paper addresses this gap via a benchmark named LAReQA, which challenges models to retrieve answers from a diverse linguistic candidate pool, demanding a higher level of semantic alignment across languages.

Task Description and Novel Contributions

LAReQA is distinct from tasks like XNLI and MLQA in its structure, facilitating the retrieval of answers across language boundaries. This task requires models to prioritize semantically relevant cross-lingual pairs over unrelated monolingual pairs. The paper defines two alignment types:

• Weak Alignment: Ensures nearest neighbors in a different language carry semantic relevance.
• Strong Alignment: Ensures relevant items, irrespective of language, are closer than irrelevant ones in the same language. LAReQA is the first benchmark targeting this alignment level.

The dataset for evaluation is derived from XQuAD and MLQA by transforming extractive QA setups into retrieval tasks. Mean Average Precision (mAP) is employed as the evaluation metric to accommodate multiple relevant targets per query.

Baseline Models and Methodologies

The study evaluated several mBERT-based dual encoder models with different training regimes to understand their alignment characteristics:

• En-En: Trained solely on English QA pairs.
• X-X / X-X-mono: Trained on translated QA pairs with varying intra-batch language homogeneity.
• X-Y: Utilized mixed-language QA examples aiming to minimize language bias.

A Translate-Test baseline leverages machine translation, testing if direct translation yields better retrieval by converting the test data into English.

Results and Analysis

Despite utilizing pretrained multilingual models, achieving strong cross-lingual alignment remains challenging. The Translate-Test baseline outperformed pure embedding models, indicating that contemporary methods might still require translation as a crutch. The X-Y model showed the most promise among purely embedding strategies, effectively diminishing language bias while maintaining competitive retrieval performance.

Further analysis revealed inherent language bias, with some models displaying a preference for same-language answers. Notably, models with strong cross-lingual alignment dependencies, like X-Y, exhibited this bias minimally, making a step towards better cross-lingual semantic matching.

Implications and Future Directions

The implications of this work are profound for the development of truly language-agnostic models. By pushing for benchmarks that test beyond zero-shot transfer, the research highlights necessary advancements in multilingual model training. Future studies could explore harmonizing alignment alongside improving within-language performance to address discovered trade-offs. There is scope for developing methodologies that reduce reliance on translation, potentially paving the way for seamless multilingual interactions in NLP applications.

This research underscores a fundamental shift from merely supporting multiple languages to advancing truly integrated multilingual comprehension, providing a rigorous testbed for future advancements in this field.