WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval

Abstract: We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss. WebFAQ 2.0 is not a static resource but part of a long-term effort. Since late 2025, structured FAQs are being regularly released through the Open Web Index, enabling continuous expansion and refinement. We publish the datasets and training scripts to facilitate further research in multilingual and cross-lingual IR. The dataset itself and all related resources are publicly available on GitHub and HuggingFace.

• The paper introduces WebFAQ 2.0, offering 198M multilingual QA pairs and 1.25M query-centric hard negatives to enhance dense retrieval model training.
• The methodology employs a direct web crawling paradigm using schema.org markup and bilingual alignment, ensuring diverse and context-rich QA collections.
• Knowledge distillation with cross-encoder soft targets shows state-of-the-art non-English retrieval performance, despite trade-offs in English retrieval.

WebFAQ 2.0: Multilingual Question Answering at Scale with Hard Negatives for Dense Retrieval

Overview and Motivation

WebFAQ 2.0 introduces an extensively enlarged, multilingual FAQ-based question-answer dataset and a mined hard negatives resource, explicitly designed for the training and evaluation of dense retrieval models (2602.17327). It addresses core limitations of prior publicly available QA corpora: restricted multilingual coverage and the absence of expertly curated hard negatives essential for robust contrastive and distillation-based retrieval model training.

The dataset comprises 198 million natural QA pairs from 104 languages—more than doubling the size and the linguistic scope of the first WebFAQ iteration, and introducing over 14.3 million bilingual aligned QAs. Methodologically, WebFAQ 2.0 pivots from dependence on structured data dumps to a direct web crawling paradigm, yielding more diverse, relevant, and contextually rich QA collections.

Figure 1: WebFAQ is a massive, publicly available repository of natural question-answer pairs. The second version focuses on QAs aligned across languages and hard negative mining.

Data Acquisition and Dataset Structure

Crawling, Extraction, and Processing

WebFAQ 2.0 harvests FAQ pages globally using a distributed crawler (OWLer), seeding from both legacy URLs and direct mining of recent Common Crawl dumps. Schema.org FAQ markup serves as the schema anchor for extraction, enabling robust and scalable identification of QAs and automatic mapping of multilingual variants using <link rel="alternate" hreflang="..."> relations.

Compared to the original WDC-based acquisition, this pipeline provides (i) continuous dataset expansion as new web content becomes available, (ii) dramatically increased non-English coverage, and (iii) richer context through page-level metadata (titles, descriptions).

For quality, malformed or ambiguous extractions are filtered, and semantic similarity scores (using Jina v3 embeddings) are computed and included to assist downstream QA pair filtering and dataset curation. Unreliable near-duplicate removal in the previous release is replaced with domain-level uniqueness constraints within retrieval test bed creation.

Distribution and Coverage

The data now captures a broader and more linguistically balanced set—the English share drops from 51% to 28% despite substantial absolute growth. Coverage across underrepresented languages and language pairs is significantly improved, e.g., Hindi and Ukrainian each exceed 2.5M QAs, and high-quality bitexts span nearly 4,000 language combinations. The topic focus shifts toward Travel and Hospitality (59% of QAs), as detected by a fine-tuned, context-aware XLM-RoBERTa classifier.

A multilingual XLM-RoBERTa-based question taxonomy classifier (trained via LLM ensemble annotation) labels QA types—though noted as preliminary due to the reliance on automated LLM-generated labels, this facilitates finer-grained IR and QA experiments.

Bilingual Alignment

Mining of bilingual aligned QA pairs employs LaBSE to retrieve high-similarity pairs (threshold ≥ 0.9), with validation via GEMBA and inclusion into MTEB for benchmarking. Alignment quality and scale enables robust cross-lingual IR evaluation, notably extending to low-resource and non-English-centric language pairs.

Hard Negatives Mining and Utility

Hard negatives are indispensable for separating trivial from genuinely challenging cases during dense retriever training—a crucial point in retrieval-augmented architectures and contrastive representation learning paradigms. WebFAQ 2.0 delivers 1.25M query-centric hard negatives across 20 languages, extracted via a two-stage process: lexical retrieval with BM25, then cross-encoder reranking with BGE-m3, assigning per-negative relevance and supporting denoising (RocketQA-style) to mitigate false negatives.

The dataset supports two dominant training strategies:

• Contrastive Learning (MultipleNegativesRankingLoss): Selects negatives using reranking and denoising, optimizing embedding separation.
• Knowledge Distillation (MarginMSE): Direct supervision on dense retriever scores from cross-encoder output, enabling soft-target learning tuned to reranker semantics.

Evaluation Protocol and Numerical Results

Evaluations are benchmarked on three multilingual retrieval tasks: WebFAQ Retrieval (in-domain), MIRACL-HN (hard-negative focused), and Mr. TyDi (zero-shot, diverse languages). Models initialize from strong XLM-RoBERTa roots (MS MARCO pre-trained).

Key findings:

• Random negatives remain highly competitive in contrastive setups due to persistent false negatives in mined sets; unfiltered hard negatives can decrease retrieval performance, particularly with non-denoised selection.
• Denoising (score thresholding) partially mitigates performance drop, but does not fully close the gap to random sampling when hard negatives may be noisy or label ambiguity cannot be eliminated.
• Knowledge distillation using cross-encoder soft targets (MarginMSE) yields state-of-the-art non-English retrieval performance, with superior NDCG@10 on multiple language tracks. However, for English, this approach can reduce performance relative to base models, signifying a cross-lingual trade-off linked to pretraining bias and data prevalence.

Implications and Forward-looking Considerations

WebFAQ 2.0’s methodology and scale facilitate several advances:

• Multilingual IR and QA Robustness: The enlarged and balanced dataset enables the training and evaluation of retrieval models on an unprecedented array of languages and domains, accelerating development of IR architectures that generalize beyond high-resource settings.
• Hard Negatives at Scale: Release of hard negatives with cross-encoder denoising protocols catalyzes experimentation with contrastive and distillation-based fine-tuning, supporting the systematic study of negative sampling mechanics—critical given the nuanced effects observed herein.
• Continuous Benchmark Evolution: Integration with the Open Web Index for ongoing FAQ curation promises an updatable public resource, positioning WebFAQ 2.0 as a backbone infrastructure for reproducible, forward-compatible multilingual IR research.

Outstanding limitations are also highlighted—notably, the persistent challenge of false negatives contaminating hard negative sets, the imperfect quality of LLM-labeled taxonomy classification, and domain/topic skew (substantial emphasis on travel/hospitality FAQs).

Conclusion

WebFAQ 2.0 sets a new standard for multilingual QA and dense retrieval evaluation resources, delivering substantial data, improved linguistic and topical diversity, and the first large-scale, multilingual hard negatives set with denoising and bitext alignments. The dataset and its auxiliary resources underpin the next generation of robust, cross-lingual retrievers and facilitate nuanced exploration of negative mining and model distillation paradigms. The platform’s open-ended, continuously expandable nature will serve as a critical infrastructure for sustained progress in large-scale semantic retrieval and multilingual QA systems.