Querit-Reranker: Training Compact Multilingual Rerankers via Efficient Label-Free Distribution Adaptation

Abstract: Deployable multilingual rerankers must generalize across languages, domains, and target ranking tasks while remaining efficient enough for second-stage reranking. However, adapting them to new target distributions typically requires extensive task-specific relevance annotations, which are costly to obtain. We present Querit-Reranker, a family of multilingual cross-encoder rerankers trained with a data-centric pipeline for label-efficient adaptation. We instantiate it as Querit-Reranker-A0.4B, initialized from an in-house MoE backbone with 0.4B activated parameters, and Querit-Reranker-4B, initialized from Qwen3-Embedding-4B. Our pipeline first learns general relevance modeling from large-scale ranking-oriented data, then adapts to target distributions through synthetic-query mining with teacher scores as continuous soft labels. To consolidate complementary task-adapted strengths, we further merge checkpoints via spherical linear interpolation, obtaining a single deployable model without runtime ensembling overhead. Using Qwen3-Embedding-0.6B as the shared first-stage retriever, Querit-Reranker-A0.4B improves average nDCG@10 from 54.11 to 59.28 on BEIR and from 59.87 to 67.70 on MIRACL. On MTEB Multilingual v2 Reranking, it also substantially outperforms larger embedding-based baselines, while Querit-Reranker-4B further achieves state-of-the-art performance among publicly available models. We release both models on Hugging Face.

• The paper introduces a three-stage pipeline that uses broad multilingual supervision, synthetic query mining, and SLERP-based model merging to achieve efficient, compact reranking.
• It leverages an adaptive pairwise ranking loss and parameter-efficient techniques, significantly improving nDCG@10 scores on BEIR, MIRACL, and MMTEB benchmarks.
• The approach minimizes reliance on manual annotations, enabling robust performance across diverse languages and domain-specific retrieval tasks.

Querit-Reranker: Efficient Label-Free Adaptation for Compact Multilingual Reranking

Motivation and Problem Statement

Deployable cross-encoder rerankers are crucial components for multilingual information retrieval and RAG systems, demanding fine-grained relevance modeling across diverse languages and corpora while maintaining second-stage inference efficiency. Conventional adaptation approaches rely heavily on dense, task-specific relevance annotations, presenting practical challenges in multilingual and domain-diverse settings. Querit-Reranker addresses the label scarcity problem by introducing a staged, data-centric adaptation pipeline that achieves high-quality reranking with minimal human annotation, leveraging synthetic data and parameter-efficient model merging strategies.

Pipeline and Technical Contributions

The key technical innovation is a three-stage pipeline for training compact and scalable multilingual rerankers:

1. Broad Multilingual Supervision: Stage I assembles large-scale, ranking-oriented supervision by combining open-source retrieval/reranking datasets and de-identified real-world search logs. The backbone model is exposed to diverse corpus and linguistic distributions, ensuring robust general relevance modeling.
2. Targeted Synthetic Query Mining: Stage II adapts the reranker to specific MMTEB tasks by generating synthetic, document-conditioned queries using task-specific LLM prompts. Teacher models (Llama-Embed-Nemotron-8B) provide continuous relevance soft-labels, enabling label-efficient adaptation without additional human annotation. This stage extends coverage to French, Russian, Chinese, and Japanese, focusing on educational, factoid, noisy web, and MS MARCO-style search.
3. Model Merging via SLERP: Stage III consolidates task-specialized checkpoints using spherical linear interpolation (SLERP). This produces a single deployable reranker leveraging complementary strengths from distinct adaptation runs, avoiding inference-time ensembles or increased latency.

   Figure 1: Training pipeline of Querit-Reranker illustrating staged data-centric adaptation and SLERP-based merging.

This pipeline is instantiated in two variants: Querit-Reranker-A0.4B—a compact MoE initialized from Querit-A0.4B (0.4B activated parameters)—and Querit-Reranker-4B, based on Qwen3-Embedding-4B (dense 4B backbone), both conforming to the cross-encoder paradigm.

Data Composition and Language Coverage

Stage I leverages multilingual open-source datasets and private data, with substantial representation across languages and retrieval tasks. The effectiveness of broad language coverage is empirically supported by improved generalization to diverse test sets.

Figure 2: Language distribution of Stage I training data, highlighting multilingual span.

Figure 3: Language distribution of open-source training data demonstrating the backbone of multilingual supervision.

The synthetic mining stage specifically targets four MMTEB corpora, generating high-quality queries tailored to local domain/query characteristics. Even without explicit cross-lingual data (query-document pairs in different languages), the overall language alignment and robustness are substantially improved.

Model Architecture and Ranking Objective

Querit-Reranker follows a cross-encoder scoring formulation, ingesting concatenated prompts comprising instruction, query, document title/content, and a terminal $[\mathrm{CLS}]$ token. The backbone encodes the input, and the $[\mathrm{CLS}]$ representation is used as the relevance score, further refined with a binary classification head. The final score is computed as the expectation over $[-1,1]$ weighted by the softmax probabilities.

During training, candidate documents are labeled with continuous teacher scores, and a pairwise hinge ranking loss is employed. The loss margin is adaptively scaled to the label difference, enforcing stronger order constraints with larger relevance gaps to directly optimize for reranking metrics.

Numerical Results and Parameter Efficiency

Querit-Reranker-A0.4B demonstrates strong parameter efficiency and competitive performance among rerankers below 4B. Querit-Reranker-4B achieves state-of-the-art reranking results on MMTEB-multilingual-v2, BEIR, and MIRACL benchmarks, outperforming numerous larger models including up to 27B parameter baselines.

Figure 4: Performance–parameter trade-offs on MTEB-multilingual-v2 reranking tasks, visualizing SOTA achieved by Querit-Reranker-4B.

Key metrics include:

• BEIR: With Qwen3-Embedding-0.6B as first-stage retriever, Querit-Reranker-A0.4B improves nDCG@10 from 54.11 to 59.28; Querit-Reranker-4B achieves 62.29, ranking first among all rerankers compared.
• MIRACL: Multilingual nDCG@10 rises from 59.87 (first-stage retriever) to 67.70 (A0.4B) and 71.13 (4B), demonstrating strong second-stage refinement.
• MMTEB: Querit-Reranker-4B attains 71.08 average on multilingual reranking subset and 65.61 on combined 8-task MMTEB, consistently ahead of SOTA baselines.

The pipeline enables label-efficient, scalable multilingual adaptation, and the model merging step via SLERP integrates complementary strengths into a single model without overhead.

Practical and Theoretical Implications

Querit-Reranker establishes a label-efficient recipe for adapting cross-encoder rerankers to multilingual retrieval settings. The synthetic mining paradigm and SLERP-based consolidation are generalizable to other tasks and architectures, facilitating rapid deployment and distribution adaptation with minimal annotation requirements. The results suggest strong practical utility for search, RAG, QA, and recommendation in resource-poor or domain-shifting setups.

From a theoretical perspective, the pairwise ranking loss with adaptive margin is well-suited for optimizing relative ordering tasks. However, the lack of explicit cross-lingual alignment or dedicated cross-lingual supervision may still leave gaps in robustness for low-resource or code-switched scenarios. The staged adaptation shows promising scalability from compact MoE architectures to larger dense models.

Limitations and Future Directions

Querit-Reranker does not optimize explicit cross-lingual reranking or alignment losses, and synthetic mining was limited to four MMTEB tasks due to resource constraints. Future work may extend synthetic query generation to more languages/tasks, incorporate explicit cross-lingual pairing and alignment objectives, and explore model merging strategies for further robustness and coverage.

Conclusion

Querit-Reranker presents a staged, data-centric pipeline for training compact and scalable multilingual rerankers, achieving SOTA performance with efficient label-free distribution adaptation. The proposed recipe—combining broad multilingual supervision, synthetic query mining, and parameter-space merging—demonstrates strong parameter efficiency and generalizability. These methods contribute both practical and theoretical advances in low-resource, multilingual retrieval, with implications for future refinement in cross-lingual adaptation and reranking.