Zero-Shot Rerankers

• Zero-shot rerankers are models that reorder retrieval candidates using only pretrained signals without task-specific fine-tuning.
• They employ pointwise, pairwise, and listwise strategies with LLMs, VLMs, and multimodal transformers to assess relevance via prompting.
• Empirical studies show these models achieve competitive nDCG scores and efficient domain adaptation for cross-modal and low-resource retrieval tasks.

A zero-shot reranker is a neural model that reorders a candidate set of retrieved items by scoring their relevance to a query, operating without any supervised fine-tuning on in-domain or labeled relevance data specific to the task. Modern zero-shot reranking utilizes large pretrained LLMs, vision-LLMs (VLMs), or multimodal transformers to assign relevance estimates, often through prompting, probabilistic modeling, or comparison strategies. This paradigm has become central to contemporary information retrieval, retrieval-augmented generation (RAG), and cross-modal search pipelines.

1. Principles and Formalisms of Zero-Shot Reranking

Zero-shot reranking diverges from classical supervised rerankers, which are fine-tuned on large, in-domain relevance datasets such as MS MARCO. Instead, zero-shot rerankers leverage only the pretraining signal from massive, unannotated corpora. Most zero-shot rerankers adopt one of the following operational frameworks:

• Pointwise: Each query–document (or query–candidate) pair is scored individually, either by generation likelihood or scalar model output.
• Pairwise: The model is prompted to compare pairs of candidates for the same query, e.g., "which of these two is most relevant?" Final rankings aggregate pairwise comparisons.
• Listwise: The full set of k candidates is provided in a single prompt and the model predicts a permutation (ordering) of the entire list.

The query likelihood model (QLM) approach is extensively used for zero-shot text reranking. The canonical form computes the probability of generating the query $q$ given candidate document $d$ as

$$S_{QLM}(q,d) = \frac{1}{|q|} \sum_{t=1}^{|q|} \log\,P(q_t \mid p, d, q_{<t})$$

where $p$ is a prompt ("generate a question") and $LLM(\cdot\,|\,\ldots)$ is the token-probability under the LLM decoder conditioned on the prompt and document (Zhuang et al., 2023).

In listwise methods, permutation modeling follows a Plackett–Luce factorization:

$$P(\pi \mid q, D) = \prod_{i=1}^{k} P(\pi(i)\mid q, D, \pi(1), \ldots, \pi(i-1))$$

where $\pi$ is a permutation over the candidate indices (Ma et al., 2023).

2. Reranker Architectures and Prompting Strategies

Zero-shot rerankers are implemented using a range of architectures:

• Decoder-only LLMs: (e.g., LLaMA, Falcon, GPT) used for QLM-style zero-shot ranking, listwise generation, or comparison-based reranking (Zhuang et al., 2023, Zhang et al., 2023).
• Encoder–Decoder (Seq2Seq) Models: (e.g., T5, mT5) adapted for listwise reranking (e.g., ListT5, LiT5) with fusion-in-decoder and sequence-gen objectives (Yoon et al., 2024, Tamber et al., 2023).
• Vision-LLMs: (e.g., LLaVA, Qwen-VL, InternVL) for cross-view, text-image, or video-text reranking; utilize image serialization and multimodal listwise prompts (Erzurumlu et al., 28 Mar 2026, Eltahir et al., 3 Nov 2025).
• Distilled/Student Models: Students distilled from large teacher rerankers using soft labels or synthetic queries for latency efficiency (Laitz et al., 2024).

Prompt design critically influences zero-shot effectiveness. For QLMs, prompts like “generate a question from this passage” maximize query likelihood discrimination. In pointwise and listwise settings, best practices include:

• Listwise: Full serialization of candidates with clear index labels (e.g., "[1] ... [n]") and explicit instruction to output a sorted identifier list (Zhang et al., 2023, Pradeep et al., 2023).
• Pairwise: Focused prompts comparing two candidates with constrained outputs (e.g., JSON-encoded preferences) (Erzurumlu et al., 28 Mar 2026).
• Fine-grained Labels: For pointwise models, using more than binary ("yes"/"no") relevance options (e.g., 3–4 levels) yields significantly better discrimination (Zhuang et al., 2023).
• Discrete Prompt Optimization: Algorithmic search for optimal discrete prompts (e.g., Co-Prompt) consistently improves zero-shot ranking (Cho et al., 2023).

3. Empirical Effectiveness and Evaluation

Zero-shot reranking closes much of the performance gap to supervised and transfer-learned rerankers on standard IR benchmarks:

• Textual Retrieval: Decoder-only LLM QLMs (e.g., Falcon-40B) reach average nDCG@10 of 53.1, surpassing strong unsupervised baselines (BM25: 38.9, classical Dirichlet QLM: 35.4), and sometimes match or exceed instruction-tuned LLMs unless question generation (QG) data is included in fine-tuning (Zhuang et al., 2023).
• Listwise LLM Rerankers: Open-source, well-finetuned LLMs (e.g., RankZephyr) can surpass closed GPT-4 models on TREC DL and out-of-domain Google search test sets, with nDCG@10 up to 0.7855 (DL19, R³ pass) (Pradeep et al., 2023).
• Open-source Listwise Models: Properly trained listwise rerankers on Code-LLaMA-Instruct achieve ≥97% of GPT-4's reranking effectiveness, given ~5–10k high-quality listwise data points (Zhang et al., 2023).
• Vision-Language and Multimodal: Pairwise or listwise VLM prompting (e.g., LLaVA, ViC/InternVL) unlocks robust zero-shot reranking in video/text retrieval and cross-view geolocalization, delivering state-of-the-art Recall@1 (+2–40 points over classical fusion) (Eltahir et al., 3 Nov 2025, Erzurumlu et al., 28 Mar 2026).
• Listwise > Pairwise > Pointwise: Presenting an entire candidate set enables better cross-candidate reasoning, and listwise strategies consistently outperform pointwise in nDCG@10 by 4–6 points (Ma et al., 2023). However, the practical context window limits window size; sliding-window and tournament strategies mitigate this.

Typical evaluation metrics include nDCG@10, MRR@10, MAP@100, and Recall@k. Few-shot prompting and hybrid retriever integration offer further incremental improvements, e.g., ~2 nDCG@10 points from 3 good/bad QG examples (Zhuang et al., 2023).

4. Design Trade-offs, Data Regimes, and Efficiency

The efficiency and generalization capacity of zero-shot rerankers involve several trade-offs:

• Prompt/Instruction Tuning: General instruction tuning (e.g., on dialog/Q&A) can harm QLM performance unless the tuning data contains QG tasks, which directly support the QLM objective (Zhuang et al., 2023).
• Data for Listwise Training: Existing pointwise or binary-labeled datasets are insufficient for optimal listwise reranker training; high-quality listwise orderings (preferably annotated or from strong teacher outputs) are essential (Zhang et al., 2023).
• Distillation for Efficiency: Large rerankers can be distilled into smaller (~60–220M parameter) models with only minor nDCG loss, enabling practical deployment (>10× speedup) with only 3–8 points reduction versus 3B-class models (Laitz et al., 2024).
• Sliding Window/Tournament Sort: Mechanisms such as m-ary tournament sort and progressive reranking control computational cost, enabling O(n) or O(n + k logₘ n) complexity comparable to pointwise models but yielding listwise discriminative power (Yoon et al., 2024).
• Cross-lingual and Low-Resource: Modular parameter-efficient adapters or SFTMs, and zero-shot listwise open-source LLMs (e.g., RankZephyr), enable robust effectiveness for cross-lingual and low-resource language reranking without labeled data (Litschko et al., 2022, Adeyemi et al., 2023).

5. Advanced Variants and Domain Adaptation

• Risk-Minimization (UR³): Bayesian risk minimization (UR³) addresses distribution bias in LLM QLMs by jointly optimizing for both query and document generation likelihood, providing consistent Top-1 accuracy gains over classical QLMs (e.g., +5.3% on NQ over UPR) with negligible extra cost (Yuan et al., 2024).
• Task Arithmetic: Parameter arithmetic enables rapid, training-free domain adaptation for rerankers by algebraically combining IR-tuned and domain-tuned model parameters: $\Theta' = \Theta_T + \alpha(\Theta_D - \Theta_0)$, with statistically significant (+2–18%) nDCG@10 gains in domain-mismatched or multilingual settings (Braga et al., 1 May 2025).
• ELO/Thurstone zELO: Modeling global document order via unsupervised pairwise preferences and Thurstone/Elo statistics gives rise to open-weight rerankers (zerank-1, zerank-1-small) that outperform closed-source commercial rerankers and generalize robustly to out-of-distribution domains (Pipitone et al., 16 Sep 2025).
• Prompt Optimization: Methods such as Co-Prompt automate discrete prompt search using constrained generation, systematically improving zero-shot reranker output and interpretability (Cho et al., 2023).
• Multimodal and Cross-Modal: ViC/LLaVA and similar frameworks generalize listwise reranking and fusion to multimodal, video, and cross-view tasks by serializing content evidence and retriever metadata in the prompt (Eltahir et al., 3 Nov 2025, Erzurumlu et al., 28 Mar 2026).

6. Practical Recommendations and Future Challenges

• Begin with an open-source, decoder-only LLM trained on generic corpora for strong zero-shot QLM reranking (Zhuang et al., 2023).
• For multilingual or cross-lingual IR, leverage adapters or open-source listwise rerankers, avoiding the need for target-language relevance labels (Litschko et al., 2022, Adeyemi et al., 2023).
• Combine first-stage zero-shot dense/sparse retrieval (BM25, HyDE) with LLM-based QLM or listwise re-ranking and simple linear or learned interpolation for optimal head effectiveness (Zhuang et al., 2023).
• Use listwise reranking or carefully constructed pairwise prompts for multimodal or vision–language tasks, avoiding pointwise absolute scoring due to calibration issues (Erzurumlu et al., 28 Mar 2026, Eltahir et al., 3 Nov 2025).
• Consider efficiency/latency requirements; distillation or smaller sequence-to-sequence models (e.g., LiT5-Distill, ListT5) offer a trade-off between reranking effectiveness and computational overhead (Tamber et al., 2023, Yoon et al., 2024).
• Prioritize development of high-quality, graded, human-annotated listwise datasets for further advances, as both data quality and scale are the main bottlenecks to progress (Zhang et al., 2023).

A plausible implication is that as LLM architectures and training regimes continue evolving, the modular, prompt-driven, and data-efficient methodologies employed by zero-shot rerankers will likely generalize to an ever-broader spectrum of retrieval and ranking scenarios across modalities and domains, provided sufficient investment in robust listwise benchmarks and scalable evaluation protocols.