---
title: Zero-shot Document Ranking Overview
url: https://www.emergentmind.com/topics/zero-shot-document-ranking
type: topic
---

# Zero-shot Document Ranking Overview

Zero-shot document ranking refers to the task of ordering a set of documents for a given query using models that have not been explicitly fine-tuned for relevance ranking on any supervised or in-domain dataset. Recent advancements exploit large language models (LLMs) in a zero-shot regime, achieving strong effectiveness by leveraging prompt-based strategies, generative modeling, or emergent comparative abilities. This wide-ranging research area spans multiple ranking paradigms (pointwise, pairwise, listwise, setwise, and tournament-inspired), prompt optimization, and hybridized retrieval pipelines, with extensive empirical validation across benchmarks such as BEIR, TREC Deep Learning, and MS MARCO.

## 1. Core Paradigms in Zero-Shot Document Ranking

Methods are typically categorized according to how they operationalize the ranking signal from the LLM:

- **Pointwise:** Each candidate document is scored independently with respect to the query. Classical query-likelihood models (QLMs) prompt the LLM to estimate $P(q|d)$ by scoring the likelihood of generating each query token given the document and a prompt (e.g., "Generate a question that is most relevant to the given article’s title and abstract") [2310.13243]. Pointwise models offer efficiency (scaling linearly in the number of candidates), and can make use of output logits for scalar relevance scores (e.g., log-likelihoods or normalized probabilities for "yes"/"no") [2506.11452].

- **Pairwise:** The LLM is given (query, document$_1$, document$_2$) and returns a preference or relative relevance judgment, often via a forced-choice prompt ("Which passage is more relevant to the query?"). All-pairs approaches require $O(n^2)$ calls, but can be made more practical through sorting algorithms that require $O(n\log n)$ (heapsort) or $O(n)$ (insertion sort) comparisons [2310.09497].

- **Listwise:** The full candidate list (or manageable windows thereof) is presented to the LLM, which outputs a permutation or a sorted subset. Pipelines such as Listwise Reranking with a Large Language Model (LRL) directly generate the identifier ordering in a single sequence, optimizing $\prod_{k=1}^m P(\mathrm{id}_{i_k}|q, D, \mathrm{id}_{i_1}, \ldots, \mathrm{id}_{i_{k-1}})$ [2305.02156]. Constraints on input length are addressed by sliding-window or progressive listwise schemes.

- **Setwise:** Building on pairwise and listwise, setwise approaches present small groups ($c \geq 3$) and prompt the LLM to select the most relevant among them, integrating selection into $c$-ary sorts. Compared to pairwise, setwise reduces the number of LLM calls by up to 60–70% while retaining or improving effectiveness [2310.09497, 2504.10509].

- **Hybrid and Tournament:** Methods such as TourRank partition candidates into groups per stage and ensemble over multiple randomized tournaments (re-initialized groupings), accumulating per-document advancement points. This yields state-of-the-art effectiveness at modest, parallelizable cost and robustly mitigates input order and context length constraints [2406.11678].

The following table summarizes the complexity and effectiveness trade-offs across core paradigms:

| Method Type    | LLM Calls per Query | Typical nDCG@10 | Core Reference          |
| :------------- | :------------------ | :-------------- | :--------------------- |
| Pointwise      | $O(n)$              | 0.42–0.66       | [2310.13243], [2506.11452] |
| Pairwise       | $O(n^2)$            | 0.66–0.68       | [2310.09497]           |
| Setwise Heap   | $O(n + k\log_c n)$  | 0.67            | [2310.09497], [2504.10509] |
| Listwise LRL   | $O(n)$ windows      | 0.66–0.68       | [2305.02156]           |
| TourRank       | $O(kn)$ (parallel)  | 0.69–0.71       | [2406.11678]           |

## 2. Query-Likelihood Models and Prompt-based Scoring

Query-likelihood models (QLMs) adapted for LLMs rank documents by directly estimating the probability of a query given a document:

\[
S_{\mathrm{QLM}}(q, d) = \frac{1}{|q|} \sum_{t=1}^{|q|} \log P(q_t | p, d, q_{<t})
\]

where $p$ is a short template prompt, $d$ the document, and $q_{<t}$ the previously generated tokens. Exact prompt engineering is crucial, with dataset- and model-specific templates empirically shown to affect ranking accuracy [2310.13243]. QLM scores are often interpolated with zero-shot retriever outputs (e.g., BM25, HyDE) for final ranking, increasing $n$DCG@10 by up to 2 points with negligible computational overhead.

Prompt optimization extends to discrete search over prompt tokens (Co-Prompt), combining a generator's priors with a discriminator's re-ranking metric via beam search, consistently outperforming manual or reinforcement-learning–based prompt selection [2305.13729].

## 3. Comparative and Anchored Strategies: Pairwise, Setwise, and Reference-based Approaches

**Pairwise and setwise:** Comparative prompting methods explicitly consider cross-document distinctions. Pairwise ranks based on $P(d_i \succ d_j|q)$, but becomes computationally expensive. Setwise prompting, as formalized in Setwise Comparator $\mathrm{SetComp}_c$, asks which document is best among a group of $c$ candidates. Setwise insertion further incorporates prior ranking information (e.g., BM25 ordering), biases the LLM via prompt to favor the highest-priority item, and employs block-binary search for efficient top-$k$ extraction, reducing LLM calls by ~31% and latency by ~23% with a slight effectiveness gain [2504.10509].

**Reference-based/anchor methods:** RefRank and its variants sidestep exhaustive pairwise comparisons by selecting a single (or a handful of) anchor document(s) to serve as reference. Each candidate is compared against the anchor(s), and final ranks are aggregated by the anchors' comparative scores. Multi-anchor (ensemble) schemes (e.g., $m=5$ anchors) nearly match or surpass full pairwise sort accuracy at a fraction of the computational cost [2506.11452].

Global-Consistent Comparative Pointwise (GCCP) approaches construct a query-focused summary over top candidates as the anchor, leveraging unsupervised spectral multi-document summarization. Pointwise contrastive scores $\Delta s(d_i, q| a) = s(d_i, q) - s(a, q)$ or direct comparative prompts yield global consistency while retaining linear computational scaling. Post-aggregation with global context (PAGC) linearly combines these contrastive scores with standard pointwise scores, achieving near pairwise-level effectiveness with minimal cost increase [2506.10859].

## 4. Embedding-based and Explicit Representation Methods

PromptReps elicits both dense and sparse representations from general LLMs without further training, using tailored prompts for each to guide the LLM in selecting key representation tokens:

- **Dense representations:** The last hidden state $h_{\mathrm{last}}$ from the document prompt is normalized to yield $e_{\text{dense}}$.
- **Sparse representations:** Raw next-token logits are post-processed (activation, log-saturation, filtering) to yield a sparse bag-of-words vector $s$.

Hybrid indices (ANN for dense, inverted index for sparse) are then searched. The min–max–normalized final score is a convex combination of dense and sparse similarities, typically $\alpha=0.5$. Hybrid PromptReps with large LLMs (Llama3-8B+) reach $n$DCG@10 = 46.2 on BEIR (dense-only 16–22, sparse-only 32–35) and 50.1 when further combined with BM25, surpassing unsupervised state-of-the-art retrievers trained with large-scale paired data [2404.18424].

Recent work also establishes the theoretical foundation of learnable late-interaction models as universal approximators of continuous scoring functions, enhancing zero-shot transfer, reducing storage, and lowering inference cost versus ColBERT and cross-encoders [2406.17968].

## 5. Practical and Architectural Innovations

### Tournament-inspired and robust strategies

TourRank employs a multi-stage, tournament-style grouping where group advances are determined by LLM-prompted selection. Points across multiple randomized tournament repetitions are aggregated, stabilizing rankings against input-order and context-limit bias. With $r=10$ repetitions, TourRank-10 achieves $n$DCG@10 = 71.63 (DL19), exceeding supervised monoT5-3B and other zero-shot baselines with lower or fully-parallelizable wall-clock latency [2406.11678].

### Temporal and event-prediction scenarios

AutoCast++ applies zero-shot document ranking for event forecasting by combining LLM-based graded-relevance scoring with a temporal reweighting function grounded in human forecaster behavior. Normalized relevance is multiplied by a recency gain function precomputed from crowd-forecast logs, promoting temporally salient context. This yields substantial end-to-end accuracy improvements in event forecasting benchmarks (e.g., $+48\%$ MCQ accuracy vs. static pipelines) [2310.01880].

### Generative anchor and answer-scent cues

ASRank introduces an "answer scent," a query-conditioned natural language semantic target synthesized by a large LLM and used to guide answer generation from each candidate. Reranking is based on the log-likelihood of each document’s ability to generate the answer scent, with final scores integrating the document’s retrieval prior. This approach achieves major gains in Top-1 open-domain QA retrieval (NQ: $22.1 \to 47.3\%$ for BM25, $19.2 \to 46.5\%$ for MSS; BEIR nDCG@10: 48.39 vs. 45.78 for monoT5) while maintaining efficient query-time computation [2501.15245].

## 6. Task Adaptation, Domain Transfer, and Zero-Shot Limits

Task Arithmetic leverages model weight arithmetic to adapt cross-encoder or LLM-based rankers for zero-shot use in new domains. Given a base IR model $\Theta_T$ and a domain-adapted LM $\Theta_D$ (and their shared pre-trained weights $\Theta_0$), one computes task vectors $\tau_D = \Theta_D - \Theta_0$ and synthesizes new parameters $\Theta' = \Theta_T + \alpha \tau_D$. This process enables training-free, modular adaptation across scientific, biomedical, multilingual, and legal domains, yielding up to 18% relative gain in nDCG@10 versus task-agnostic baselines [2505.00649].

Comprehensive evaluation of long-document ranking models under zero-shot transfer (MS MARCO $\rightarrow$ TREC DL, Robust04, FarRelevant) reveals that chunk aggregation approaches (MaxP, PARADE) outperform pure first-chunk scoring on collections lacking positional bias. However, only modest gains ($\leq 5\%$) are attributed to true long-context modeling unless test distributions are explicitly adversarial to positional priors, underscoring the importance of robust aggregation strategies in zero-shot settings [2207.01262].

## 7. Frontiers: Visual and Multi-Modal Zero-Shot Retrieval

In the document image domain, SERVAL establishes a generate-and-encode zero-shot baseline: a vision-language model generates a rich description for each image, which is then embedded via standard (multilingual) text encoders for retrieval. Without any contrastive text-image training, SERVAL achieves $n$DCG@5 = 63.4% on ViDoRe-v2 (surpassing ColNomic-7B) and $n$DCG@10 = 72.1% on MIRACL-VISION, demonstrating the generality of zero-shot pipelines in high-dimensional multi-modal settings [2509.15432].

## References

- "Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking" [2310.13243]
- "PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval" [2404.18424]
- "A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models" [2310.09497]
- "Zero-Shot Listwise Document Reranking with a Large Language Model" [2305.02156]
- "Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion" [2504.10509]
- "Leveraging Reference Documents for Zero-Shot Ranking via Large Language Models" [2506.11452]
- "Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model" [2409.17745]
- "SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models" [2509.15432]
- "Efficient Document Ranking with Learnable Late Interactions" [2406.17968]
- "AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval" [2310.01880]
- "Document Ranking with a Pretrained Sequence-to-Sequence Model" [2003.06713]
- "TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy" [2406.11678]
- "Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker" [2305.13729]
- "Investigating Task Arithmetic for Zero-Shot Information Retrieval" [2505.00649]
- "Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding" [2207.01262]
- "ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval" [2501.15245]
- "Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information" [2506.10859]

Source: https://www.emergentmind.com/topics/zero-shot-document-ranking