MonoT5: T5-Based Document Reranker

• MonoT5 is a sequence-to-sequence ranking model that leverages the standard T5 architecture without modifications, generating a relevance token from query-document text prompts.
• It uses dense cross-attention over the joint query-document sequence, trading off some speed for enhanced context sensitivity compared to bi-encoder models.
• MonoT5’s effectiveness is demonstrated through varied scaling, fine-tuning on MS MARCO, and distillation techniques, enabling strong zero-shot performance in diverse retrieval tasks.

Searching arXiv for relevant MonoT5 papers and supporting context. arXiv search: MonoT5 ranking T5 reranking information retrieval. MonoT5 is a sequence-to-sequence ranking model that reuses the standard T5 encoder–decoder stack for document reranking without architectural changes. In the formulation summarized in recent retrieval work, a query–document pair is linearized into a single text prompt and the decoder is asked to generate a relevance token, typically “true” or “false”; the resulting logits are interpreted as ranking scores. Within neural information retrieval, MonoT5 occupies the cross-encoder end of the design spectrum: it trades the speed advantages of bi-encoders and late-interaction models for dense cross-attention over the joint query–document sequence, and is therefore commonly used as a second-stage reranker rather than as a first-stage retriever (Farivar, 25 Aug 2025).

1. Architectural identity and model family

MonoT5 is described as a simple re-use of the T5 encoder–decoder Transformer with no architectural changes, only new fine-tuning. The survey treatment emphasizes the same point in different terms: no extra classification layer or pooling strategy is grafted onto T5, and the standard decoder’s first-step projection into vocabulary space serves as the classifier. This means that the model’s ranking behavior is obtained from the pretrained sequence-to-sequence machinery itself rather than from an added ranking head (Rosa et al., 2022).

Several MonoT5 sizes are reported across the literature. The legal case entailment study uses HuggingFace’s “T5-3B” configuration for monoT5-3B, with 24 encoder layers, 24 decoder layers, model hidden size $d_{\text{model}}=1024$, feed-forward intermediate size $4096$, and 16 self-attention heads per layer. The same study states that monoT5-base uses 12 encoder and 12 decoder layers with $d_{\text{model}}=768$ and 12 heads, while monoT5-large uses 24 encoder and 24 decoder layers with $d_{\text{model}}=1024$ and 16 heads. The distillation work additionally reports monoT5-60M as a “T5-small” variant with 6 encoder and 6 decoder layers, hidden size 512, feed-forward size 2048, and 8 attention heads, and monoT5-220M as a “T5-base” variant with 12 encoder and 12 decoder layers, hidden size 768, feed-forward size 3072, and 12 attention heads (Laitz et al., 2024).

Variant	Encoder/decoder	Parameters
monoT5-60M	6 + 6	$\approx 60$ million
monoT5-base / monoT5-220M	12 + 12	$220$ million
monoT5-large	24 + 24	$770$ million
monoT5-3B	24 + 24	$\approx 3$ billion

This configuration pattern places MonoT5 within the T5 scaling family rather than defining a separate Transformer architecture. A plausible implication is that many of its empirical properties, including zero-shot behavior under scale, are inherited from T5’s pretrained encoder–decoder capacity and then exposed through ranking fine-tuning.

2. Input representation, scoring, and optimization

The canonical MonoT5 input format is a flat text sequence of the form

$$\text{“Query: } q \text{ Document: } d \text{ Relevant:”}$$

where $q$ is the query and $4096$0 is the candidate passage or paragraph. In the legal zero-shot setting, the same template is instantiated as “Query: 〈fragment of new judicial decision〉 Document: 〈candidate paragraph〉 Relevant:”. After tokenization, the model sees a SentencePiece sequence such as “▁Query: ▁q ▁Document: ▁d ▁Relevant: ▁”, using T5’s 32k subword vocabulary (Rosa et al., 2022).

On the decoder side, the standard MonoT5 formulation uses exactly one of two single-token outputs: “true” if the document is relevant and “false” otherwise. Because “true” and “false” each occupy exactly one token in the T5 vocabulary, the first decoder position provides direct relevance logits. In the legal case entailment study, if $4096$1 and $4096$2 are the emitted logits, the relevance score is

$4096$3

Training uses standard cross-entropy over the two target tokens. One formulation writes

$4096$4

while the survey states equivalently that the loss for a labeled pair $4096$5 with gold label $4096$6 is $4096$7. The survey further notes that because only one decoder step is instantiated at fine-tuning time, training is almost as cheap as a one-step classification head, while still retaining full encoder–decoder attention learned during pre-training (Farivar, 25 Aug 2025).

MonoT5 has also been used with graded output labels. In the query-expansion study, the model is described as generating a relevance label token such as “0”, “1”, …, “4”, and the logit of a positive or graded label is interpreted as the relevance score. This indicates that the basic sequence-to-sequence ranking template is not limited to binary relevance, although the binary “true”/“false” formulation remains the standard presentation (Li et al., 2023).

3. Position within semantic retrieval pipelines

The main architectural contrast between MonoT5 and other semantic retrievers is how query–document interaction is computed. In a dense bi-encoder such as DPR, query $4096$8 and passage $4096$9 are encoded independently into fixed-size vectors $d_{\text{model}}=768$0, and relevance is scored by $d_{\text{model}}=768$1. In late-interaction methods such as ColBERT, the two texts are again encoded separately into per-token vectors, followed by many-to-many maximum-dot-product aggregation. MonoT5 instead interleaves the full sequence $d_{\text{model}}=768$2 in one encoder pass, and the decoder’s cross-attention sees every token of $d_{\text{model}}=768$3 and $d_{\text{model}}=768$4 jointly at every layer. The survey characterizes this dense cross-attention as more expressive and context-sensitive, but also as computationally slower because it must be rerun for each candidate passage (Farivar, 25 Aug 2025).

This computational profile largely determines where MonoT5 is used in practice. The survey states that in the original MonoT5 study, the model was fine-tuned on MS MARCO passage ranking and used at evaluation time to rerank the top $d_{\text{model}}=768$5 passages retrieved by a first-stage retriever such as BM25 or a dense bi-encoder. Standard evaluation metrics mentioned in that context include MRR@10, nDCG@10, and Recall@1000. The same survey recalls that MonoT5-Base matched or exceeded the MRR@10 of strong encoder-only rerankers, that MonoT5-Large improved further, and that the model placed in the top 2 of the public leaderboard at the time; it also reports substantial gains over DPR and ColBERT on TREC Deep Learning when used as a second-stage reranker (Farivar, 25 Aug 2025).

Throughput remains the principal operational constraint. The survey reports that MonoT5-Base processes roughly 50–100 query-passage pairs per second on modern GPUs, whereas DPR can operate at thousands of pairs per second when passage embedding is amortized through offline indexing. The legal deployment study gives a more application-specific view: inference scores each $d_{\text{model}}=768$6 pair independently, batches several pairs on GPU to saturate throughput, and uses typical batch sizes from 16 to 64 pairs depending on GPU memory. It also reports latency optimizations through FP16 inference, persistent CUDA kernels, caching of repeated queries and of early BM25 reranking, and asynchronous batching, yielding 1–2 s per query for a candidate pool of approximately 30 paragraphs in a live system (Rosa et al., 2022).

A common misconception is that MonoT5’s cross-encoder cost precludes production use. The legal search-engine demonstration does not eliminate the latency trade-off, but it does show that real-time deployment is possible when the candidate pool is small and batching, FP16 inference, and lightweight answer-selection logic are applied.

4. Parameter scaling and the legal case entailment study

A detailed case study of MonoT5 appears in the legal case entailment task of the COLIEE 2022 competition. The study uses no additional in-domain fine-tuning on COLIEE data; the approach is explicitly described as pure zero-shot. Fine-tuning is instead performed on MS MARCO, where the model is trained to complete the token “Relevant:” with either the single token “true” or “false,” and the legal task is approached by directly applying the resulting ranker to query–paragraph pairs from judicial decisions (Rosa et al., 2022).

The main empirical question in that study is parameter scaling. Three zero-shot model sizes are compared: monoT5-base with 220M parameters, monoT5-large with 770M parameters, and monoT5-3B with approximately 3,000M parameters. On the COLIEE 2021 test set, monoT5-base achieves $d_{\text{model}}=768$7, monoT5-large $d_{\text{model}}=768$8, and monoT5-3B $d_{\text{model}}=768$9. On the COLIEE 2022 test set, monoT5-base achieves $d_{\text{model}}=1024$0 and monoT5-3B $d_{\text{model}}=1024$1. The authors summarize the trend as a modest drop between base and large, but a clear jump once the 3 billion-parameter scale is reached, supporting the hypothesis that very large models have stronger out-of-domain zero-shot capabilities for this task (Rosa et al., 2022).

The same study situates those numbers against baselines and ensembles. On COLIEE 2021, BM25 reaches $d_{\text{model}}=1024$2 and DeBERTa (350M) reaches $d_{\text{model}}=1024$3, while the monoT5-base + monoT5-3B ensemble reaches precision $d_{\text{model}}=1024$4, recall $d_{\text{model}}=1024$5, and $d_{\text{model}}=1024$6. On COLIEE 2022, the median of submissions is $d_{\text{model}}=1024$7, the second-best single team reaches $d_{\text{model}}=1024$8, monoT5-3B reaches precision $d_{\text{model}}=1024$9, recall $\approx 60$0, and $\approx 60$1, and the monoT5-base + monoT5-3B ensemble reaches $\approx 60$2 (Rosa et al., 2022).

Answer selection in this system is not a pure threshold on pairwise scores. After scoring, three rules with thresholds $\approx 60$3 chosen by grid-search on COLIEE’s training split are applied: keep all paragraphs whose score satisfies $\approx 60$4; or keep the top $\approx 60$5 paragraphs by score; or keep any paragraph whose score is within a factor $\approx 60$6 of the top score. The reported settings include $\approx 60$7 for monoT5-base and $\approx 60$8 for monoT5-3B. This suggests that effective zero-shot deployment of MonoT5 can depend not only on model scale but also on post-scoring selection rules calibrated to task structure.

5. Query expansion, zero-shot generalization, and a correction to an earlier negative result

Query expansion for strong cross-encoder rankers has been treated as a difficult case. The 2023 study explicitly notes a recent result by Weller et al. reporting that current expansion techniques benefit weaker models such as DPR and BM25 but harm stronger rankers such as MonoT5. It then re-examines that conclusion by asking whether query expansion can improve the generalization of strong cross-encoder rankers, with MonoT5$\approx 60$9 as one of the target rerankers (Li et al., 2023).

The proposed pipeline rests on two principles: high-quality keyword generation and minimal-disruptive query modification. Keyword generation uses an instruction-following LLM, Flan-PaLM2-S, prompted with chain-of-thought reasoning to produce concise supporting keywords or subtopics. A second route, called Q2D2K in the paper’s ablations, first generates two hypothetical answer documents conditioned on the query and then extracts salient phrases as keywords. Repeating the LLM pipeline three times yields 30 raw candidates, after which self-consistency filtering selects the top $220$0 keywords by vote majority; the reported best setting is $220$1 (Li et al., 2023).

The study argues against bulk concatenation of all keywords into a single expanded query. Instead, it forms one expanded query per keyword, reranks each independently, and fuses the resulting rankings. If $220$2 is the top-1 document for the original query, then each expansion receives a reciprocal-rank weight

$220$3

and the final score combines the expanded-query scores with the original-query score using a regularization weight $220$4. First-stage retrieval is BM25 top-1,000; second-stage reranking is performed without cross-validation or fine-tuning on TREC Deep Learning or BEIR test sets (Li et al., 2023).

Under this protocol, MonoT5$220$5 improves in zero-shot evaluation. The reported nDCG@10 values are 0.695 to 0.724 on TREC DL 2019, 0.720 to 0.730 on TREC DL 2020, 0.470 to 0.476 on BEIR overall, and 0.519 to 0.532 on the “Wiki+News” subset in BEIR, with all improvements statistically significant under paired $220$6-test at $220$7. The ablations further report that Q2D2K-fusion reaches 0.751 on DL2019 versus 0.745 for the next-best PRF+D2K baseline, that performance peaks at $220$8, that reciprocal-rank weighting outperforms mean pooling and the other tested weighting strategies, and that inserting all three keywords at once hurts performance whereas one-by-one minimal expansions plus fusion help (Li et al., 2023).

The substantive implication is not that query expansion universally helps MonoT5, but that naive expansion can degrade strong cross-encoders while carefully controlled expansion and fusion can improve zero-shot generalization.

6. Distillation, synthetic in-domain data, and smaller MonoT5 variants

MonoT5’s inference cost motivated later work on distillation. InRanker is described as a version of monoT5 distilled from monoT5-3B with increased effectiveness on out-of-domain scenarios. The central idea is to generate synthetic “in-domain” training data that closely resembles the data seen at retrieval time, and then distill a frozen monoT5-3B teacher into smaller students using teacher soft labels rather than hard binary decisions (Laitz et al., 2024).

The pipeline has two phases. Phase 1 performs supervised distillation on MS MARCO passage ranking using teacher logits for one positive pair and $220$9 BM25 negatives sampled from the top 1000. The teacher produces two logits, $770$0 and $770$1, which are normalized by subtracting their mean:

$770$2

The student is trained with mean squared error on both normalized logits. Phase 2 repeats the same loss on synthetic “in-domain” queries generated with the InPars pipeline over each BEIR corpus: 100K synthetic queries per dataset, 1.6M queries total, 9 BM25 negatives per query, and therefore 16M training pairs (Laitz et al., 2024).

The reported training hyperparameters are AdamW with learning rate $770$3 and default $770$4 values, batch size 32, 400K steps for human soft distillation, and 329K steps for synthetic soft distillation. Compute on one A100-80GB is reported as approximately 7 h and 5.5 h for the 60M model, 15 h and 12 h for the 220M model, and 300 h and 250 h for the 3B model across the two phases (Laitz et al., 2024).

The effectiveness gains are substantial on BEIR zero-shot evaluation. Across 16 datasets, monoT5-60M improves from 0.4125 baseline to 0.4356 after human soft distillation and to 0.4807 after synthetic soft distillation; monoT5-220M improves from 0.4638 to 0.4870 and then to 0.5008; and the 3B teacher improves from 0.5174 to 0.5253 after the synthetic phase. Per-dataset examples include TREC-COVID for T5-small, 0.6928 to 0.7247 to 0.7775; HotpotQA for T5-base, 0.6950 to 0.7546 to 0.7742; and SciFact for T5-small, 0.6963 to 0.7022 to 0.7490. The paper further states that InRanker-60M is 50$770$5 smaller than monoT5-3B, InRanker-220M is 13$770$6 smaller, and T5-small runs roughly 4–5$770$7 faster in end-to-end latency while fitting comfortably on a single server GPU or CPU deployment (Laitz et al., 2024).

This line of work suggests that MonoT5 is not only a high-capacity reranker but also a teacher architecture whose cross-encoder behavior can be transferred to substantially smaller models. In that sense, MonoT5 functions both as a retrieval model and as a source of soft relevance structure for zero-shot generalization in compressed rankers.