Comparative Retriever with CMC Framework

• Comparative Retriever (CR) is a retrieval reranking method that leverages the CMC framework to jointly compare multiple candidate embeddings using shallow self-attention.
• It integrates between bi-encoder and cross-encoder stages to improve recall by 3.5%-4.8%-points on benchmarks like ZeSHEL while incurring minimal latency.
• The CMC framework employs a scalable Transformer architecture that enhances throughput and precision in tasks such as entity linking and dialogue ranking.

A Comparative Retriever (CR) implemented via the Comparing Multiple Candidates (CMC) framework is an advanced reranking architecture designed to simultaneously improve retrieval recall and top-1 ranking efficiency within classic retrieve-and-rerank paradigms. CMC enables joint, contextualized comparison of a query with multiple candidate embeddings through shallow self-attention, bridging the speed–accuracy gap between bi-encoder (BE) and cross-encoder (CE) methods by directly modeling their intermediate candidate sets with a scalable Transformer. Designed for deployability as either an intermediate reranker or a final-stage scoring mechanism, CMC supports high-throughput settings, outperforming traditional pipelines on a range of retrieval tasks in both accuracy and end-to-end latency (Song et al., 2024).

1. Pipeline Integration and Function

Typical retrieval pipelines consist of a fast bi-encoder (BE) for large-scale candidate retrieval, followed by a slower, more expressive cross-encoder (CE) reranker for a limited candidate pool. The CMC layer is introduced between BE and CE:

• Stage 1: Bi-Encoder Retrieval The query $q$ and all candidate $c_i$ embeddings are processed with a BE, with Maximum Inner Product Search (MIPS) returning the top-$K$ candidates as $C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$.
• Stage 2: CMC Reranking CMC ingests the single query $q$ and all $K$ candidate embeddings, jointly encoding them via shallow self-attention to compute contextualized similarity scores for all $K$ candidates in parallel.
• Stage 3: Optional Cross-Encoder Final Rerank The highest-scoring $K'$ ($K' < K$) candidates from CMC may be forwarded to a CE for final strict reranking.

This yields an enhanced pipeline: BE $\rightarrow$ CMC $c_i$0 CE, where CMC supplies a "virtually enhanced" retrieval stage, producing improved recall metrics (e.g., $c_i$1R@16$c_i$2+4.8%-p, $c_i$3R@64$c_i$4+3.5%-p on ZeSHEL) with negligible latency increase ($c_i$57%) and, as a standalone reranker, is both faster (11$c_i$6) and more effective on specific downstream tasks (+0.7%-p on Wikipedia EL, +3.3 MRR on DSTC7 dialogue ranking) relative to CE (Song et al., 2024).

2. Mathematical Formulation

CMC is composed of several key formal elements:

2.1 Sentence-level Encodings

A query $c_i$7 and each candidate $c_i$8 are encoded as follows:

• Query: $c_i$9
• Candidate: $K$0

With two separate encoders $K$1 and $K$2, extract single-vector summaries:

$K$3

2.2 Multi-Candidate Self-Attention

Stack query and $K$4 candidate embeddings:

$K$5

Process $K$6 via $K$7 Transformer encoder layers (no positional encoding, $K$8, $K$9, $C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$0):

$C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$1

with $C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$2.

2.3 Scoring & Selection

Score each contextualized candidate against the contextualized query:

$C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$3

Select top-1:$C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$4; for further reranking, pass top-$C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$5 by $C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$6 downstream.

2.4 Training Loss

Employ a multi-class cross-entropy over $C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$7 candidates:

$C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$8

where $C_q = \{c_{q,1}, \dotsc, c_{q,K}\}$9. Optionally, include a KL regularizer to bias toward the BE's retrieval distribution $q$0:

$q$1

3. Theoretical and Empirical Efficiency

CMC offers a favorable tradeoff between computational complexity and retrieval effectiveness:

Approach	Typical Cost per Query	Latency (Empirical)
BE	$q$2	$q$3 ms (K=512, ZeSHEL)
CE	$q$4	$q$5 ms (K=64, ZeSHEL)
CMC	$q$6	$q$7 ms (K=512, ZeSHEL)

• Complexity: For moderate $q$8 (e.g., $q$9), $K$0 (CMC) versus $K$1 for typical CE token lengths ($K$2).
• Empirical Throughput: CMC can process up to $K$3 candidates in a single batch, limited by GPU memory, unlike CE which scales poorly beyond a few dozen documents.
• Latency: On ZeSHEL, BE+CMC ($K$4) requires $K$5 ms ($K$6 BE alone); CMC-filtered 16 + CE takes $K$7 ms ($K$8 CE@64).

4. Empirical Evaluation and Benchmarks

CMC achieves notable improvements across a spectrum of retrieval and ranking tasks:

4.1 ZeSHEL Retrieval (Recall@K)

• BE@64 Recall: 87.95%
• BE+CMC@64 Recall: 91.51% ($K$9%-p)
• BE+CMC@16 Recall: 86.32% vs. BE@16: 81.52% ($K$0%-p)

4.2 Downstream Accuracy

• Wikipedia EL (AIDA/MSNBC/WNED-CWEB avg):
  • CE: 80.2%-p
  • CMC: 80.9%-p ($K$1%-p)
• DSTC7 Dialogue (MRR@10):
  • CE: 73.2
  • CMC: 76.5 ($K$2 MRR)

All improvements are statistically significant at $K$3.

4.3 Datasets and Hyperparameters

• Entity Linking: AIDA-CoNLL, WNED-CWEB, MSNBC, ZeSHEL ($K$4 k–$K$5 k entities)
• Passage Ranking: MS MARCO ($K$6 M passages)
• Dialogue: DSTC7 track1 ($K$7 candidates)

CMC configurations: BERT-base/large encoders (e.g., BLINK/CoCondenser), 2-layer/$K$8-head/$K$9-dim self-attention, max query‑lengths $K'$0/$K'$1, max doc length $K'$2/$K'$3; learning rate in $K'$4, batch $K'$5–$K'$6, epochs $K'$7–$K'$8, with hard negatives from BE.

5. Engineering and Practical Deployment

For insertion into BE→CE pipelines:

1. Offline corpus preparation: Encode all items via $K'$9, store $K' < K$0 for lookup.
2. Online per-query retrieval:
   • Encode $K' < K$1 using $K' < K$2.
   • BE retrieves top-$K' < K$3 candidate IDs.
   • Retrieve precomputed $K' < K$4.
   • Concatenate with $K' < K$5 to form $K' < K$6, process through CMC's Transformer.
   • Optionally invoke CE on CMC's top-$K' < K$7 outputs.

Parameter guidance:

• $K' < K$8 in $K' < K$9 recommended; CMC robust up to $\rightarrow$0.
• 2 self-attention layers, 12 heads, are sufficient.
• Use skip connections per self-attention block for training stability.

6. Training and Inference Workflows

Training pseudocode:

$\rightarrow$1

Inference pseudocode:

$\rightarrow$2

CMC thus unifies the scalability of BE retrieval with the contextual discrimination of cross-attention, offering a high-throughput comparative retriever suitable for modern large-vocabulary ranking tasks (Song et al., 2024).