LoRE: Logit-Ranked Retriever Ensemble

• LoRE is an open-domain question answering framework that integrates a retriever ensemble with logit-based ranking to mitigate positional bias.
• It employs both BM25 and dense retrieval using reciprocal rank fusion to aggregate diverse evidence and generate multiple candidate answers.
• LoRE computes a weighted LoR score by combining T5’s mean token confidence with the inverse retrieval rank, leading to significant improvements on SQuAD and NarrativeQA.

LoRE, short for Logit-Ranked Retriever Ensemble, is an open-domain question answering framework that modifies a standard retrieval-augmented generation pipeline in two places: it broadens evidence coverage through a retriever ensemble, and it selects the final answer with a logit-based ranking rule that combines language-model confidence with retrieval rank (Sanniboina et al., 2024). The method is motivated by positional bias in retrieval-based QA systems: once passages are retrieved and passed to the generator, content appearing earlier or in higher-ranked contexts tends to dominate the model’s answer, even when more relevant evidence exists lower in the retrieval list. In the paper’s framing, this produces factually wrong answers or hallucinations; LoRE addresses that failure mode by letting a candidate answer from a lower-ranked context win when its model support is stronger (Sanniboina et al., 2024).

1. Problem formulation and motivation

LoRE is defined in the setting of open-domain question answering in which a user asks a question and the system must retrieve relevant supporting text from a corpus and then generate an answer. The paper writes the basic retrieval-then-generation pipeline as

$A_i = \text{LM}(R(Q_i,D),Q_i).$

Its critique is that this pipeline can overtrust the top of the ranked list, a behavior the paper names positional bias (Sanniboina et al., 2024).

The paper’s motivating examples are concrete. For the question “What was Hero of Alexandria’s nationality?”, the system may answer “Polish” because it latches onto irrelevant retrieved material about famous people from Warsaw rather than historically relevant passages. The paper also characterizes this as especially harmful when the top result is lexically related but semantically wrong. In that sense, LoRE is not primarily a retriever replacement; it is a response to the interaction between retrieval ordering and answer generation (Sanniboina et al., 2024).

Retriever diversity is the first proposed remedy. The paper argues that different retrievers capture different notions of relevance. BM25 is described as strong at exact lexical overlap, while dense retrieval based on sentence embeddings can capture semantic similarity and paraphrasing. If one retriever misses relevant evidence or ranks it poorly, another may recover it. This motivates the ensemble stage that precedes generation.

A plausible implication is that LoRE belongs to a broader class of inference-time RAG control methods rather than to training-heavy QA systems. The paper itself emphasizes inference-time answer selection and explicitly avoids a learned reranker.

2. Retriever ensemble and pipeline structure

The overall LoRE pipeline has four main stages: retrieval with two retrievers, rank fusion, answer generation per retrieved context, and logit-ranked selection of the final answer (Sanniboina et al., 2024).

The implemented retrievers are limited to two components. The sparse retriever is BM25. The dense retriever is sentence-transformer-based dense retrieval using vector similarity, with FAISS used as the dense index for efficient nearest-neighbor search. No other retriever is actually implemented in the methodology section beyond those two, even though the introduction mentions other retrievers as general background.

For the dense retriever, similarity between query $q$ and document $d$ is defined by cosine similarity: $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$

For BM25, the paper gives the following printed formula: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$ and notes in effect that it is intended to be the standard BM25 form involving $f(t_i,d)$, $|d|$, $\text{avgdl}$, $k_1$, and $b$, but explicit values for $q$0 and $q$1 are not provided.

The ranked outputs are fused with Reciprocal Rank Fusion: $q$2 where the paper says $q$3 is typically set to $q$4. Here that $q$5 is the RRF constant, not the top-$q$6 retrieval depth. The paper includes a worked example with BM25 list $q$7 and sentence-transformer list $q$8, then sums RRF scores and sorts the result.

After fusion, the paper states that the “top $q$9 retrieved documents” are processed. It also notes an internal inconsistency: one subsection is titled “Segmentation into Context Chunks,” but the text explicitly says the method processes “the entire content of the top $d$0 retrieved documents rather than traditional segmentation.” In the described implementation, each retrieved full context $d$1 is paired with the query $d$2, and T5 Large generates one candidate answer per context: $d$3 where $d$4 denotes concatenation.

This design is operationally important. LoRE does not concatenate all retrieved passages into one generator prompt and then rely on the model’s internal preference over positions. It generates one answer per context and postpones final selection to a separate scoring stage.

3. Logit-ranked answer selection

The central innovation of LoRE is the LoR score, which combines a generation-side confidence signal with a retrieval-side prior (Sanniboina et al., 2024). The T5 Large model is used not only to generate answers but also to produce token-level logits that are converted into probabilities and averaged across the generated answer tokens.

The paper describes extracting output logits, applying a softmax, and then averaging the resulting token probabilities. The printed softmax equation is malformed, but the intended formula is: $d$5 where $d$6 is the logit for the $d$7-th token. The mean confidence score is then

$d$8

with $d$9 the number of answer tokens. The method does not mention sequence log-probability, beam-search score, calibration, or any normalization beyond the mean.

This token-confidence score is combined with the retrieval-side signal called context rank, defined as the original passage position after RRF fusion. The paper says this rank is “inversely applied,” giving higher credit to higher-ranked contexts via $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$0. The final score is

$$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$1

with empirically chosen weights $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$2 and $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$3.

The meaning of the terms is explicit in the paper. Mean Score is the average token probability of the generated answer under T5 and is the primary confidence signal. $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$4 is a retrieval relevance prior. The highest-scoring candidate answer is returned as the output. The paper emphasizes that LoRE does not train a separate reranker; it uses logits as a cheap alternative to “expensive re-ranking steps.”

A concise reconstruction of the inference procedure is:

1. Take input question $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$5.
2. Retrieve candidate documents/passages using BM25.
3. Retrieve candidate documents/passages using sentence-transformer embeddings searched with FAISS.
4. Fuse the ranked lists with RRF:

$$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$6

1. Select the top fused contexts.
2. For each context $$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$7, generate

$$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$8

1. Extract T5 logits for generated answer tokens.
2. Compute mean token probability.
3. Compute

$$\text{similarity}(q, d) = \frac{q \cdot d}{\|q\| \|d\|}.$$9

10. Return the candidate with the highest LoR score.

The implementation details are only moderately specified. The generator is always T5 Large. The paper contrasts its roughly 770M parameters with Blended RAG at 11B parameters. By contrast, several practical details are omitted: the exact sentence-transformer checkpoint, corpus preprocessing, retrieval depth per retriever, the top-$\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$0 number of contexts passed to T5, T5 decoding settings, maximum input length, duplicate-handling strategy, and even the exact indexing convention for context rank. The appendix reportedly shows rank $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$1 in one case, which conflicts with the use of $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$2.

4. Evaluation and reported results

LoRE is evaluated on SQuAD and NarrativeQA, using Exact Match (EM), F1, and ROUGE-L as metrics (Sanniboina et al., 2024). SQuAD is described as over 100,000 question-answer pairs from Wikipedia. NarrativeQA is described as over 3,460 question-answer pairs based on summaries and full texts of 767 literary works and movie scripts.

The baselines include three internal retrieval-generation variants and three external comparison rows. The internal baselines are:

• Naive - BM25
• Naive - FAISS
• Naive - Ensemble

The external comparison rows are:

• RAG original
• RAG end2end
• Blended RAG

The most informative ablations are the comparisons between each naive setup and its LoRE-scored counterpart: Naive BM25 vs LoRE-BM25, Naive FAISS vs LoRE-FAISS, and Naive Ensemble vs LoRE-Ensemble.

Main reported scores

Dataset	System	EM	F1	ROUGE-L
SQuAD	Naive BM25	19.41	34.78	31.90
SQuAD	Naive FAISS	26.40	42.80	39.40
SQuAD	Naive Ensemble	38.62	54.32	50.30
SQuAD	LoRE-BM25	56.66	63.91	59.30
SQuAD	LoRE-FAISS	57.28	65.19	60.10
SQuAD	LoRE-Ensemble	61.45	69.27	64.80
NarrativeQA	Naive BM25	0.00	3.33	2.34
NarrativeQA	Naive FAISS	0.00	3.23	2.32
NarrativeQA	Naive Ensemble	0.00	3.29	2.33
NarrativeQA	LoRE-BM25	31.99	48.92	41.39
NarrativeQA	LoRE-FAISS	27.68	42.55	35.41
NarrativeQA	LoRE-Ensemble	31.23	47.92	40.37

On SQuAD, the paper highlights the gain of LoRE-Ensemble over Naive Ensemble as:

• ROUGE-L: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$3
• EM: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$4
• F1: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$5

These are exactly the improvement values quoted in the abstract.

On NarrativeQA, the table shows especially large jumps relative to naive retrieval baselines. Using the table values, LoRE-Ensemble improves over Naive-Ensemble by:

• EM: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$6
• F1: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$7
• ROUGE-L: $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$8

The paper contains two internal inconsistencies that are important for interpretation. First, for NarrativeQA, the narrative text claims Ensemble + logits achieves ROUGE-L 48.8, while the table reports 40.37. Second, the results section reportedly repeats a 14.5 ROUGE-L improvement figure on NarrativeQA, which does not match the table. The structured table is therefore the more stable source.

The qualitative examples are consistent with the scoring mechanism. In the appendix example “When was the last Super Bowl in California?”, the correct answer “2003” is not necessarily the first retrieved candidate, but it receives a high logit probability, reported as 0.984, and is selected. Another example selects “296” from context rank 7, illustrating that LoRE can let a lower-ranked context win. In the main text, for “What petroleum company was a Super Bowl sponsor?”, LoRE predicts “Chevron”, while the base ensemble predicts the hallucinated string “neo-maeli.”

These examples support the paper’s claim that logits can act as a hallucination filter even when they do not guarantee correctness.

5. Interpretation, strengths, and limitations

The paper’s explanation of LoRE’s behavior is straightforward. It balances two complementary signals: retrieval relevance from the ensemble ranking and generation confidence from T5 logits (Sanniboina et al., 2024). The retriever ensemble increases the chance that relevant evidence is present at all. The logit-based ranking then mitigates positional bias by allowing an answer generated from a lower-ranked context to beat one from a higher-ranked context when the model is much more confident in it.

This suggests a specific mechanistic view of LoRE. It is not a retriever-only method and not a generator-only confidence method. Its contribution lies in coupling a modest retrieval prior with a stronger answer-confidence signal at inference time. Because the retrieval contribution is only $\text{score}(q, d) = \sum_{i=1}^{n} \text{IDF}(t_i) \cdot \frac{f(t_i, d) \cdot (k_1 + 1)}{f(t_i, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl})}$9 in the final score, the system still favors higher-ranked contexts, but only weakly relative to model confidence.

The method also has clear limitations, several of which the paper states explicitly. Its effectiveness still depends on retrieval quality: if the answer is not present in the retrieved context, the method cannot recover it. The appendix reportedly notes that in such cases LoRE may prefer an answer that “has the most similar format” to the true answer. The value of the logit score also depends on the training data and the representativeness of the model. Computationally, LoRE calls the generator $f(t_i,d)$0 times, once per context, which increases inference cost even if the model is smaller than 11B-scale baselines.

The paper suggests future work in two directions: prompting the model to produce all outputs at once, and learning the LoR weights rather than setting $f(t_i,d)$1 and $f(t_i,d)$2 empirically.

A broader interpretation is possible but should be stated cautiously. The mechanism is framed around open-domain QA, and the evidence is QA-specific. Still, because the procedure is simply “multiple retrieved contexts, multiple candidate generations, rank with confidence plus retrieval prior,” a plausible implication is that the same pattern could apply to other RAG settings. The paper does not experimentally validate that broader claim.

6. Scope, terminology, and acronym ambiguity

In this context, LoRE refers specifically to “Logit-Ranked Retriever Ensemble for Enhancing Open-Domain Question Answering” (Sanniboina et al., 2024). The acronym is not unique in the literature. Other arXiv works use closely related forms for unrelated topics, including “LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editing” (Ouyang et al., 5 Aug 2025), “Local Rule-Based Explanations of Black Box Decision Systems” (Guidotti et al., 2018), “LoRe: Personalizing LLMs via Low-Rank Reward Modeling” (Bose et al., 20 Apr 2025), and “LORE: Logical Location Regression Network for Table Structure Recognition” (Xing et al., 2023).

That ambiguity matters because the QA method’s defining concepts—positional bias, retriever ensembling, RRF fusion, T5-generated candidate answers, and the LoR score—are specific to retrieval-augmented question answering and should not be conflated with those unrelated uses of the acronym.

Within open-domain QA, however, LoRE denotes a distinct inference-time framework: ensemble BM25 and sentence-transformer/FAISS retrieval, generate one answer per retrieved context with T5 Large, and choose the final output by a weighted combination of mean token probability and inverse context rank. That combination is the basis of its reported improvements on SQuAD and NarrativeQA and of its claim to mitigate positional bias in retrieval-based generation (Sanniboina et al., 2024).