FiD-Light: Efficient Retrieval-Augmented Generation

• FiD-Light is a retrieval-augmented generation framework that compresses encoder outputs to efficiently transfer information while preserving retrieval coverage.
• It introduces a source pointer re-ranking mechanism to enhance passage precision, yielding improved KILT scores on benchmarks like TriviaQA and FEVER.
• Empirical evaluations demonstrate that FiD-Light achieves state-of-the-art effectiveness with significantly reduced latency compared to traditional FiD models.

FiD-Light is a retrieval-augmented generation architecture designed for efficient, effective text generation with provenance grounded in external knowledge bases. It builds on the Fusion-in-Decoder (FiD) framework by introducing a compression approach for encoder–decoder information transfer, coupled with a re-ranking strategy for source pointer interpretation. FiD-Light establishes superior efficiency and state-of-the-art effectiveness on diverse knowledge-intensive language tasks, particularly those in the KILT benchmark suite (Hofstätter et al., 2022).

1. Architectural Overview

FiD-Light modifies the FiD approach (Izacard & Grave, 2020), which retrieves $n$ passages $\{p_1, \ldots, p_n\}$ for a query $q$ and encodes each with a T5 encoder, yielding $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$. The sequence $e_i$ has length $|q| + |p_i|$. In FiD, all $e_i$ are concatenated into $[e_1;\dots;e_n]$ and passed to the T5 decoder, producing $o = T_D([e_1; \ldots; e_n])$. This mechanism often generates extremely long decoder input sequences, leading to high computational overhead.

FiD-Light introduces a compression function $f_k$ that selects $\{p_1, \ldots, p_n\}$0 token vectors from each $\{p_1, \ldots, p_n\}$1, typically the first $\{p_1, \ldots, p_n\}$2. Concatenating these compressed vectors across all passages, the decoder input becomes $\{p_1, \ldots, p_n\}$3. This reduces input length to $\{p_1, \ldots, p_n\}$4, while still exposing all retrieved passages to the decoder. The model thus sharply reduces the quadratic (or greater) self-attention cost in the decoder, without compromising retrieval coverage.

2. Mathematical Formulation and Complexity

Let $\{p_1, \ldots, p_n\}$5 be the number of passages per query, $\{p_1, \ldots, p_n\}$6 the query length, $\{p_1, \ldots, p_n\}$7 the passage length, and $\{p_1, \ldots, p_n\}$8 the per-passage compression size. The decoder input length in FiD is

$\{p_1, \ldots, p_n\}$9

while in FiD-Light it is

$q$0

with $q$1. Decoder layer cost per step is $q$2, where $q$3 is the hidden dimension, making FiD-Light’s decoder complexity $q$4 compared to FiD’s $q$5. This yields dramatic reductions in inference cost and runtime.

3. Efficiency Gains and Latency Benchmarking

In empirical evaluations using T5-Base and $q$6 passages on a TPUv4, FiD spends 93% of a 250 ms per-query budget in the decoder (230 ms), with encoder latency at 20 ms. Even halving retrieved passages, the decoder dominates latency. FiD-Light with $q$7 matches encoder and decoder time at 20 ms each (total 40 ms). Notably, FiD-Light at $q$8 outpaces FiD at $q$9 in overall latency, while processing four times as many passages.

In full sweeps varying $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$0 for FiD-Light and passage count for FiD across models T5-Base/Large/XL, FiD-Light variants at $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$1 continually form a better Pareto frontier for query latency versus end-task effectiveness on six out of seven KILT tasks.

4. Source Pointer Re-Ranking Mechanism

Source pointer mechanisms in FiD-Ex (Lakhotia et al., 2021) prepend textual markers to passages, prompting the model to output passage indices as pointers. However, generated indices often fail to match the number of relevant sources, degrading R-Precision. FiD-Light introduces the "FiD-Light$$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$2" variant, treating the sequence of output indices as a re-ranking signal rather than a strict explanation.

During inference, the output predicted set of indices $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$3 is parsed from the generated text. The final passage ranking $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$4 is constructed:

$$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$5

where $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$6. This permutation moves cited passages to the top, thereby improving passage/document R-Precision. Ablation tests confirm that FiD-Light-Ex (no re-ranking) underperforms FiD-Light$$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$7, especially on TriviaQA (+5 percentage points) and FEVER (+2 points).

5. Experimental Setup Across KILT Tasks

FiD-Light is benchmarked on seven tasks from the KILT suite:

• Open-domain QA: HotpotQA, TriviaQA, Natural Questions
• Slot filling: T-REx, zsRE
• Fact verification: FEVER
• Dialogue: Wizard of Wikipedia (WoW)

Retrieval employs a GTR-Base dense retriever pretrained on MS-MARCO, fine-tuned on KILT. Generation uses multi-task fine-tuning of T5 v1.1 in T5X with task markers. Default hyperparameters include an input cap of 384 tokens per (query+passage), 64-token maximum outputs, batch size of $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$8 passages, learning rate $$e_i = T_E([\text{``query: ''}; q; \text{``context: ''}; p_i])$$9 ($e_i$0 for T5-XL), and beam size 4. Key evaluation metrics are text matching (Exact Match, Accuracy, F1), R-Precision on retrieved provenance, and the combined KILT-score

$e_i$1

6. Empirical Results and Comparative Effectiveness

Empirical results demonstrate:

• Retriever Impact: KILT-fine-tuned GTR yields $e_i$2 to $e_i$3 KILT-score points uplift versus zero-shot. Oracle retrieval (with all gold passages) suggests further gains ($e_i$4 to $e_i$5).
• Re-Ranking Efficacy: Re-ranking improves passage/document precision over generation-only pointer use, most notably on TriviaQA and FEVER.
• Latency-Effectiveness Trade-off: FiD-Light$e_i$6 with $e_i$7 and T5-XL matches or exceeds FiD-Base at $e_i$8 for six tasks, with less than half the latency.
• Leaderboard Position: FiD-Light$e_i$9 achieves new state-of-the-art combined KILT-scores on six of seven tasks, outperforming prior systems (e.g., HotpotQA: +11.1 K-EM, 61.3% over previous; NQ: +7.5 K-EM, 17.2%; zsRE: +10.8 K-AC, 14.8%) while being $|q| + |p_i|$0–$|q| + |p_i|$1 faster than FiD-Base-40.

Model	HotpotQA	NQ	TriviaQA	zsRE	T-REx	FEVER	Latency Rel. FiD
FiD-Light$|q| + |p_i|$2 (T5-XL, k=8)	+11.1	+7.5	+5.8	+10.8	+0.5	+6.0	2–3× faster

7. Implications, Trade-Offs, and Directions

FiD-Light constrains encoder-to-decoder information flow by compressing each passage’s output, reducing computational demand without removing passages from consideration. The reduction in effectiveness per example is minor and can be offset by using larger backbone models. Treating source pointers as re-ranking signals rather than strict explanations robustly improves top-$|q| + |p_i|$3 passage precision and thus the joint retrieval-generation metric. FiD-Light$|q| + |p_i|$4 establishes a new Pareto frontier in effectiveness and efficiency across KILT, indicating that architectural optimization and lightweight post-processing suffice to realize fast, accurate retrieval-augmented generation systems without the classical trade-off between speed and performance (Hofstätter et al., 2022).