FiDO: Fusion-in-Decoder Optimized

• The paper introduces architectural modifications such as Layer-Sparse and Multi-Query cross-attention that accelerate inference by up to 7x without sacrificing answer accuracy.
• FiDO employs asymmetric model scaling by reallocating computational resources from the encoder to a deeper decoder, effectively reducing memory and compute overhead.
• Dynamic token elimination and passage pruning significantly cut down redundant computations, ensuring efficient use of memory bandwidth while maintaining or improving performance metrics.

Fusion-in-Decoder Optimized (FiDO) encompasses a set of architectural and algorithmic innovations that significantly accelerate and strengthen the Fusion-in-Decoder (FiD) paradigm for retrieval-augmented generation. FiDO variants target the inefficiencies intrinsic to naive FiD—where memory and compute overheads in the decoder dominate at inference—by restructuring cross-attention, redistributing computational resources, and introducing dynamic token/pruning strategies without sacrificing answer accuracy. These advances manifest in both architectural overhauls (notably (Jong et al., 2022)) and token-level inference techniques (Berchansky et al., 2023), collectively constituting the FiDO family.

1. Motivation and Limitations of Vanilla FiD

The original FiD formulation concatenates the outputs from a parallel encoder applied independently to each retrieved passage (together with the question) and uses a T5-based decoder to generate answers via cross-attention over this large representation set (Jong et al., 2022). That design, largely inherited from standard T5, front-loads the majority of floating-point operations (FLOPs) into the encoder, despite empirical findings that inference latency is dominated by decoder-side cross-attention, especially for long-form outputs and large numbers of retrieved passages.

Memory constraints and memory bandwidth, not just FLOPs, become critical: in vanilla FiD, every decoding step incurs cross-attention over $K \times L$ encoder tokens (where $K$ passages, each of length $L$, are fed), rapidly exhausting both memory and compute resources as $K$ and $L$ grow. In this setting, performance scaling is suboptimal since most retrieved tokens are irrelevant to final answer selection, yet equally weighted in terms of decoder computation.

2. Architectural Innovations in FiDO

FiDO introduces two architectural modifications that jointly reduce decoder memory bandwidth requirements, shift model capacity to the decoder, and enable up to 7x faster inference without loss in generative quality (Jong et al., 2022):

• Layer-Sparse Cross-Attention (LSA): Instead of executing full cross-attention in every decoder layer, FiDO sparsifies this operation, restricting it to every $K$-th layer (e.g., layers $\{6,12,18,\dots\}$ in a 24-layer decoder). All other layers perform only self-attention and MLP computation, removing redundant memory traffic and FLOPs.
• Multi-Query Cross-Attention (MQ): Standard T5 decoders utilize multi-head attention with independent keys/values per head, inflating key/value cache sizes (and memory traffic) by the number of heads $h$. FiDO collapses these into a single shared key/value per head (multi-query attention), reducing memory and cache pressure by almost $1/h$.
• Asymmetric Model Scaling: The combined efficiency gains of LSA and MQ permit architectural rebalancing towards a smaller encoder (fewer layers) and a much larger decoder (many more layers than FiD). For instance, FiDO-Base-XL uses a 12-layer encoder and 48-layer decoder, whereas a model with 24 encoder layers and 96 decoder layers (FiDO-Large-XXL) outperforms FiD-Large and decodes faster than FiD-Base.
• FLOPs and Memory Bandwidth Allocation: These modifications invert the encoder:decoder compute ratio (from 6:1 to a more balanced allocation), and—critically—move the operational intensity of the decoder toward the regime where it is no longer memory bandwidth-bound, as confirmed by the roofline model analysis given in (Jong et al., 2022).

Variant	Decoder ms/sample (batch 24, base model)	Relative Speedup
FiD (no change)	88.0	1×
+LSA (K=6)	16.0	5.5×
+MQ	1.0	88×
+XL Decoder	4.0	22×

3. Token- and Passage-Level Pruning for Inference Efficiency

Algorithmic approaches complementary to architectural changes target redundant attention computation at finer granularity:

• Token Elimination (TE, "FiDO via Token Elimination") (Berchansky et al., 2023):
  • Leverages cross-attention scores at decoding step $t_0$ and layer $K$0 to score the salience of each encoder token.
  • Prunes all but the top $K$1 tokens based on these scores (typically $K$2 provides the best speedup-accuracy tradeoff), dramatically reducing the cross-attended set for all subsequent decoding steps.
  • No model retraining is needed; token elimination operates entirely at inference.
  • Speedups of up to 62.2% are reported (e.g., ELI5, MS MARCO, NQ), with negligible (sometimes positive) changes in ROUGE/accuracy.
• Dynamic Passage and Layer Pruning:
  • Results can be further amplified by combining token elimination with conditional layer skipping methods such as CALM (Berchansky et al., 2023).
  • Passage pruning (as in Multi-Granularity FiD, see below) trims the retrieved set based on relevance predictions, further minimizing cross-attention cost.

Method	Latency Reduction	ROUGE-L Δ	Dataset (FiD-Base)
Token Elimination	48% (MS MARCO, p=30%)	+0.42	MS MARCO
FiDO Comb (TE+CALM)	62% (MS MARCO)	+0.78	MS MARCO

4. Variants Incorporating Reranking, Filtering, and Rationality

Several recent extensions of FiDO/FID frameworks incorporate fine-grained filtering and evidence selection mechanisms:

• Rational Fusion-in-Decoder (RFiD) (Wang et al., 2023): Supplements FiD with a per-passage rationale classifier (binary: is a passage "causal" or "spurious"?). Rationale predictions affect both encoder representation (small auxiliary classifier head) and decoder fusion (additional rationale token), yielding +1.5 EM on NQ and +0.7 EM on TQA over FiD-Large. Encoder-only rationalization yields moderate gains, but full decoder guidance via rationale tokens is critical for maximal improvement.
• Multi-Granularity Guided FiD (MGFiD) (Choi et al., 2024): Harmonizes passage-level reranking and fine-grained sentence classification in a multi-task setting.
  • Sentence-level evidence is pooled (max over predicted-evidential sentences), and the resulting "anchor" vector is injected into the decoder's "[BOS]" token.
  • Passage pruning based on reranking probability reduces the number of attended passages by up to 76% with <0.5 EM accuracy drop.
  • MGFiD sets new accuracy standards: +3.5 EM on NQ over vanilla FiD-KD, while also reducing cross-attention cost.
• Knowledge Graph Infused FiD (KG-FiD) (Yu et al., 2021): Constructs question-specific graphs among retrieved passages using KG triples, processes embedding updates through GAT, and performs passage reranking in multiple stages, significantly filtering out noisy information and reducing the computation required by the encoder/decoder.

5. Quantitative Performance Analysis

Performance is consistently reported in both inference speed and answer quality (EM, ROUGE-L). Key empirical findings include:

• FiDO-Base-XL offers faster inference than FiD-Base and higher accuracy than FiD-Large for a comparable budget (Jong et al., 2022).
• Token elimination at $K$3 typically achieves 30–48% speedups, sometimes with small increases in ROUGE-L versus unpruned FiD (Berchansky et al., 2023).
• MGFiD sustains high accuracy (e.g., NQ EM 50.1 vs. 48.4 for vanilla FiD), and with passage pruning, maintains performance with a drastic reduction in decoding passage count (from 20 to ≈5 or 8 on NQ/TQA) (Choi et al., 2024).
• KG-FiD improves FiD EM by up to 1.5% (Natural Questions), achieving similar performance at 40% of the computational cost (Yu et al., 2021).
• RFiD yields consistent EM gains across model scales (FiD-base to FiD-large) and datasets (Wang et al., 2023).

6. Implementation, Tuning, and Practical Considerations

• Training: Many FiDO-related speedups (LSA, MQ, Token Elimination) do not require retraining; they can be applied at inference to vanilla FiD models (Jong et al., 2022, Berchansky et al., 2023).
• Hyperparameter tuning (token-level pruning): Pruning ratio $K$4 is robust for speedup/quality tradeoff; best pruning step $K$5 is typically within first 15 generation steps; cross-attention scoring is most reliable in layers 2–4 (Berchansky et al., 2023).
• Compatibility: Most FiDO techniques are mutually compatible with FID augmentations (e.g., rationale signals, knowledge graph reranking, multi-granularity evidence heads).

7. Broader Impact and Future Directions

FiDO substantially improves the deployability of large retrieval-augmented models in latency- and cost-sensitive settings (e.g., production QA, conversational assistants) by decoupling generative performance from decoder bottlenecks. The simplicity of key methods (pruning via existing cross-attention weights, architectural sparsification) recommends them for routine adoption.

Potential extensions include dynamic or learned token/pruning schedules, integration with gradient- or reinforcement-based salience signals, and finer-grained (span-level, continuous) evidence scoring. Combining FiDO’s architectural changes with multi-task supervision (i.e., rationalization, multi-granularity evidence selection) defines a research direction for models that are both faster and more interpretably faithful to underlying facts (Jong et al., 2022, Wang et al., 2023, Choi et al., 2024, Berchansky et al., 2023).