---
title: 'FiDO: Fusion-in-Decoder Optimized'
url: https://www.emergentmind.com/topics/fido-fusion-in-decoder-optimized
type: topic
---

# FiDO: Fusion-in-Decoder Optimized

Fusion-in-Decoder Optimized (FiDO) encompasses a set of architectural and algorithmic innovations that significantly accelerate and strengthen the Fusion-in-Decoder (FiD) paradigm for retrieval-augmented generation. FiDO variants target the inefficiencies intrinsic to naive FiD—where memory and compute overheads in the decoder dominate at inference—by restructuring cross-attention, redistributing computational resources, and introducing dynamic token/pruning strategies without sacrificing answer accuracy. These advances manifest in both architectural overhauls (notably [2212.08153]) and token-level inference techniques ([2310.13682]), collectively constituting the FiDO family.

## 1. Motivation and Limitations of Vanilla FiD

The original FiD formulation concatenates the outputs from a parallel encoder applied independently to each retrieved passage (together with the question) and uses a T5-based decoder to generate answers via cross-attention over this large representation set [2212.08153]. That design, largely inherited from standard T5, front-loads the majority of floating-point operations (FLOPs) into the encoder, despite empirical findings that inference latency is dominated by decoder-side cross-attention, especially for long-form outputs and large numbers of retrieved passages.

Memory constraints and memory bandwidth, not just FLOPs, become critical: in vanilla FiD, every decoding step incurs cross-attention over $K \times L$ encoder tokens (where $K$ passages, each of length $L$, are fed), rapidly exhausting both memory and compute resources as $K$ and $L$ grow. In this setting, performance scaling is suboptimal since most retrieved tokens are irrelevant to final answer selection, yet equally weighted in terms of decoder computation.

## 2. Architectural Innovations in FiDO

FiDO introduces two architectural modifications that jointly reduce decoder memory bandwidth requirements, shift model capacity to the decoder, and enable up to 7x faster inference without loss in generative quality [2212.08153]:

- **Layer-Sparse Cross-Attention (LSA):** Instead of executing full cross-attention in every decoder layer, FiDO sparsifies this operation, restricting it to every $K$-th layer (e.g., layers $\{6,12,18,\dots\}$ in a 24-layer decoder). All other layers perform only self-attention and MLP computation, removing redundant memory traffic and FLOPs.

- **Multi-Query Cross-Attention (MQ):** Standard T5 decoders utilize multi-head attention with independent keys/values per head, inflating key/value cache sizes (and memory traffic) by the number of heads $h$. FiDO collapses these into a single shared key/value per head (multi-query attention), reducing memory and cache pressure by almost $1/h$.

- **Asymmetric Model Scaling:** The combined efficiency gains of LSA and MQ permit architectural rebalancing towards a smaller encoder (fewer layers) and a much larger decoder (many more layers than FiD). For instance, FiDO-Base-XL uses a 12-layer encoder and 48-layer decoder, whereas a model with 24 encoder layers and 96 decoder layers (FiDO-Large-XXL) outperforms FiD-Large and decodes faster than FiD-Base.

- **FLOPs and Memory Bandwidth Allocation:** These modifications invert the encoder:decoder compute ratio (from 6:1 to a more balanced allocation), and—critically—move the operational intensity of the decoder toward the regime where it is no longer memory bandwidth-bound, as confirmed by the roofline model analysis given in [2212.08153].

| Variant             | Decoder ms/sample (batch 24, base model) | Relative Speedup |
|---------------------|------------------------------------------|------------------|
| FiD (no change)     | 88.0                                     | 1×               |
| +LSA (K=6)          | 16.0                                     | 5.5×             |
| +MQ                 | 1.0                                      | 88×              |
| +XL Decoder         | 4.0                                      | 22×              |

## 3. Token- and Passage-Level Pruning for Inference Efficiency

Algorithmic approaches complementary to architectural changes target redundant attention computation at finer granularity:

- **Token Elimination (TE, "FiDO via Token Elimination")** [2310.13682]:
   - Leverages cross-attention scores at decoding step $t_0$ and layer $\ell_0$ to score the salience of each encoder token.
   - Prunes all but the top $p \cdot T$ tokens based on these scores (typically $p=0.3$ provides the best speedup-accuracy tradeoff), dramatically reducing the cross-attended set for all subsequent decoding steps.
   - No model retraining is needed; token elimination operates entirely at inference.
   - Speedups of up to 62.2% are reported (e.g., ELI5, MS MARCO, NQ), with negligible (sometimes positive) changes in ROUGE/accuracy.

- **Dynamic Passage and Layer Pruning:**
   - Results can be further amplified by combining token elimination with conditional layer skipping methods such as CALM [2310.13682].
   - Passage pruning (as in Multi-Granularity FiD, see below) trims the retrieved set based on relevance predictions, further minimizing cross-attention cost.

| Method              | Latency Reduction        | ROUGE-L Δ            | Dataset (FiD-Base) |
|---------------------|-------------------------|----------------------|--------------------|
| Token Elimination   | 48% (MS MARCO, p=30%)   | +0.42                | MS MARCO           |
| FiDO Comb (TE+CALM) | 62% (MS MARCO)          | +0.78                | MS MARCO           |

## 4. Variants Incorporating Reranking, Filtering, and Rationality

Several recent extensions of FiDO/FID frameworks incorporate fine-grained filtering and evidence selection mechanisms:

- **Rational Fusion-in-Decoder (RFiD)** [2305.17041]: Supplements FiD with a per-passage rationale classifier (binary: is a passage "causal" or "spurious"?). Rationale predictions affect both encoder representation (small auxiliary classifier head) and decoder fusion (additional rationale token), yielding +1.5 EM on NQ and +0.7 EM on TQA over FiD-Large. Encoder-only rationalization yields moderate gains, but full decoder guidance via rationale tokens is critical for maximal improvement.

- **Multi-Granularity Guided FiD (MGFiD)** [2404.02581]: Harmonizes passage-level reranking and fine-grained sentence classification in a multi-task setting.
   - Sentence-level evidence is pooled (max over predicted-evidential sentences), and the resulting "anchor" vector is injected into the decoder's "[BOS]" token.
   - Passage pruning based on reranking probability reduces the number of attended passages by up to 76% with <0.5 EM accuracy drop.
   - MGFiD sets new accuracy standards: +3.5 EM on NQ over vanilla FiD-KD, while also reducing cross-attention cost.

- **Knowledge Graph Infused FiD (KG-FiD)** [2110.04330]: Constructs question-specific graphs among retrieved passages using KG triples, processes embedding updates through GAT, and performs passage reranking in multiple stages, significantly filtering out noisy information and reducing the computation required by the encoder/decoder.

## 5. Quantitative Performance Analysis

Performance is consistently reported in both inference speed and answer quality (EM, ROUGE-L). Key empirical findings include:

- **FiDO-Base-XL** offers faster inference than FiD-Base and higher accuracy than FiD-Large for a comparable budget [2212.08153].
- **Token elimination** at $p=30\%$ typically achieves 30–48% speedups, sometimes with small increases in ROUGE-L versus unpruned FiD [2310.13682].
- **MGFiD** sustains high accuracy (e.g., NQ EM 50.1 vs. 48.4 for vanilla FiD), and with passage pruning, maintains performance with a drastic reduction in decoding passage count (from 20 to ≈5 or 8 on NQ/TQA) [2404.02581].
- **KG-FiD** improves FiD EM by up to 1.5% (Natural Questions), achieving similar performance at 40% of the computational cost [2110.04330].
- **RFiD** yields consistent EM gains across model scales (FiD-base to FiD-large) and datasets [2305.17041].

## 6. Implementation, Tuning, and Practical Considerations

- **Training:** Many FiDO-related speedups (LSA, MQ, Token Elimination) do not require retraining; they can be applied at inference to vanilla FiD models [2212.08153, 2310.13682].
- **Hyperparameter tuning (token-level pruning):** Pruning ratio $p=0.3$ is robust for speedup/quality tradeoff; best pruning step $t_0$ is typically within first 15 generation steps; cross-attention scoring is most reliable in layers 2–4 [2310.13682].
- **Compatibility:** Most FiDO techniques are mutually compatible with FID augmentations (e.g., rationale signals, knowledge graph reranking, multi-granularity evidence heads).

## 7. Broader Impact and Future Directions

FiDO substantially improves the deployability of large retrieval-augmented models in latency- and cost-sensitive settings (e.g., production QA, conversational assistants) by decoupling generative performance from decoder bottlenecks. The simplicity of key methods (pruning via existing cross-attention weights, architectural sparsification) recommends them for routine adoption.

Potential extensions include dynamic or learned token/pruning schedules, integration with gradient- or reinforcement-based salience signals, and finer-grained (span-level, continuous) evidence scoring. Combining FiDO’s architectural changes with multi-task supervision (i.e., rationalization, multi-granularity evidence selection) defines a research direction for models that are both faster and more interpretably faithful to underlying facts [2212.08153, 2305.17041, 2404.02581, 2310.13682].

Source: https://www.emergentmind.com/topics/fido-fusion-in-decoder-optimized