Shallow Fusion in Sequence Models

• Shallow fusion is a technique that fuses auxiliary information—such as external language model scores or feature cues—during early inference or encoding to improve model performance.
• It is applied across domains like ASR, NMT, and multimodal systems by combining base model outputs with dynamic priors or early feature integration.
• The approach offers modularity by leveraging pre-trained components without retraining the core model, though it remains sensitive to hyperparameters and interface design.

Shallow fusion denotes a family of model-coupling strategies in which an auxiliary scorer or representation is introduced at a shallow stage of inference or encoding rather than by fully retraining a unified model. In the cited literature, the term most often refers to decoding-time combination of a primary sequence model with an external LLM or weighted finite-state transducer (WFST), but it also denotes early-stage representation fusion, such as concatenating domain keywords with text tokens or exchanging shallow cross-modal features before deeper processing (Stahlberg et al., 2018, Qiu et al., 2023).

1. Core definitions and scope

The term is not used uniformly across research areas. In sequence generation, shallow fusion is typically an inference-time procedure: a base model produces hypotheses, and an external LM or biasing structure contributes an additional score during beam search or reranking. In multimodal or knowledge-enhanced architectures, “shallow” instead refers to the location of interaction, namely at the input or early-feature level rather than in later layers (Cabrera et al., 2021, Dong et al., 2024).

Context	What is fused	Meaning of “shallow”
ASR / NMT / TN decoding	Base model score with LM or WFST score	Decoding-time or reranking-time combination
Knowledge-enhanced NLP	Sentence tokens with domain keywords	Input-level or early encoding fusion
Multimodal vision / medical imaging	RGB, IR, vesselness, or multidepth features	Early feature interaction before deeper fusion

In neural machine translation, shallow fusion is described as combining LM and NMT scores at inference time in a log-linear model (Stahlberg et al., 2018). In streaming RNNT speech recognition, it is likewise an inference-time language-model integration method, implemented as a log-linear combination of RNNT and LM scores (Cabrera et al., 2021). In adverse drug reaction detection, by contrast, shallow fusion occurs before the Transformer encoder layers by concatenating the original sentence and domain-keyword sequence into a single input sequence, so that self-attention can model interactions jointly (Qiu et al., 2023).

A common misconception is that shallow fusion always denotes late score interpolation. The literature does not support that restriction. Some works use the term for decoder-level score combination, whereas others use it for early token-level or feature-level interaction (Qiu et al., 2023, Lu et al., 14 Mar 2025).

2. Decoder-level shallow fusion in language generation

The canonical formulation in sequence generation is a log-linear decoding objective. In machine translation, the decoding rule is given as

$$\hat{\mathbf{y}}=\arg\max_{\mathbf{y}} \log P_{\mathrm{TM}}(\mathbf{y}\mid\mathbf{x})+\lambda \log P_{\mathrm{LM}}(\mathbf{y}),$$

where the translation model contributes source-conditioned adequacy and the LM contributes target-side fluency (Stahlberg et al., 2018). The same pattern appears in multilingual end-to-end ASR: $$y^*=\underset{y}{\arg\max}\,\big[\log p(y\mid x)+\lambda \log p_{\mathrm{LM}}(y)\big],$$ with the LM consulted at each beam-search step (Hu et al., 2023).

In streaming RNNT, the stepwise form is

$$\log P(y_t)=\log P_{\mathrm{rnnt}}(y_t)+\beta \log P_{\mathrm{lm}}(y_t).$$

A specific RNNT implementation detail is blank handling: because the LM does not naturally model the blank symbol, the LM probability of blank is defined to match the RNNT probability, and the LM state is not updated on blanks (Cabrera et al., 2021).

Text normalization uses a related but structurally distinct form. A non-deterministic WFST first enumerates normalization candidates, and a pretrained LM then picks the best candidate, which the authors explicitly frame as similar to shallow fusion in ASR (Bakhturina et al., 2022). The WFST side remains rule-based and safe, while the LM resolves contextual ambiguity such as whether 1/4 should be verbalized as one quarter or January fourth. This is still shallow in the sense that the neural component enters only at candidate selection time rather than through retraining the normalization model.

The main attraction of this decoder-level usage is modularity. A separately trained LM can be attached at test time, the underlying ASR or NMT model remains unchanged, and text-only corpora can be exploited without modifying paired-data training (Cabrera et al., 2021, Stahlberg et al., 2018).

3. ASR specializations: contextualization, rare words, backward models, and LLMs

Speech recognition work has diversified shallow fusion into several specialized forms. In multilingual ASR, a sparse mixture-of-experts LM based on GLaM is used for shallow fusion across up to 84 languages, with only the top 2 experts activated at each decoding step. On a 50-language evaluation, GLaM-64E yields an average WER of 8.22% versus 8.50% for the baseline 2B E2E model, improves 43 of 50 languages relative to the baseline, and improves 41 of 50 languages over a dense 140M Conformer LM with an average relative WER reduction of 3.85% (Hu et al., 2023). This establishes that shallow fusion can benefit from a much stronger multilingual text prior while keeping inference computation roughly comparable to a dense LM of similar active size.

Contextual biasing extends the same idea with structured dynamic priors. A streaming RNNT system uses WFST shallow fusion with a biasing graph built dynamically from an utterance-specific bias list and applied at the WordPiece level before pruning (Le et al., 2021). In that system, shallow fusion is one component of a broader contextualization stack that also includes trie-based deep biasing in both the RNNT and an external NNLM. The WFST component remains valuable because it is dynamic, modular, and robust when the bias list grows beyond what neural biasing modules saw during training.

Rare-word handling can also be implemented through an extremely lightweight shallow-fusion prior. Unigram shallow fusion replaces a full LM with a unigram FST that gives a fixed reward to selected rare words during decoding (Ravi et al., 2020). In the main experiments, rare words are selected from unigram counts with $n_{\mathrm{thresh}}=250$ and $\alpha=0.75$, yielding a unigram FST of about 1 MB for roughly 200k words. The reported result is a 3.8% relative WER improvement on the rare-word set with about a $-0.3\%$ relative effect on the general test set in the plain RNN-T case, and the gains are additive to second-pass neural LM rescoring (Ravi et al., 2020).

A different extension introduces a backward LM during first-pass decoding. Iterative shallow fusion recomputes backward-LM scores over partial hypotheses at each step, because the conditioning context of a backward LM changes as the hypothesis grows (Ogawa et al., 2023). On TED-LIUM2, standard SF-FLM gives 14.6 dev / 9.9 test WER, while combining forward shallow fusion with iterative shallow fusion using a partial sentence-aware backward LM yields the best reported result, 13.5 dev / 9.1 test at $I=2$ (Ogawa et al., 2023).

LLM-based variants focus on scale and tokenization mismatch. Generative Fusion Decoding maps token-level probabilities to a shared byte space so that an ASR or OCR model and an LLM can be fused even when their token spaces differ (Hsu et al., 2024). The framework is training-free and allows the LLM to participate in the sampling process step by step. The paper reports significant WER reductions of up to 17.7% in instruction-aware settings and improvements on both ASR and OCR benchmarks (Hsu et al., 2024). Delayed fusion addresses the same practical problem differently: it applies LLM scores to ASR hypotheses with a delay during decoding, after pruning, which reduces both the number of scored hypotheses and the number of LLM inference calls (Hori et al., 16 Jan 2025). In a CTC prefix beam-search comparison using OpenLLaMA 3B, delayed fusion on a 6K-vocabulary ASR model reaches 3.05 / 5.68 WER at 0.115 RTF, whereas shallow fusion on a separately retrained 32K-tokenizer ASR model gives 3.05 / 5.75 at 0.257 RTF, so delayed fusion is about 2.2× faster in that condition (Hori et al., 16 Jan 2025).

4. Limits of late combination and training-aware alternatives

Several papers treat shallow fusion as an effective baseline but not the endpoint. The central critique is that it is a late-combination method. In RNN-T personalization, the stated limitation is that “the model needs to have the potential to produce the correct output in the first place without access to biasing information,” which is problematic for rare names and spiky transducer decoding (Le et al., 2020). That work therefore introduces deep PLM fusion, where a personalized LM embedding conditions the joiner directly, and reports 15.4% relative WER improvement on name-prod and 34.5% relative WER improvement on name-rare compared to a strong RNN-T baseline that already uses shallow fusion and text-to-speech augmentation (Le et al., 2020).

Machine translation reaches a similar conclusion from a different angle. Shallow fusion in NMT often leads to minor improvements over the baseline for both single systems and ensembles, while the training-time “Simple Fusion” variants PreNorm and PostNorm outperform both shallow fusion and cold fusion on all language pairs, with test-set gains of up to +2.36 BLEU on top of ensembles without LM (Stahlberg et al., 2018). The underlying claim is that if the LM enters during training, the translation model can focus more of its capacity on source-conditioned residual information rather than rediscovering target fluency.

Streaming RNNT results also favor tighter integration. In one comparison across Norwegian, Greek, and Sinhala voice search, shallow fusion gives only 1.22%, 0.68%, and 0.14% relative WER improvement, whereas cold fusion works consistently better, with up to a 8.5% WER improvement (Cabrera et al., 2021). The paper attributes part of shallow fusion’s weakness to non-ideal blank handling and to the fact that the RNNT is not trained jointly with LM features.

Minimum WER training introduces another intermediate position. MWER-SF uses the shallow-fusion score both to generate N-best hypotheses and to compute the normalized posteriors inside the MWER loss (Meng et al., 2021). This reduces dependence on post-hoc LM-weight tuning, but it does not correct for the internal LM of the end-to-end recognizer. In the reported averages over six test sets, the baseline Transformer-Transducer yields 12.00% WER, MWER gives 11.43%, MWER-SF gives 11.07%, and MWER-ILME gives 10.43%, with MWER-ILME achieving an average 5.8% relative WER reduction from MWER-SF (Meng et al., 2021).

These results collectively indicate that shallow fusion is often strongest as a modular baseline or as part of a larger system. This suggests that its ceiling is influenced by how much the base model already exposes correct hypotheses before external scores are applied.

5. Shallow fusion as early representation-level interaction

In knowledge-enhanced and multimodal models, shallow fusion usually refers to early interaction in representation space rather than late score interpolation. In adverse drug reaction detection, KESDT extracts domain keywords from a MedDRA-derived domain dictionary containing 16,339 ADR-related words, uses a Trie to match keyword subsequences in the input sentence, and concatenates the original text and keyword sequence as

$$S=\{[\mathrm{CLS}],S_1,[\mathrm{SEP}],S_2,[\mathrm{SEP}]\}.$$

The fused embedding sequence is then passed to a BERT or BioBERT encoder, enabling self-attention across sentence words and domain keywords (Qiu et al., 2023). The authors argue that this allows the model to “fully exploit the interactive relationships between domain keywords and other words in the sentence.” Ablation removes the keyword component and lowers F1 from 87.82 to 87.56 on CADEC, from 68.63 to 68.37 on TwiMed, and from 72.20 to 72.00 on Twitter (Qiu et al., 2023).

Cross-modality detection and tracking papers use “shallow fusion” in a similarly architectural sense. Fusion-Mamba places shallow fusion in the State Space Channel Swapping module, which exchanges channel groups between RGB and IR features and then refines them with a VSS block before deeper hidden-state fusion (Dong et al., 2024). On FLIR-Aligned, removing this shallow-fusion stage drops performance from 84.9 to 82.9 on $mAP_{50}$ and from 47.0 to 45.9 on $mAP$ (Dong et al., 2024). In RGBT tracking, TPF argues that shallower fusion yields smaller modality gaps as measured by Maximum Mean Discrepancy, but naive pixel-level fusion remains under-discriminative and noise-sensitive (Lu et al., 14 Mar 2025). Its answer is a learned Pixel-level Fusion Adapter with three ViM blocks and two convolution layers, trained through adaptive multi-expert distillation and decoupled representation fine-tuning; the PFA is reported to have only 14.3 KB parameters and to reach 201.8 FPS in image fusion (Lu et al., 14 Mar 2025). The paper explicitly states that shallow fusion is not inherently weak; the problem is that naive shallow interaction cannot yet separate task-relevant evidence from modality-specific noise.

Medical-image segmentation offers another variation. HarmonySeg treats vesselness maps as an auxiliary modality and couples them with image features through a Shallow and Deep Fusion module composed of Deep Mutual Query and Shallow Query (Huang et al., 10 Apr 2025). A simple concatenation baseline, denoted VS$$y^*=\underset{y}{\arg\max}\,\big[\log p(y\mid x)+\lambda \log p_{\mathrm{LM}}(y)\big],$$0, is essentially unchanged from the base model on the cited ablation, at 60.07% Dice versus 60.15%, whereas adding SADF with deep-to-shallow decoding raises Dice to 64.20% and clDice to 73.08%; the full model with the growth-suppression balanced loss reaches 66.79% Dice and 72.04% clDice on HVS (Huang et al., 10 Apr 2025).

PolSAR classification uses the phrase more literally as shallow-to-deep feature fusion. SDF2Net extracts shallow, medium, and deep complex-valued CNN features in three parallel branches, concatenates them channel-wise, and applies attention after fusion (Alkhatib et al., 2024). On Flevoland, attention after fusion yields OA $$y^*=\underset{y}{\arg\max}\,\big[\log p(y\mid x)+\lambda \log p_{\mathrm{LM}}(y)\big],$$1, compared with 95.84% for attention before fusion and 95.14% without attention; the three-branch fusion also outperforms any single branch or two-branch combination (Alkhatib et al., 2024).

6. Empirical behavior, trade-offs, and recurring misconceptions

Across the cited literature, the main advantage of shallow fusion is modularity. External LMs, WFSTs, personalized lists, or auxiliary modalities can be introduced without retraining the core model, and dynamic context can be incorporated at inference time (Le et al., 2021, Hsu et al., 2024). This is especially attractive when paired training data are scarce, when context changes per utterance, or when pretrained LMs are much stronger than task-specific text models.

Its empirical behavior is heterogeneous. In NMT and some streaming RNNT settings, gains are small or inconsistent: shallow fusion improves English–Turkish BLEU from 11.56 to 11.61 and Estonian–English from 16.57 to 16.57, while RNNT voice-search experiments report only marginal relative WER improvements in three languages (Stahlberg et al., 2018, Cabrera et al., 2021). By contrast, shallow fusion becomes markedly stronger when the external prior is better aligned with the failure mode: multilingual GLaM improves 43 of 50 languages over the baseline ASR model, WFST contextualization materially improves rare long-tail words, and unigram shallow fusion corrects rare-word underestimation in RNN-T decoding (Hu et al., 2023, Le et al., 2021, Ravi et al., 2020). This suggests that the method is most effective when it supplies information that the base search process systematically underweights but can still recover before pruning.

Several limitations recur. Decoder-level shallow fusion is sensitive to LM weights and, in RNNT, to blank handling; it can also be computationally expensive with LLMs and awkward under vocabulary mismatch (Cabrera et al., 2021, Hori et al., 16 Jan 2025). In personalization and contextual biasing, late combination can fail when the correct prefix is pruned early or when the base model never assigns enough probability to the desired path (Le et al., 2020). In vision, shallow feature fusion can reduce modality discrepancy while still degrading task performance if it propagates background clutter or other task-irrelevant noise (Lu et al., 14 Mar 2025).

A final misconception is that “shallow” implies weak or simplistic modeling. The cited papers do not support that equation. Shallow fusion may be a plain log-linear interpolation, a dynamic WFST prior, a byte-level LLM coupling mechanism, an input-level keyword injection scheme, or an early cross-modal adapter (Bakhturina et al., 2022, Hsu et al., 2024, Qiu et al., 2023). What unifies these uses is not a single architecture, but a shared design choice: auxiliary information is injected near the surface of search or representation, and the deeper model is left largely intact.