---
title: Lattice Rescoring Protocols in ASR
url: https://www.emergentmind.com/topics/lattice-rescoring-protocols
type: topic
---

# Lattice Rescoring Protocols in ASR

Lattice rescoring protocols constitute a critical component of modern automatic speech recognition (ASR) pipelines, enabling improved hypothesis selection by incorporating more flexible or powerful language models (LMs) in a second pass over decoding lattices. These protocols allow integration of advanced neural or context-enhanced LMs, minimize computational and memory costs during first-pass decoding, and facilitate rich USE of supplementary data including large-scale text corpora or external context. The following article surveys and synthesizes major algorithmic frameworks, mathematical formulations, and empirical trade-offs underpinning lattice rescoring, with particular emphasis on recent innovations in low-resource ASR, neural network integration, contextualization, and computational optimization.

## 1. Foundations of Lattice Rescoring

Lattice rescoring is a two-pass inference strategy historically motivated by the need to balance efficiency and expressivity in ASR. In the first pass, a fast decoder (often an n-gram LM composed with a weighted finite-state transducer, WFST) produces a word lattice, a directed acyclic graph (DAG) where paths correspond to alternative hypotheses. Each arc is typically annotated with an acoustic likelihood and an LM (usually n-gram) score. In the second pass, the lattice arcs are re-evaluated using a more expressive LM, such as an LSTM or Transformer, and a new best path is selected. The general objective is  
$$
\hat W = \arg\max_W P(O | W) P(W)
$$
where $O$ is the observed acoustic sequence and $P(W)$ is updated in the second pass via rescoring. The push-forward algorithm and its variants are canonical mechanisms for integrating neural LMs into lattice rescoring, maintaining per-node histories for efficient dynamic programming over the lattice structure [1711.05448].

## 2. Minimally Augmented Lattice Rescoring in Low-Resource Settings

Significant challenges arise in low-resource ASR, where the baseline LM, typically trained on scarce transcripts, cannot generate lattices inclusive of plausible out-of-vocabulary (OOV) words. "Initial Decoding with Minimally Augmented Language Model for Improved Lattice Rescoring in Low Resource ASR" introduces a two-pass protocol that efficiently injects missing unigram probabilities for OOV terms derived from a large external text corpus, such as Wikipedia [2403.10937]. The workflow is as follows:

1. **OOT-word selection:**  
   - Let $V_{\mathrm{base}}$ be the baseline LM vocabulary and $V_{\mathrm{wiki}}$ the external text vocabulary. The out-of-train set $OOT = V_{\mathrm{wiki}} \setminus V_{\mathrm{base}}$ is determined.
   - Unigram counts $c_{\mathrm{wiki}}(w)$ are extracted for $w \in OOT$.

2. **OWALM construction:**  
   - Interpolate the baseline $n$-gram LM and the OOT unigram LM:
     $$
     P_{\mathrm{OWALM}}(w) = \lambda\, P_{\mathrm{base}}(w) + (1-\lambda)\, \frac{c_{\mathrm{wiki}}(w)}{\sum_{u \in OOT} c_{\mathrm{wiki}}(u)}
     $$
     with $\lambda$ tuned on held-out data.

3. **Initial decoding:**  
   - ASR decoding proceeds using $G_{\mathrm{OWALM}}$, yielding richer lattices containing more alternate hypotheses, including previously OOV words.

4. **Rescoring:**  
   - The lattices are rescored using a full n-gram LM trained on all available data (baseline + Wikipedia).
   - A linear score interpolation is applied:
     $$
     \mathrm{score}_{\mathrm{rescore}}(h) = \log P_{\mathrm{acoustic}}(h) + \alpha \log P_{\mathrm{full}}(h) + (1-\alpha) \log P_{\mathrm{OWALM}}(h)
     $$
     Typically, $\alpha=1$ is used.

This protocol achieves relative WER reductions of 21.8% (Telugu) and 41.8% (Kannada), nearly matching single-pass decoding with a full-augmented LM, but at 1/8 of the memory cost [2403.10937].

## 3. Advanced Neural Lattice Rescoring Methods

The application of neural LMs—LSTM, Transformer, or hybrid architectures—within lattice rescoring protocols leverages their superior modeling of longer-range and contextual dependencies. A canonical framework [1711.05448, 2312.12764] comprises:

- **Push-forward algorithm:**  
  Maintains $k$ best hypotheses at each node, updating LSTM/Transformer states along lattice arcs and pruning by cost. For LSTMs, each arc $a$ is rescored as $-\log P_{\mathrm{LSTM}}(w_a \mid \mathrm{history})$.
- **State-pooling and arc-beam:**  
  Pool predecessor states (weighted by, e.g., max-prob or posterior) or, for each arc, select the best predecessor; both trade off between accuracy and computation.

An iterative ensemble protocol [2312.12764] applies multiple NLMs (e.g., forward/backward LSTM and Transformer) sequentially by convexly interpolating their scores at each pass:
$$
L_{\mathrm{final}}(a) = \sum_{i=0}^I \gamma_i\, \log p^i(w_t | \cdot)
$$
where $\gamma_i=1/(I+1)$, ensuring equal weighting across $I$ NLMs plus the base model. Context carry-over (using previous lattice's final hidden state) further enhances performance in continuous speech domains.

Empirical findings demonstrate up to 24.4% WER reduction over 1-best baselines using an 8-model NLM ensemble with context in long-form ASR [2312.12764]. Self-normalization and product quantization enable efficiency and compression for large neural LMs [1711.05448].

## 4. Contextual, Semantic, and Metadata-Driven Extensions

Modern protocols increasingly extend lattice rescoring with contextual adaptation. Proposed directions include:

- **Context-aware RNNLM rescoring:**  
  Lattices from adjacent utterances in conversational speech are concatenated via tagged arcs (e.g., sentence boundary, speaker ID, intent), enabling RNNLMs to model long-range dependencies. Selective connection based on tf-idf similarity ensures only topically relevant contexts propagate, reducing character error rates by up to 13.1% [2011.09301].
- **Attention and pointer-based integration of external metadata:**  
  Lattice rescoring models may attend to or copy from video metadata, such as titles or descriptions, with an attention or hybrid pointer network LM [2005.07394]. Such models interpolate standard generation and pointer distributions for each lattice arc, yielding optimal performance when the external context is rich in rare or informative tokens.
- **Semantic and Transformer-based rescoring:**  
  Transformer LMs trained with cross-entropy are used to rescore compacted or n-best paths, combining acoustic, n-gram, and semantic model log-probabilities with tunable weights [2310.09680].

## 5. Computational and Memory-Efficiency Innovations

Lattice rescoring poses non-trivial computational burdens, particularly with large neural LMs or in resource-constrained environments. Several strategies have been presented:

- **Single-shot non-autoregressive lattice rescoring:**  
  The LT-LM approach processes the entire lattice in parallel using a non-autoregressive Transformer encoder, outputting scores for all arcs simultaneously. Artificial training lattices are synthesized from large-scale text to augment data scarcity. LT-LM yields 300–600× speedup over streaming autoregressive rescoring, with only ~0.4% absolute WER gap on LibriSpeech [2104.02526].
- **Parallel path-based GPU rescoring:**  
  Path-cover strategies convert expanded lattices into a minimal set of full paths, batch-score these using neural LMs (e.g., in PyTorch), and integrate results back into the lattice, achieving significant runtime reductions [2103.05081].
- **Memory-efficient lattice graph construction:**  
  In low-resource protocols [2403.10937], minimally augmented LMs inject only missing unigrams, reducing HCLG memory usage by 8× while preserving almost the entire WER gain of a full augmented LM.
- **Efficient state-merging and context-carrying for LLM rescoring:**  
  Lattice arcs are merged by context, converting tree outputs into compact DAGs, and 1-best context is prepended across segments for LLMs. This protocol enables practical LLM rescoring in long-form and code-switched tasks [2306.08133].

## 6. Discriminative and Hybrid Training Objectives

Integrating discriminative losses, as in MWER or MWED, into lattice rescoring allows models to optimize directly for sequence-level error metrics. RescoreBERT [2202.01094] demonstrates the following:

- Fine-tune a BERT model via MWER/MWED loss over n-best hypotheses, optionally fusing an MLM-based distillation pretraining target.
- Compose final score per hypothesis as $s_i = s^a_i + \beta s^l_i$.
- Achieve 6.6%/3.4% relative WER gains on LibriSpeech test sets compared to non-discriminative BERT, at competitive inference latencies.

This approach demonstrates the complementarity of deep bidirectional LMs and discriminative objectives for second-pass rescoring.

## 7. Rescoring Beyond ASR: Applications in Lattice Codes

Although the term "lattice rescoring" most commonly refers to ASR, analogous protocols arise in finite-dimensional lattice codes for communications [2501.04307]. Here, "rescoring" is implemented by multi-trial decoding using embedded error-detection (CRC) and per-trial optimization of decoding coefficients. Retry decoding achieves up to 1.31 dB coding gain for two-user compute-forward relays at $10^{-5}$ error rate, all at minimal extra decoding cost.

---

**References**:  
[2403.10937], [1711.05448], [2312.12764], [2011.09301], [2005.07394], [2310.09680], [2202.01094], [2104.02526], [2103.05081], [2306.08133], [2501.04307], [1907.01030]

Source: https://www.emergentmind.com/topics/lattice-rescoring-protocols