Deferred NAM: Efficient ASR Context Biasing

• The paper introduces a two-pass encoding strategy that decouples lightweight phrase selection from deferred full-context encoding to reduce computational cost.
• Deferred NAM scales efficiently by selecting top-k phrases for heavy processing, achieving sub-33 ms delay while handling tens of thousands of bias phrases.
• The method leverages auxiliary cross-entropy losses to enhance phrase retrieval and token alignment, leading to significant improvements in word error rate.

Deferred NAM is a low-latency context injection framework for non-streaming automatic speech recognition (ASR) that enables efficient large-scale contextual biasing. It introduces a two-pass context encoding strategy: a fast lightweight phrase selection phase followed by deferred processing of only the most relevant context items in a full-fidelity encoder. Deferred NAM allows scaling to tens of thousands of user- or application-specific phrases with sub-33 ms pre-decoding delay, achieving substantial inference speedups and improved word error rate (WER) relative to traditional attention-based methods (Wu et al., 2024).

1. Motivation and Problem Setting

Contextual biasing in end-to-end ASR enables transcribers to recognize infrequent or out-of-vocabulary terms—such as contact names or song titles—available at inference time but rare in the training corpus. Modern attention-based biasers typically comprise three components: a context encoder that converts each bias phrase or its wordpiece sequence into dense embeddings; a context filter that dynamically selects a relevant phrase subset (e.g., top-K attention); and a cross-attention mechanism to inject these phrase embeddings into the recognition pipeline. Standard fully end-to-end architectures require encoding all $N$ bias items before decoding can begin, leading to latencies on the order of hundreds of milliseconds or more when $N$ is large. This frontloads expensive computation onto the critical path and limits practical scaling to large context sets.

2. Deferred Context Encoding Methodology

Deferred NAM restructures the standard biaser pipeline by decoupling initial lightweight phrase selection from deferred full-context encoding. The procedure consists of:

1. Lightweight Phrase Selection: Each phrase $W_n$ is represented as a sequence of $L$ wordpiece embeddings. A Deep Averaging Network (DAN) computes low-dimensional phrase encodings $E^p_\text{light}[n]$ via a shallow, efficient network with a few hundred thousand parameters, running in $O(N L d)$ time.
2. Phrase Retrieval: Global phrase attention computes scores $z^p \in \mathbb{R}^{1+N}$ between the audio query representation $x^q$ and all $E^p_\text{light}$, utilizing a NO_BIAS token.
3. Top-K Pruning: The top-$k$ phrase indices $I^p$ are selected for further high-fidelity processing.
4. Deferred Full Encoding: Only phrases in $I^p$ undergo context encoding in a full Conformer stack $E^w[i,j]$.
5. Bias Context Application: Standard wordpiece-level cross-attention fuses $x^q$ and the $k$ selected phrase embeddings, producing a bias context $c^w$ injected into the decoder as $x^\text{biased} = x + \lambda c^w$.

The following pseudocode summarizes the workflow:

x^q ← AudioEncoder(A) # O(|A|) for n in 1..N: # Lightweight encoding E^p_light[n] ← DAN_stopgrad(W_n) # O(N·L·d_small) z^p ← GlobalPhraseAttention(x^q, E^p_light) I^p ← TopK(z^p[2:], k) # O(N + k·log N) for i in I^p: # Deferred heavy encoding E^w_i ← ContextEncoder(W_i) # O(k·L·C), C = heavy model cost c^w ← WPAttention(x^q, {E^w_i}_{i∈I^p}) Return x + λ·c^w

3. Complexity Analysis

Let $C$ denote the per-phrase cost for the full context encoder (“heavy encoder”) and $c \ll C$ the per-phrase cost of the DAN light encoder. The time complexity comparison is as follows:

Biasing Method	Pre-Decoding Latency
Standard (all N)	$O(NC)$
Deferred NAM	$O(Nc + kC)$

With $k \ll N$ and $c \ll C$, Deferred NAM reduces the pre-decoding context encoding cost by approximately $N/k$ ($3\,000 \to 32$ yields a $\sim 100\times$ reduction in heavy work), shifting the bulk of computation away from expensive model invocations (Wu et al., 2024).

4. Training Objectives and Loss Functions

Deferred NAM supplements the base RNN-T loss ($L_\text{ASR}$) with two auxiliary cross-entropy (CE) losses:

• Phrase-Level Cross-Entropy ($L_p$): Derived from multi-head logits $z^p \in \mathbb{R}^{1+N}$, with target labels $y^p$ assigning $1$ to the NO_BIAS index and any phrase that is a longest substring of the ground-truth transcript. $$L_p = -\sum_{i=0}^{N} y_i^p \log \operatorname{softmax}_i(z^p)$$.
• Wordpiece-Level Cross-Entropy ($L_w$): Computed from averaged logits $\bar z^w$ derived from per-token WP logits $z^w$, reinforcing phrase retrieval quality and token-level alignment.
• Total Loss: $$L_\text{total} = L_\text{ASR} + \lambda_p L_p + \lambda_w L_w$$, with $\lambda_p = \lambda_w = 0.1$ in best-performing models.

This formulation enhances both phrase retrieval effectiveness and fine-grained cross-attention, leading to measurable accuracy gains (Wu et al., 2024).

5. Empirical Performance

Model Architectures

• Baseline: Dual-mode NAM with either a 3-layer or 1-layer Conformer context encoder (B1, B2).
• Deferred NAM: 4-layer DAN ($0.8$M params) for lightweight encoding plus a 1-layer Conformer for the deferred phase.

Word Error Rate (WER)

Average WER (%) on in-context test sets for $k^p = 32$:

Experiment	ANTI	WO_PREFIX	W_PREFIX
B1	2.3	3.0	2.4
D1	1.9	2.6	1.9
D2	1.8	2.3	1.8
D3	1.8	2.0	1.5

D3 attains a 37.5% relative WER reduction over B1 on W_PREFIX ($2.4 \to 1.5$) (Wu et al., 2024).

Latency

Pre-decoding delay at $k^p = 32$:

#Phrases	QueryEnc	LightEnc	PhraseAttn	ContextEnc	WPAttn	Total
3 K	2.3 ms	3.5 ms	0.9 ms	1.3 ms	0.7 ms	8.7 ms
20 K	2.3 ms	22.8 ms	5.2 ms	1.3 ms	0.7 ms	32.3 ms

Compared to dual-mode NAM (B2: 520 ms at 20 K, B1: 1549 ms), Deferred NAM achieves $8.3\times$ to $16.1\times$ speedup (Wu et al., 2024).

6. Discussion and Implications

Deferred encoding scales context biasing to 20,000 phrases with $<33$ ms delay by limiting heavy computation to $k$ selected phrases. The linear growth in pre-decoding delay with $N$ affects only the lightweight encoder, while deferred heavy encoding grows with $k$ alone. This approach also reduces memory footprint since only $k$ wordpiece embeddings are materialized per utterance. Enhanced recognition accuracy is attributed to supervised phrase retrieval (CE-PA) and wordpiece gating (CE-WA). However, small $k$ risks omitting relevant rare phrases if retrieval fails, highlighting a trade-off between latency and recall (Wu et al., 2024).

7. Future Directions

Potential directions for Deferred NAM include adaptive $k$ selection, lighter pre-filters such as learned hashing, streaming extensions, and stricter on-device privacy constraints. Two-pass encoding and auxiliary CE losses on phrase and wordpiece attention together suggest an effective strategy for large-scale, low-latency ASR contextualization (Wu et al., 2024).