---
title: 'Deferred NAM: Efficient ASR Context Biasing'
url: https://www.emergentmind.com/topics/deferred-nam
type: topic
---

# Deferred NAM: Efficient ASR Context Biasing

Deferred NAM is a low-latency context injection framework for non-streaming automatic speech recognition (ASR) that enables efficient large-scale contextual biasing. It introduces a two-pass context encoding strategy: a fast lightweight phrase selection phase followed by deferred processing of only the most relevant context items in a full-fidelity encoder. Deferred NAM allows scaling to tens of thousands of user- or application-specific phrases with sub-33 ms pre-decoding delay, achieving substantial inference speedups and improved word error rate (WER) relative to traditional attention-based methods [2404.10180].

## 1. Motivation and Problem Setting

Contextual biasing in end-to-end ASR enables transcribers to recognize infrequent or out-of-vocabulary terms—such as contact names or song titles—available at inference time but rare in the training corpus. Modern attention-based biasers typically comprise three components: a context encoder that converts each bias phrase or its wordpiece sequence into dense embeddings; a context filter that dynamically selects a relevant phrase subset (e.g., top-K attention); and a cross-attention mechanism to inject these phrase embeddings into the recognition pipeline. Standard fully end-to-end architectures require encoding all $N$ bias items before decoding can begin, leading to latencies on the order of hundreds of milliseconds or more when $N$ is large. This frontloads expensive computation onto the critical path and limits practical scaling to large context sets.

## 2. Deferred Context Encoding Methodology

Deferred NAM restructures the standard biaser pipeline by decoupling initial lightweight phrase selection from deferred full-context encoding. The procedure consists of:

1. **Lightweight Phrase Selection:** Each phrase $W_n$ is represented as a sequence of $L$ wordpiece embeddings. A Deep Averaging Network (DAN) computes low-dimensional phrase encodings $E^p_\text{light}[n]$ via a shallow, efficient network with a few hundred thousand parameters, running in $O(N L d)$ time.
2. **Phrase Retrieval:** Global phrase attention computes scores $z^p \in \mathbb{R}^{1+N}$ between the audio query representation $x^q$ and all $E^p_\text{light}$, utilizing a NO_BIAS token.
3. **Top-K Pruning:** The top-$k$ phrase indices $I^p$ are selected for further high-fidelity processing.
4. **Deferred Full Encoding:** Only phrases in $I^p$ undergo context encoding in a full Conformer stack $E^w[i,j]$.
5. **Bias Context Application:** Standard wordpiece-level cross-attention fuses $x^q$ and the $k$ selected phrase embeddings, producing a bias context $c^w$ injected into the decoder as $x^\text{biased} = x + \lambda c^w$.

The following pseudocode summarizes the workflow:

```python
# Inputs:
# - Audio frames A
# - Bias phrases W = {W₁,…,W_N}, each W_n ∈ ℝ^{L×d}
# - Retrieval size k

x^q ← AudioEncoder(A)                   # O(|A|)
for n in 1..N:                          # Lightweight encoding
    E^p_light[n] ← DAN_stopgrad(W_n)    # O(N·L·d_small)
z^p ← GlobalPhraseAttention(x^q, E^p_light)
I^p ← TopK(z^p[2:], k)                  # O(N + k·log N)
for i in I^p:                           # Deferred heavy encoding
    E^w_i ← ContextEncoder(W_i)         # O(k·L·C), C = heavy model cost
c^w ← WPAttention(x^q, {E^w_i}_{i∈I^p})
Return x + λ·c^w
```

## 3. Complexity Analysis

Let $C$ denote the per-phrase cost for the full context encoder (“heavy encoder”) and $c \ll C$ the per-phrase cost of the DAN light encoder. The time complexity comparison is as follows:

| Biasing Method       | Pre-Decoding Latency  |
|----------------------|----------------------|
| Standard (all N)     | $O(NC)$              |
| Deferred NAM         | $O(Nc + kC)$         |

With $k \ll N$ and $c \ll C$, Deferred NAM reduces the pre-decoding context encoding cost by approximately $N/k$ ($3\,000 \to 32$ yields a $\sim 100\times$ reduction in heavy work), shifting the bulk of computation away from expensive model invocations [2404.10180].

## 4. Training Objectives and Loss Functions

Deferred NAM supplements the base RNN-T loss ($L_\text{ASR}$) with two auxiliary cross-entropy (CE) losses:

- **Phrase-Level Cross-Entropy ($L_p$):** Derived from multi-head logits $z^p \in \mathbb{R}^{1+N}$, with target labels $y^p$ assigning $1$ to the NO_BIAS index and any phrase that is a longest substring of the ground-truth transcript. $L_p = -\sum_{i=0}^{N} y_i^p \log \operatorname{softmax}_i(z^p)$.
- **Wordpiece-Level Cross-Entropy ($L_w$):** Computed from averaged logits $\bar z^w$ derived from per-token WP logits $z^w$, reinforcing phrase retrieval quality and token-level alignment.
- **Total Loss:** $L_\text{total} = L_\text{ASR} + \lambda_p L_p + \lambda_w L_w$, with $\lambda_p = \lambda_w = 0.1$ in best-performing models.

This formulation enhances both phrase retrieval effectiveness and fine-grained cross-attention, leading to measurable accuracy gains [2404.10180].

## 5. Empirical Performance

### Model Architectures

- **Baseline:** Dual-mode NAM with either a 3-layer or 1-layer Conformer context encoder (B1, B2).
- **Deferred NAM:** 4-layer DAN ($0.8$M params) for lightweight encoding plus a 1-layer Conformer for the deferred phase.

### Word Error Rate (WER)

Average WER (%) on in-context test sets for $k^p = 32$:

| Experiment | ANTI | WO_PREFIX | W_PREFIX |
|------------|------|-----------|----------|
| B1         | 2.3  | 3.0       | 2.4      |
| D1         | 1.9  | 2.6       | 1.9      |
| D2         | 1.8  | 2.3       | 1.8      |
| D3         | 1.8  | 2.0       | 1.5      |

D3 attains a 37.5% relative WER reduction over B1 on W_PREFIX ($2.4 \to 1.5$) [2404.10180].

### Latency

Pre-decoding delay at $k^p = 32$:

| #Phrases | QueryEnc | LightEnc | PhraseAttn | ContextEnc | WPAttn | Total  |
|----------|----------|----------|------------|------------|--------|--------|
| 3 K      | 2.3 ms   | 3.5 ms   | 0.9 ms     | 1.3 ms     | 0.7 ms | 8.7 ms |
| 20 K     | 2.3 ms   | 22.8 ms  | 5.2 ms     | 1.3 ms     | 0.7 ms | 32.3 ms|

Compared to dual-mode NAM (B2: 520 ms at 20 K, B1: 1549 ms), Deferred NAM achieves $8.3\times$ to $16.1\times$ speedup [2404.10180].

## 6. Discussion and Implications

Deferred encoding scales context biasing to 20,000 phrases with $<33$ ms delay by limiting heavy computation to $k$ selected phrases. The linear growth in pre-decoding delay with $N$ affects only the lightweight encoder, while deferred heavy encoding grows with $k$ alone. This approach also reduces memory footprint since only $k$ wordpiece embeddings are materialized per utterance. Enhanced recognition accuracy is attributed to supervised phrase retrieval (CE-PA) and wordpiece gating (CE-WA). However, small $k$ risks omitting relevant rare phrases if retrieval fails, highlighting a trade-off between latency and recall [2404.10180].

## 7. Future Directions

Potential directions for Deferred NAM include adaptive $k$ selection, lighter pre-filters such as learned hashing, streaming extensions, and stricter on-device privacy constraints. Two-pass encoding and auxiliary CE losses on phrase and wordpiece attention together suggest an effective strategy for large-scale, low-latency ASR contextualization [2404.10180].

Source: https://www.emergentmind.com/topics/deferred-nam