--- title: Streaming Transformer Transducer (T-T) url: https://www.emergentmind.com/topics/streaming-transformer-transducer-t-t type: topic --- # Streaming Transformer Transducer (T-T) The Streaming Transformer Transducer (commonly abbreviated as Streaming T-T or TT) is a framework for end-to-end automatic speech recognition (ASR) and related sequence transduction tasks that unifies streaming and non-streaming inference within a single Transformer-based model. TT combines the sequence modeling capacity of deep Transformer encoders with the latency- and alignment-flexibility of neural transducers, leveraging constrained self-attention and architectural innovations to deliver low-latency, high-accuracy recognition, adaptable to both real-time and batch-processing contexts. ## 1. Architectural Foundations and Model Formulation The canonical Streaming Transformer Transducer architecture comprises three major components: (1) an audio encoder built from a stack of Transformer layers, (2) a label prediction network (label encoder or predictor), and (3) a joint network that fuses encoder and prediction states at each output hypothesis. ### Encoder Structure The audio encoder ingests a sequence of acoustic feature frames $x_1,\ldots,x_T$ (typically log-Mel features sampled every 30 ms or 10 ms). The encoder stack is structurally split: - **No-lookahead (“stem”) layers:** $N_1$ layers with constant left context and right context $R=0$, guaranteeing strict causality for streaming. - **Variable-context (“top”) layers:** Additional $N_2$ layers where each layer $\ell$ exposes a tunable right context $R_\ell$. This design admits a block diagram: ``` x → [Layer₁…Layer_{N₁} (R=0)] → h⁰ ↓ ↙ ↘ [Layer_{N₁+1}…Layer_{N₁+N₂} (small R)] [Layer_{N₁+1}…Layer_{N₁+N₂} (large R)] h^{enc}_{low} h^{enc}_{high} ``` The encoder states are computed as $h^{enc}_t = \mathrm{Encoder}(x_{1:t+R};\theta_{enc})$, where $R$ controls access to future frames for each layer. ### Label Encoder and Joint Network - **Label encoder:** Consumes the sequence of previous non-blank labels $y_{1:u}$, emitting states $h^{pred}_u$. - **Joint network:** Computes $h^{joint}_{t,u} = W_{enc} h^{enc}_t + W_{pred} h^{pred}_u$. - **Output distribution:** $P(k | t,u) = \mathrm{Softmax}_k \left(W_{joint} \tanh(h^{joint}_{t,u})\right)$ over the union of vocabulary and “blank” token. ### RNNT Loss The model is trained with the RNN-Transducer (RNNT) loss: $$ L_{RNNT}(x,y) = - \ln p(y|x) \qquad p(y|x) = \sum_{\pi: B(\pi) = y} \prod_{i=1}^{|\pi|} P(\pi_i | h^{enc}_{t_i}, h^{pred}_{u_i}), $$ with $B(\pi)=y$ denoting the valid alignments collapsing to $y$. Self-attention masks for streaming enforce, layerwise, a window $[t-L, t+R_\ell]$: $$ \mathrm{Mask}^{(\ell)}_{t,s} = \begin{cases} 1 & \text{if } t-L \leq s \leq t+R_\ell \\ 0 & \text{otherwise} \end{cases} $$ ## 2. Training Paradigms: Variable Context Sampling and Constrained Alignment ### Variable-Context Sampling During training, stochastic right-context configurations are sampled for the variable-context layers. Each batch selects $c_j \in \mathcal{C}$ (where $\mathcal{C}$ is a set of right-context vectors) and applies the associated masking for all layers. This process yields a model simultaneously conditioned to operate well under both strict streaming (low $R$) and high-accuracy (large $R$) settings, as only the last $N_2$ layers differ across sampled contexts, while the main “stem” is always causal. ### Alignment Delay Constraints To directly control latency, an optional constrained-alignment RNNT loss restricts alignments such that predicted word boundaries $T_i(\pi)$ remain within $D_{max}$ of reference times $T_i^{ref}$: $$ |T_i(\pi) - T_i^{ref}| \leq D_{max}, \quad \forall i $$ The RNNT loss is evaluated over this restricted path set, yielding models with bounded output delay, critical for latency-sensitive applications. ## 3. Unified Streaming and Non-Streaming Inference: The Y-Model Streaming TT provides a mechanism to optimize the latency–accuracy tradeoff at inference by varying only the right context $R_\ell$ of the variable-context layers: - **Low-latency mode ($R_\text{small}$):** immediate, partial results with $R_\ell$ as low as 0 or 4 frames ($\sim\!240\,\mathrm{ms}$ lookahead). - **High-latency mode ($R_\text{large}$):** batch/poststreaming final results, up to $2.4\,\mathrm{s}$ lookahead per layer. The Y-model architecture enables both streaming and final-pass decoding. The bulk of the computation is shared (no-lookahead “stem”), while the top $N_2$ layers operate in parallel with small and large right contexts: - **Branch A (low-latency):** operates online, emits partial transcriptions - **Branch B (high-latency):** run post-stream (or on user endpointing) for a higher-quality, lower-error final hypothesis - **Hypothesis merging:** Branch B’s (final) output simply replaces the last segment of Branch A’s hypothesis seamlessly ### Quantitative Latency–Accuracy Tradeoff Empirical evaluations demonstrate that, for voice-search (30k hours training; test $\sim$14k utterances): | Mode | WER (%) | Alignment Delay (ms) | |-------------------------------------------------|---------|---------------------| | Full-context (non-streaming, 34 s) | 4.8 | 60 | | Zero-context streaming | 6.1 | — | | Y-Model (2.4 s lookahead, high branch) | 5.0 | 742 | | Y-Model (240 ms lookahead, low branch) | 5.3 | 767 | | With alignment-constrained training, 2.4 s mode | 4.9 | 74 | | With constraint, 240 ms mode | 6.5 | 119 | Only $1$–$2\,\mathrm{s}$ lookahead plus $50$–$100\,\mathrm{ms}$ end-of-utterance latency suffices to nearly match the accuracy of unrestricted-context models ($4.8\%$ vs $4.9$–$5.0\%$). ## 4. Efficient Streaming and Non-Streaming Inference Optimizations ### Audio Encoder Compute - **Batch-Step Streaming:** Batches $B$ new frames and processes them through all layers in parallel, maintaining per-layer attention caches ($B=1\ldots120$); higher $B$ increases throughput but at a cost to per-block latency. - **Query Slicing for Non-Streaming:** Splits $T$ queries into blocks of size $Q$, each block attending to a sliding window of $K=Q+R$; memory grows as $O(Q\times(Q+R))$ (not $O(T^2)$). ### Label Encoder Acceleration - **Bigram Embedding Lookup:** Replaces the full label encoder with a $|V|^2\times d$ table (embedding for consecutive label pairs), caching $h^{pred}$ for repeated label histories. Empirically: - 40-gram context Transformer: WER=4.8%, runtime factor=0.3 - 3-gram: WER=4.8%, runtime factor=0.02 - 2-gram bigram lookup: WER=4.9%, runtime factor=0.01 ### Measured Throughput - **Single TPU core:** 100 seconds of audio processed in 0.3s (full-batch), 0.6s (query-slicing), and 1.8s (batch-step, $B=1$). - **8-core CPU:** 36 minutes of audio recognized in less than 3 minutes ($<8\%$ real-time), using query slicing and bigram lookup. ## 5. Empirical Evaluation and Design Analysis The unified Streaming Transformer Transducer was validated on large-scale voice-search data, showing: - Substantial reduction in WER (up to $20\%$ relative) for high-latency (Y-Model, $2.4\,\mathrm{s}$) vs. baseline streaming - Sharp improvement in streaming (WER 6.1%) to near offline with modest right-context (WER 4.9–5.3%) - Constrained-alignment branch lowers output delay to near the reference at only marginal cost to accuracy A major outcome is that decoupling right-context at inference allows for real-time streaming with minimal quality loss, and for extreme accuracy (matching full-context models) with only a brief window of additional latency at turn end. ## 6. Implementation Considerations and Practical Deployment ### Masked Self-Attention Right-context is implemented by masking keys in self-attention per $$ \mathrm{Mask}^{(\ell)}_{t,s} = \begin{cases} 1 & t-L \leq s \leq t+R_\ell \\ 0 & \text{otherwise} \end{cases} $$ sampling $R_\ell$ for layers during training. Key/value caches and per-batch context selection make this paradigm compatible with both streaming hardware and accelerated server inference. ### Branching Only Top Layers at Inference By building one causal “stem” and several “heads” with separate right-contexts, the system reconstructs both low-latency streaming and batched high-quality decoding from a single trained model. This minimizes memory and compute duplication. ### On-Device and Server Use Batch-mode, block-level, and cache-aware implementations enable throughput scaling from TPUs to multi-core CPUs. Reduced label-encoder (bigram lookup) further lowers inference cost, critical for deployment in constrained environments. ## 7. Contextualization and Broader Impact The streaming TT framework unifies, in a single trainable and deployable model, both low-latency and full-accuracy automatic speech recognition use cases. Unlike prior approaches that required separate or hybrid models for streaming and non-streaming, this architecture offers: - Unified model for interactive, low-latency (phone, voice-assistants) and high-accuracy batch (dictation, offline processing) scenarios - Direct control of latency/accuracy at runtime via right-context selection - Minimal WER degradation even with constrained output delay This unified approach streamlines deployment, reduces operational complexity, and approaches state-of-the-art WER with such techniques as constrained-alignment loss and low-latency optimization. The model directly extends to other monotonic sequence transduction tasks, contingent on the provisions for future and past context inherent to each application. A plausible implication is that the broad class of sequence modeling problems where alignment flexibility and bounded latency are critical (ASR, streaming translation, diarization) now have a single, hyperparameter-controlled, deployment pathway, mitigating the need for separate architectures specialized for online vs. offline modes [2010.03192]. Source: https://www.emergentmind.com/topics/streaming-transformer-transducer-t-t