Chunk-wise Attention Transducer (CHAT)

• CHAT is a streaming sequence-to-sequence model that partitions encoder outputs into fixed-size chunks with local cross-attention for flexible alignment.
• It preserves the transducer’s blank/non-blank logic while enabling parallel processing and achieving significant reductions in computational latency.
• Empirical results show up to 46.2% memory savings, faster training, and improved WER and BLEU scores across speech recognition and translation tasks.

Chunk-wise Attention Transducer (CHAT) is a streaming sequence-to-sequence model that extends standard RNN-T by processing encoder outputs in fixed-size chunks and by using cross-attention within each chunk in the joiner rather than a single encoder frame. In this formulation, time advances in chunks rather than individual frames, the transducer-style blank/non-blank logic is preserved, and local alignment flexibility is introduced inside each chunk. Before the name CHAT was used explicitly, closely related systems had already introduced chunk-wise attention in transducer joint networks and had shown that a chunked attention-based encoder-decoder with an End-of-Chunk symbol is equivalent to a transducer that operates on chunks instead of frames (Xu et al., 27 Feb 2026, Wang et al., 2020, Zeineldeen et al., 2023).

1. Genealogy and scope

The conceptual roots of CHAT lie in online attention mechanisms that replaced global soft attention with monotonic, chunk-restricted attention. Adaptive monotonic chunk-wise attention (AMoChA) combined a latency-controlled bidirectional listener with adaptive monotonic chunk-wise attention and produced an online LAS model with only 3.5% relative performance reduction to LAS baseline on an internal Mandarin corpus (Fan et al., 2018). Multi-head monotonic chunk-wise attention then extended MoChA by splitting encoder and decoder representations into multiple heads, computing per-head boundaries and per-head chunk-wise contexts, and improving AISHELL-1 test CER from 8.96% to 7.68% when combined with pooling and MWER (Liu et al., 2020).

A transducer realization appeared when chunk-wise attention was introduced directly in the joint network. In the 2020 attention-based transducer, encoder outputs were grouped into non-overlapping chunks of width $w$, the decoder state attended inside the current chunk, and the paper noted that $w=1$ reduces the model to conventional RNN-T while a full-length chunk makes it similar to AED; on 500 hours of LAIX non-native English training data, the best configuration reported ~10.6% WER reduction over baseline RNN-T together with over 1.7x training speedup (Wang et al., 2020).

A parallel line of work made the transducer equivalence explicit on the decoder side. In a chunked attention-based encoder-decoder, a special End-of-Chunk symbol advances from one chunk to the next chunk and effectively replaces the conventional end-of-sequence symbol; the model is described as equivalent to a transducer model that operates on chunks instead of frames, where EOC corresponds to the blank symbol (Zeineldeen et al., 2023).

2. Formal architecture

In the explicit 2026 CHAT model, the encoder remains streaming and the predictor is unchanged, but encoder outputs are partitioned into fixed-size chunks of $C$ frames and the joiner performs multi-head cross-attention between the predictor state and all encoder frames inside the current chunk. The reported encoder is FastConformer-Large with 17 layers, model dimension 512, causal convolutions, 8× subsampling, and a default chunk size of 12 encoder frames, corresponding to about 960 ms of audio (Xu et al., 27 Feb 2026).

Let $h^{\text{enc}}_{1:T}$ be the encoder output sequence and let chunk $n$ contain encoder frames

$$\text{chunk}(n)=\{h^{\text{enc}}_{t}: t \in \{Cn, Cn+1, \ldots, Cn+C-1\}\}.$$

CHAT appends one extra zero frame to each chunk, so the joiner can attend to an explicit blank location. With predictor state $h_u^{\text{pred}}$, the joiner computes

$$q_u = W_Q h_u^{\text{pred}}, \qquad k_t = W_K h_t^{\text{enc}}, \qquad v_t = W_V h_t^{\text{enc}},$$

$$\alpha_{t,u} = \frac{\exp\left( \frac{q_u^{\top} k_t}{\sqrt{d_{\text{joint}}}} \right)} {\sum_{j \in \text{chunk}(n)} \exp\left( \frac{q_u^{\top} k_j}{\sqrt{d_{\text{joint}}}} \right)},$$

$$c_{n,u} = \sum_{t \in \text{chunk}(n)} \alpha_{t,u} v_t, \qquad h_{n,u}^{\text{joint}} = \operatorname{ReLU}\left(c_{n,u} + h_u^{\text{pred}}\right), \qquad p_{n,u} = \operatorname{softmax}\left(W_{\text{out}} h_{n,u}^{\text{joint}}\right).$$

The transducer lattice is correspondingly reindexed. If $w=1$0, time now advances over chunk indices $w=1$1 rather than frame indices $w=1$2, so the alignment lattice is $w=1$3 rather than $w=1$4. Blank advances the chunk index; a non-blank token advances the label index. The joiner tensor therefore changes from $w=1$5 to $w=1$6. Global monotonicity is retained at chunk granularity, while local alignment can be non-monotonic inside the chunk (Xu et al., 27 Feb 2026).

3. Encoder-side chunking and context control

Encoder-side work on chunk-wise Transformer and Conformer models indicates that CHAT behavior depends on more than the joiner alone. One family of methods modifies self-attention masks. The shifted chunk encoder alternates regular chunk partitioning and chunk partitioning shifted by $w=1$7, so that successive layers create cross-chunk connections while each layer still performs attention in windows of size $w=1$8; complexity remains linear in sequence length $w=1$9 for fixed $C$0 (Wang et al., 2022). Variable masking in transformer transducers compares fixed masking with chunked masking and shows that chunked masking achieves a better accuracy vs latency trade-off compared to fixed masking, while variable masking trains a single configurable model across multiple masks and improves the acoustic re-scoring scenario by up to 8% relative (Swietojanski et al., 2022).

Conformer-based models add a second issue: convolution can leak future context even when attention is chunked. Dynamic Chunk Convolution confines non-causal convolution to each chunk, synchronizes convolutional chunk boundaries with self-attention chunk masks, and on LibriSpeech reduces the degradation of streaming mode over the non-streaming full-contextual model from 41.7% and 45.7% to 16.7% and 26.2% on test-clean and test-other (Li et al., 2023). SSCFormer pushes the same idea further with Sequential Sampling Chunk and Chunked Causal Convolution, reporting 5.33% CER on AISHELL-1 while keeping chunk-wise MHSA computation linear in input length and preserving large-batch parallel training (Wang et al., 2022).

For CHAT, these encoder-side results are significant because they show that chunk boundaries, attention masks, left-context reuse, and chunk-aware convolution jointly determine the effective receptive field, latency, and train-test consistency of the overall transducer.

4. Monotonic alignment and decoding regimes

Decoder-side CHAT mechanisms inherit from monotonic chunk-wise attention. In MoChA and its extensions, a monotonic process first chooses a boundary $C$1 and then applies soft attention over a local window ending at that boundary. MTH-MoChA generalizes this with per-head boundaries $C$2, per-head chunk-wise contexts $C$3, and the averaged context

$C$4

which improved AISHELL-1 test CER from 8.96% to 7.68% (Liu et al., 2020).

AMoChA replaces a fixed chunk size with a predicted chunk length, using either

$C$5

or

$C$6

and was introduced together with smoothing methods that alleviate the boundary shift caused by latency-controlled encoders (Fan et al., 2018).

A later simplification, Monotonic Truncated Attention, removes explicit chunk width and defines attention over a truncated prefix. With

$C$7

and a decoding end-point $C$8 chosen by the first $C$9 such that $h^{\text{enc}}_{1:T}$0, the context becomes

$h^{\text{enc}}_{1:T}$1

Online hybrid CTC/attention systems paired MTA with truncated CTC prefix scores and Dynamic Waiting Joint Decoding to synchronize label-synchronous attention with frame-synchronous CTC in streaming decoding (Miao et al., 2023). On HKUST, the Transformer-based online CTC/attention architecture with chunk-SAE and MTA-based SAD achieved 23.66% CER with 320 ms latency and as little as 0.19% absolute CER degradation compared with the offline baseline (Miao et al., 2020).

5. Efficiency, latency, and empirical performance

Explicit CHAT derives most of its efficiency from replacing the frame axis with a chunk axis. With chunk size $h^{\text{enc}}_{1:T}$2, $h^{\text{enc}}_{1:T}$3, and a joiner tensor of shape $h^{\text{enc}}_{1:T}$4 rather than $h^{\text{enc}}_{1:T}$5, the model reports up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. On LibriSpeech test-clean and test-other, CHAT improved WER from 3.01 to 2.82 and from 7.61 to 7.45. On speech translation it improved BLEU from 29.44 to 32.33 for EN→DE, from 34.01 to 39.55 for EN→ZH, and from 18.95 to 23.10 for EN→CA. The method is described as particularly effective for speech translation, where RNN-T’s strict monotonic alignment hurts performance (Xu et al., 27 Feb 2026).

Earlier chunk-wise attention transducers already indicated the same scaling pattern. The 2020 attention-based transducer reported over 1.7x training speedup, ~10.6% WER reduction over baseline RNN-T on 500 hours of LAIX non-native English training data, and, after 8-bit weight quantization without WER degradation, RTF and latency of 0.34~0.36 and 268~409 milliseconds on a single CPU core of a production server (Wang et al., 2020).

These results locate CHAT in a characteristic operating regime: the model reduces the temporal dimension handled by the transducer, preserves a bounded-latency streaming path, and reintroduces flexibility inside each chunk precisely where frame-synchronous RNN-T is most rigid.

6. Limitations and research directions

A recurrent limitation across CHAT-style work is that chunk size is usually fixed or hand-tuned. The named CHAT paper points to adaptive chunk sizing as future work (Xu et al., 27 Feb 2026). Dynamic Chunk Convolution likewise notes that chunk size is not learned, is uniformly sampled during training, and is hand-tuned at inference (Li et al., 2023).

Another recurring issue is train-test mismatch between full-context training operations and bounded-context streaming inference. The principal remedies have been chunk-confined attention masks, chunk-aware convolution, and variable masking; chunked masking achieves a better accuracy vs latency trade-off than fixed masking, and variable masking enables a single configurable model to serve as both first pass streaming recognition and second pass acoustic rescoring (Swietojanski et al., 2022).

Chunked models also appear to generalize well to long-form speech. In chunked AED experiments on TED-LIUM-v2, global AED WER rose from 6.9 to 62.4 as up to 20 consecutive sequences were concatenated, whereas the chunked encoder plus chunked decoder model remained around 7.0–7.3 and was described as generalizing very well to long-form speech (Zeineldeen et al., 2023). This suggests that future CHAT systems may combine chunk-level lattice reduction, adaptive chunk sizing, and chunk-aware encoder modules within a single streaming transducer design.