StreamUni: Unified ASR & ST Models

• StreamUni is a unified framework that enables both offline and streaming automatic speech recognition and translation using a single model without retraining.
• It leverages advanced neural architectures—such as hybrid CTC/attention, RNNT with consistency regularization, and decoder-only models—via chunk-based attention and causal convolutions.
• Unified training and consistency mechanisms minimize the latency-accuracy trade-off, reducing engineering overhead and streamlining deployment.

StreamUni refers to a class of unified neural architectures for streaming automatic speech recognition (ASR) and streaming speech translation (ST) that enable a single model to operate efficiently across both non-streaming (offline, full-context) and streaming (low-latency, partial-context) modes, without structural modifications or retraining. This approach eliminates the need for distinct systems for each latency regime, reducing engineering overhead and closing the historical performance gap between offline and real-time systems. Recent instantiations of StreamUni span hybrid CTC/attention models, transducer (RNNT) architectures with consistency regularization, decoder-only discrete unit models, and unified large speech-LLMs (LSLMs) for streaming speech translation, each leveraging chunk-based attention, causal convolutions, and policy learning to balance latency against recognition or translation accuracy (Zhang et al., 2020, Chen et al., 2024, Xia et al., 11 Mar 2026, Andrusenko et al., 21 Apr 2026, Guo et al., 10 Jul 2025).

1. Core Principles and Problem Formulation

StreamUni is motivated by the need to serve both real-time low-latency settings and accuracy-maximizing batch scenarios using a single model instance and checkpoint. The primary technical objectives are:

• Latency-Accuracy Trade-off: Minimizing token emission delay (streaming) while maintaining full-context accuracy (offline).
• Unified Model Parameters: Shared parameters for all inference modes, eschewing mode-specific branches or models.
• Train/Inference Consistency: Mechanisms to close the mismatch between partial-context streaming inference and standard full-context training.

Formally, for speech recognition, StreamUni models estimate the output $y$ (tokens) given input $x$ (audio frames), supporting factorizations such as:

$p(y|x_{1:g_i}, y_{<i})$

where $g_i$ indicates the last frame seen when emitting $y_i$. For streaming translation, policy ($g_i$), truncation ($a,b$), and generation decisions are explicitly parameterized to accommodate both streaming and non-streaming settings, e.g.,

$$p(\mathbf{y}|\mathbf{x},\mathbf{g},\mathbf{a},\mathbf{b})=\cdots$$

as described in (Guo et al., 10 Jul 2025).

2. Model Architectures and Unified Design

StreamUni implementations adopt several architectural paradigms:

• Hybrid CTC/Attention: A conformer-based encoder feeds both a CTC branch for real-time emission and an attention decoder for rescoring, with dynamic chunk-based attention to bound latency (Zhang et al., 2020).
• RNNT with Consistency Regularization: A conformer (FastConformer) encoder, LSTM predictor, and joint network support both offline and streaming via chunk-limited attention and dynamic chunked convolution (DCConv). Offline and streaming computations are regularized for consistency, enforcing latent distribution alignment (Andrusenko et al., 21 Apr 2026).
• Decoder-Only Discrete Token Models: Speech is quantized (HuBERT+k-means) and fed, with boundary tokens, to a causal Transformer decoder trained with right-chunk masking and boundary-triggered text prediction (Chen et al., 2024).
• Unified LSLM for StreamST: A large multimodal Transformer ingests audio embeddings and text, realizing a chain-of-thought (CoT) paradigm coordinating segmentation, policy decision, and translation through stage-wise decoding (Guo et al., 10 Jul 2025).

A commonality across designs is chunk-based self-attention and/or convolution, masking future context appropriately for streaming. In practice, a single checkpoint can switch between full-context (offline) and chunked (streaming) attention/convolution at inference, with no architectural re-instantiation (Zhang et al., 2020, Andrusenko et al., 21 Apr 2026, Xia et al., 11 Mar 2026).

Model Type	Encoder	Decoder	Streaming?
CTC/Attention	Conformer	Transformer	Dynamic chunking
RNNT	FastConformer	LSTM	Chunk-limited attn
Discrete D-only	Quantized (HuBERT)	Causal Transformer	Boundary token, mask
LSLM (Trans+Trans)	wav2vec2.0, logmel	Multimodal Transformer	Chain-of-Thought

3. Unified Training and Consistency Mechanisms

Successful StreamUni models are trained jointly on both streaming and non-streaming data flows:

• Mixed Mode Training: Training batches sample non-streaming, standard streaming, and context-aware streaming examples (in 1:1:1 ratio for Uni-ASR), each with dedicated losses ($\mathcal{L}_{NS}$, $\mathcal{L}_{SS}$, $x$0), and sum them uniformly per step (Xia et al., 11 Mar 2026).
• Chunk Sampling for Robustness: During training of CTC/attention or transducer models, chunk sizes are randomly sampled per batch to force the encoder to learn representations compatible with varying latency constraints (Zhang et al., 2020, Andrusenko et al., 21 Apr 2026).
• Consistency Regularization: Unified RNNT leverages a mode-consistency regularization term ($x$1) that mutually KL-regularizes the latent distributions between offline (teacher) and streaming (student) runs on the same input. Optimizing

$x$2

ensures that decisions and representations are aligned across modes (Andrusenko et al., 21 Apr 2026).

• Data Augmentation (Discrete D-only): Techniques like speed perturbation, boundary shift, time-masking, and deduplication further bridge the offline-streaming gap (Chen et al., 2024).
• Streaming CoT Curriculum (StreamST): For streaming speech translation, training data is synthesized by truncating full-utterance examples at random chunk boundaries, constructing multi-stage targets to supervise transcription, segmentation, policy, and translation together (Guo et al., 10 Jul 2025).

4. Decoding and Policy Strategies in Streaming

Efficient low-latency emission requires inference-time policies that align with the model's training:

• Latest-Token Fallback: In Uni-ASR, a fallback decoding strategy emits the last token of each chunk immediately, but permits one step of correction upon the arrival of the next chunk. This is accomplished by pre-filling the LLM with previous outputs padded at the last position and iteratively overriding only the boundary token with negligible latency increment (Xia et al., 11 Mar 2026).
• Chunked Attention/Convolution: During streaming, each layer restricts attention (MHA or DCConv) to a window defined by the current and past frames, plus (optionally) a learnable or user-tuned right context for look-ahead (Zhang et al., 2020, Andrusenko et al., 21 Apr 2026).
• Boundary-Token Insertion (BTI) and Right-Chunk Masking: For discrete decoder-only models, text token emission is triggered by explicit boundary tokens within the speech token sequence; right-chunk attention permits trade-off between latency and context (Chen et al., 2024).
• Unified Chain-of-Thought (CoT) Policies: StreamUni for translation explicitly models both truncation and generation policies through CoT. Decoding proceeds in stages: transcribe, decide truncation, and, conditional on breakpoint, generate translation for current or all segments (Guo et al., 10 Jul 2025).
• Policy Decision Logic: Policy switches are based on stability in transcription (no change for consecutive chunks) or punctuation boundaries, with segment truncation and translation generation integrated into a single model pass.

Pseudocode for streaming policy logic in StreamUni (ST) is as follows (Guo et al., 10 Jul 2025):

$x$3

5. Performance, Ablation, and Benchmark Comparisons

Empirical evaluations demonstrate that StreamUni frameworks achieve minimal degradation when operating under streaming constraints compared to full-context baselines.

ASR Results

Model	CER/WER Offline	Streaming Latency	CER/WER Streaming	Notable Metrics
Uni-ASR/LLM AED (Xia et al., 11 Mar 2026)	1.44% CER (AISHELL-1)	1000ms chunks	2.15% CER	Fallback reduces degradation by 10–20% per chunk
CTC/Attention (Zhang et al., 2020)	4.64% CER (AISHELL-1)	640ms chunks	5.33% CER	Degrades to 5.71% at 160ms latency, remains competitive
Discrete Decoder-Only (Chen et al., 2024)	6.2% CER	Delta-limited	6.4% CER (BTI)	Streaming penalty BTI ≈ 0.4–0.5%

Transducer

• Unified RNNT + MCR achieves WER of 6.63% (L-size), 5.91% (XL-size) offline, and 7.47%/6.52% at 0.56s latency; larger right context improves both streaming and offline metrics (Andrusenko et al., 21 Apr 2026). Mode-consistency reduces the streaming penalty to ≤0.2% relative for XL models.

Streaming Speech Translation

• StreamUni (Phi4) attains BLEU ≈ 35 at AL ≈ 1000 on CoVoST2 Fr→En, surpassing DiSeg (BLEU ≈ 28) and prior SimulST models (Guo et al., 10 Jul 2025).
• On MuST-C En→De, StreamUni yields SacreBLEU ≈ 32 at StreamLAAL ≈ 600, outperforming policy-specific and segmentation-dependent baselines.

Ablation studies identify right-chunk attention and label smoothing as the most influential in minimizing the streaming penalty.

6. Significance, Limitations, and Open Research Problems

StreamUni advances unified modeling by:

• Holistic Model Design: Single-architecture deployment for both streaming and non-streaming use cases, reducing R&D and maintenance costs.
• Minimized Train/Inference Gap: Mode-consistency regularization, context-aware training, and chunk-sampling approaches yield minimal degradation under tight latency budgets.
• Seamless Mode Switching: Inference transitions between full-context and streaming regimes by toggling chunk sizes and mask parameters, eliminating retraining or structural changes.

Key limitations and open directions:

• Language Generalization: Many approaches have been validated primarily on Chinese or English data; robust multilingual scaling remains to be demonstrated (Xia et al., 11 Mar 2026).
• Latency Floor: Current chunked attention strategies limit effective granularity to 100–320 ms; methods for sub-100 ms latency, e.g., trainable monotonic attention, are nascent (Xia et al., 11 Mar 2026, Andrusenko et al., 21 Apr 2026).
• Policy Learning for Segmentation: Rule-based truncation (e.g., via transcription stability or punctuation) is brittle in noisy conditions; reinforcement or confidence-aware learning is a promising avenue (Guo et al., 10 Jul 2025).
• Edge Deployment: Large model capacity (e.g., Qwen3-1.7B, Phi4-7B) incurs high computation; distillation and quantization are identified as future work (Xia et al., 11 Mar 2026).
• Cache Passing for Chunked Attention: Current implementations recompute left context per decoding step; optimized state-passing could further reduce runtime bottlenecks (Andrusenko et al., 21 Apr 2026).

StreamUni, encompassing RNNT, AED, decoder-only, and LSLM-based paradigms, sets a technical precedent for unified, low-latency, high-accuracy speech recognition and translation under streaming constraints (Zhang et al., 2020, Chen et al., 2024, Xia et al., 11 Mar 2026, Andrusenko et al., 21 Apr 2026, Guo et al., 10 Jul 2025).