Speech ReaLLM: Streaming ASR & Speech Reasoning

• Speech ReaLLM is a family of architectures for real-time streaming ASR that integrates decoder-only LLMs with BLANK token interleaving to manage temporal structure.
• It employs a fixed-monotonic alignment strategy using an external CTC teacher to map continuous audio into text tokens without reliance on forward-backward marginalization.
• Empirical results on LibriSpeech and subsequent studies demonstrate competitive WERs and low latency, while later extensions broaden capabilities to speech understanding, reasoning, and generation.

to=arxiv_search.search 大发快三大小单双json {"query":"\"Speech ReaLLM\" OR ReaLLM speech", "max_results": 10} to=arxiv_search.search 񹚊ppjson {"query":"(Seide et al., 2024, Lin et al., 3 Apr 2026, Wang et al., 2023)", "max_results": 10} Speech ReaLLM denotes both a specific architecture and an emerging research direction in speech-capable LLMs. In its original and most explicit sense, Speech ReaLLM was introduced as a real-time streaming automatic speech recognition system that marries “decoder-only” ASR with the RNN-T and teaches a multimodal LLM to generate after every input token received in real time, often emitting an empty response via a special BLANK token rather than waiting for endpointing (Seide et al., 2024). In subsequent work, the term or closely related formulations have been used more broadly for speech reasoning LLMs that unify speech understanding, structured temporal inference, retrieval, explainability, speech generation, and full-duplex interaction, including speaker-attributed ASR, speech-to-speech conversion, and explainable speech emotion recognition (Lin et al., 3 Apr 2026, Wang et al., 2023, Su et al., 29 Sep 2025).

1. Terminological scope and lineage

Across recent arXiv literature, “Speech ReaLLM” no longer refers to a single invariant model family. It has instead come to name a cluster of architectures that preserve LLM-style autoregressive reasoning while extending it to continuous audio, structured temporal decisions, or speech-native generation. The 2024 formulation is the canonical one: a streaming ASR system with decoder-only generation and BLANK-mediated real-time interleaving (Seide et al., 2024). Later papers use the phrase more expansively for retrieval-augmented speech understanding, explainable SpeechLMs, or as a target notion for a “full Speech Reasoning LLM” (Wang et al., 2023, Su et al., 29 Sep 2025, Lin et al., 3 Apr 2026).

Paper	Use of “Speech ReaLLM”	Core emphasis
“Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time” (Seide et al., 2024)	Explicit model name	Streaming ASR without explicit end-pointing
“Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding” (Wang et al., 2023)	“Speech ReaLLM family of models” in the technical summary	Speech understanding with adapter and retriever
“Reasoning Beyond Majority Vote: An Explainable SpeechLM Framework for Speech Emotion Recognition” (Su et al., 29 Sep 2025)	“Speech ReaLLM” in the technical summary	Transcript, emotion label, and rationale generation
“Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR” (Lin et al., 3 Apr 2026)	“paving the way toward a full Speech Reasoning LLM (Speech ReaLLM)”	Agentic multi-turn temporal reasoning
“Raon-Speech Technical Report” (Kim et al., 8 Apr 2026)	Raon-SpeechChat as “Speech ReaLLM” in the technical summary	Speech understanding, generation, and full-duplex conversation

A common misconception is that Speech ReaLLM names only the 2024 streaming-ASR architecture. The literature shows a broader usage: some works retain the original emphasis on real-time interleaving, whereas others treat Speech ReaLLM as a broader program for speech-native reasoning, interaction, and generation (Seide et al., 2024, Lin et al., 3 Apr 2026).

2. Decoder-only streaming ASR by teaching the flow of time

The original Speech ReaLLM architecture was proposed as the first “decoder-only” ASR architecture designed to handle continuous audio without explicit end-pointing (Seide et al., 2024). Its central idea is to reproduce an RNN-T-like blank/non-blank alternation inside a decoder-only LLM stack. A streaming speech encoder, such as an Emformer or Conformer, consumes raw audio frame-by-frame or chunk-by-chunk and outputs a sequence of fixed-rate embeddings. These embeddings are interleaved with text token embeddings inside a stack of Transformer-decoder layers in a Llama-2 style decoder (Seide et al., 2024).

The model receives 80-dim log-Mel filterbank features every 20 ms, packs them into 240 ms chunks, and processes them with a 20-layer Streaming Conformer. The resulting speech embeddings are denoted $\{e_1,e_2,\dots,e_T\}$ with $e_t\in\mathbb{R}^d$, where $d=256$ in the reported experiments (Seide et al., 2024). At each real-time step, after receiving $e_t$, the decoder is invoked immediately and generates zero or more text tokens until it predicts BLANK. BLANK is treated as just another token in the decoder’s vocabulary, so whenever no new words can yet be emitted, the model predicts BLANK and terminates the inner generation loop (Seide et al., 2024).

Formally, the decoding process is written as

$$w_{t,k}=\arg\max_{w\in\mathcal V} P(w\mid h_{t,k-1}),$$

with iteration continuing until $w_{t,k}=\text{BLANK}$ (Seide et al., 2024). The significance of this construction is architectural rather than merely algorithmic: it allows a decoder-only LLM to behave like a real-time transducer while remaining within an autoregressive next-token framework. The paper explicitly frames this as a special case of a more general ReaLLM approach in which generation occurs after every input token received in real time, often producing an empty response (Seide et al., 2024).

3. Training, alignment, and adaptation to pretrained LLMs

Speech ReaLLM does not use the forward-backward marginalization of a conventional RNN-T. Instead, training relies on a fixed monotonic alignment “teacher” generated by an external CTC model (Seide et al., 2024). For each utterance, the CTC teacher provides a frame-level alignment

$$\alpha=[\alpha_1,\dots,\alpha_{T'}],\quad \alpha_t\in\{\text{word}_j\}\cup\{\text{BLANK}\},$$

where each $\alpha_t$ corresponds to one 240 ms chunk. Speech embeddings and token embeddings are then interleaved according to $\alpha$, and the model is trained with a standard cross-entropy loss over the resulting sequence (Seide et al., 2024):

$$\mathcal{L}=-\sum_{n=1}^{T'+K}\ln P(\hat y_n\mid \mathbf h_n).$$

This training strategy is important because it shows that a decoder-only LLM can be taught the timing structure of streaming ASR using fixed alignments plus cross-entropy, rather than a transducer-specific objective. The paper’s analysis argues that the decoder thereby learns to represent and reproduce the flow of time (Seide et al., 2024).

The same work also demonstrates compatibility with a large pretrained LLM. A frozen Llama-2-7B decoder was adapted by inserting a small two-layer MLP that maps the 256-dim speech embeddings into the 4096-dim Llama embedding space, repurposing the underscore token “_” as BLANK, and adding rank-16 LoRA adapters on all Transformer projection matrices. All original Llama-2 weights and the speech encoder were frozen; only the LoRA adapters and the embedding adapter MLP were trained. The system was fine-tuned on LibriSpeech (960 h) for 27 epochs with the same cross-entropy loss (Seide et al., 2024). This is a narrow but consequential result: it shows that streaming ASR behavior can be induced in a pretrained 7B decoder without rearchitecting it into a conventional transducer.

4. Empirical characteristics of the original system

On LibriSpeech “test,” the reported 80M Speech ReaLLM achieved WERs of 3.0% and 7.4% in real time, without an external LM or auxiliary loss (Seide et al., 2024). In the same comparison, a non-streaming Speech LLM of the same size achieved 4.8% and 8.0%, and an RNN-T achieved 3.6% and 9.4% (Seide et al., 2024). The pre-trained Llama-7B adaptation achieved 4.7% and 9.1% (Seide et al., 2024). These numbers establish that the streaming decoder-only formulation was competitive with same-scale streaming baselines while remaining close to a larger attention-based encoder-decoder reference.

System	WER test clean	WER test other
Speech ReaLLM	3.0%	7.4%
Speech LLM (non-streaming)	4.8%	8.0%
RNN-T	3.6%	9.4%
+ pre-trained Llama-7B	4.7%	9.1%

Inference cost was reported in real-time factors. Speech ReaLLM achieved RTF 0.94 with beam size 4 and greedy RTF 0.79, while maintaining dev WER 2.7/7.6 and 2.8/7.8 respectively (Seide et al., 2024). The paper also provides several ablations that clarify what the model learned. A non-streaming “Time-Aligned Speech LLM” trained with BLANK symbols, but still seeing all speech up front, reached Alignment Error Rate 4.9% and 6.5% and Length Error Rate 1.0% and 1.4% on dev clean/other, indicating that a decoder-only LLM can indeed learn and reproduce the passage of time via BLANK tokens (Seide et al., 2024).

A future-context ablation further showed that reducing the Conformer’s right context by 480 ms and delaying labels by two decoder tokens degraded WER from 2.7% and 7.6% to 4.4% and 9.1%, supporting the conclusion that future audio context is better handled in the encoder than by forcing the decoder to “look ahead” (Seide et al., 2024). Length robustness was equally notable: on the longest 100 utterances in dev, Speech ReaLLM stayed at 2.6% and 7.4%, whereas the non-streaming Speech LLM rose to 17.8% and 22.0%, and the time-aligned variant to 17.3% and 17.4% (Seide et al., 2024). This indicates that BLANK-mediated interleaving is not merely a latency device; it also changes the effective context-management behavior of the decoder.

5. Expansion toward speech reasoning, structured understanding, and explainability

Later work generalized the Speech ReaLLM idea from streaming transcription to broader speech reasoning. In “Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding,” the technical summary explicitly describes a “Speech ReaLLM family of models” built from three components: CTC-based blank-filtering, a Speech2Text adapter that maps compressed speech into the text embedding space of a frozen T5 (XXL), and, in the retrieval-augmented ReSLM variant, a Speech2Entity retriever that prepends relevant entities to the LLM input (Wang et al., 2023). On Speech MultiWOZ, the reported dialog state tracking performance improved from 24.7% JGA for SLM without adapter or retrieval, to 28.4% with the Speech2Text adapter, and to 34.6% for ReSLM; WER improved from 11.5% to 9.2% to 8.5% across those variants (Wang et al., 2023). Here, Speech ReaLLM is not primarily about streaming, but about aligning speech and language representations and handling rare entities through retrieval.

The same broadening appears in explainable speech emotion recognition. “Reasoning Beyond Majority Vote: An Explainable SpeechLM Framework for Speech Emotion Recognition” presents a two-stage pipeline in which a frozen Qwen2-Audio speech encoder and a LoRA-augmented Qwen2-Audio-7B-Instruct first produce a transcript and then emit both an emotion label and a concise natural-language rationale in a single autoregressive pass (Su et al., 29 Sep 2025). On MSP-Podcast v1.12, the reported open-form Macro-F1 rose from 30.11 to 36.14 under majority-vote scoring and from 56.78 to 66.78 under annotator-aware scoring when rationale supervision was added; human evaluation reported 69.1% overall acceptance of generated rationales (Su et al., 29 Sep 2025). In this usage, Speech ReaLLM denotes a reasoning-centric SpeechLM in which explanation is part of the generated target string.

A further extension is “Speaker-Reasoner,” an end-to-end Speech LLM with agentic multi-turn temporal reasoning for timestamped speaker-attributed ASR (Lin et al., 3 Apr 2026). Rather than single-pass SOT decoding, the model performs a global turn over the entire audio to predict speaker count and gender summary, followed by local turns over time slices that jointly emit speaker IDs, gender tags, start and end timestamps, and transcriptions. Its reasoning state is summarized by

$e_t\in\mathbb{R}^d$0

where the model attends over projected acoustic tokens for the current slice, previously generated tokens, and serialized speaker-aware cache entries (Lin et al., 3 Apr 2026). On AISHELL-4 Eval, it reported DER 5.26%, CER 13.83%, cpCER 14.73%, and $e_t\in\mathbb{R}^d$1 0.90%; on AliMeeting-Far test, DER 7.34%, CER 20.57%, cpCER 20.43%, and $e_t\in\mathbb{R}^d$2 –0.14% (Lin et al., 3 Apr 2026). The paper states that these results pave the way toward a full Speech Reasoning LLM, explicitly abbreviating that target as Speech ReaLLM (Lin et al., 3 Apr 2026).

6. Speech-to-speech, full-duplex interaction, and the broader ReaLLM agenda

The most expansive interpretation of Speech ReaLLM concerns systems that both understand and generate speech while preserving text-side reasoning. Several 2025–2026 papers address this transition from streaming ASR to full speech interaction. “SimulS2S-LLM” proposes simultaneous inference for speech-to-speech translation using an offline-trained speech LLM, CIF-extracted boundary-aware speech prompt vectors, a wait-$e_t\in\mathbb{R}^d$3 test-time policy, and incremental beam search over discrete output speech tokens (Deng et al., 22 Apr 2025). On CVSS, it reported, for example, Es→En at ATD≈3.44 s with ASR-BLEU ≈ 26.3 versus StreamSpeech ≈ 22.9, and the abstract summarizes this as improving ASR-BLEU scores by 3 points at similar latency (Deng et al., 22 Apr 2025).

“PRIME-Speech” takes a different position: adding speech output should not overwrite the speech-to-text reasoning path of a strong frozen backbone (Hu et al., 29 Jun 2026). Its frozen-backbone S2S conversion framework attaches a causal audio post-decoder synchronized with intermediate hidden states of the frozen backbone, uses mixed hidden-state, text, and audio-history conditioning, and introduces multi-token prediction to reduce the effective codec prediction rate (Hu et al., 29 Jun 2026). The paper reports FLEURS BLEU 31.40 → 33.24 for S2T versus S2S under ASR evaluation, first-token-to-first-audio latency dropping from 1.07 s to 0.39 s with MTP $e_t\in\mathbb{R}^d$4, and a failure mode in multi-turn dialogue where, without audio-cache reset, WER explodes > 100% by turn 5 (Hu et al., 29 Jun 2026). This directly addresses a recurrent concern in Speech ReaLLM design: adding speech generation can degrade S2T behavior unless the backbone is protected.

“MOSS-Speech” pushes further toward a “true speech-to-speech” LLM without text guidance (Zhao et al., 1 Oct 2025). It combines a modality-based layer-splitting architecture with a frozen pre-training strategy on top of Qwen-3-8B, uses the first 32 layers for joint fusion, and then branches into separate text and speech heads (Zhao et al., 1 Oct 2025). The paper reports 84.87 on spoken StoryCloze, 90.32 on zh-StoryCloze, and 67.19/69.53 on MMLU/CMMLU in pre-training evaluation, while in supervised fine-tuning it reports LLaMA-QA 77.33/63.67 for S→T/S→S and UTMOS 4.37 (Zhao et al., 1 Oct 2025). This suggests a shift from text-guided speech systems toward speech-native generation that still preserves competitive text performance.

The most directly full-duplex formulation in the supplied literature is Raon-Speech and Raon-SpeechChat. Raon-Speech transforms a pretrained Qwen3-VL-8B-Instruct backbone into a 9.1 B SpeechLM with an AuT speech encoder, Mimi codec, a speech-generation expert, residual code predictor, and speaker encoder; Raon-SpeechChat replaces the non-causal encoder with a causal streaming Voxtral encoder and reaches 9.8 B parameters (Kim et al., 8 Apr 2026). The report states that Raon-Speech trains on 1.38M hours of highly curated English and Korean speech and text datasets through speech modules alignment, end-to-end SpeechLM pre-training with knowledge distillation, and multi-task preference optimization-based post-training, while Raon-SpeechChat continues training on 119K hours of time-aligned real and synthetic dialogue data (Kim et al., 8 Apr 2026). Across 42 benchmarks, the report gives English Libri-clean WER 1.44, Libri-other WER 2.89, VoiceBench SpokenQA 76.79, and MMLU-Pro 64.05 for Raon-Speech, and on FDB v1.0 it reports for Raon-SpeechChat Candor TOR 0.213 for pause handling, backchannel frequency 0.081, smooth turn-take TOR 0.832, and user interruption TOR 0.980 (Kim et al., 8 Apr 2026). In the technical summary, this full-duplex extension is explicitly described as “Speech ReaLLM” (Kim et al., 8 Apr 2026).

A broader implication of these later works is that Speech ReaLLM has evolved from a narrowly defined real-time ASR mechanism into a research program organized around four persistent constraints: preserving a pretrained backbone’s text-side capabilities, maintaining temporal fidelity under streaming or multi-turn conditions, handling structure beyond plain transcription such as speaker identity or rationale generation, and producing speech output without reintroducing a rigid ASR→LLM→TTS bottleneck (Seide et al., 2024, Lin et al., 3 Apr 2026, Hu et al., 29 Jun 2026, Kim et al., 8 Apr 2026).