---
title: 'Speech ReaLLM: Streaming ASR & Speech Reasoning'
url: https://www.emergentmind.com/topics/speech-reallm
type: topic
---

# Speech ReaLLM: Streaming ASR & Speech Reasoning

to=arxiv_search.search  大发快三大小单双json
{"query":"\"Speech ReaLLM\" OR ReaLLM speech", "max_results": 10}
to=arxiv_search.search 񹚊ppjson
{"query":"2406.09569 2604.03074 2306.07944", "max_results": 10}
Speech ReaLLM denotes both a specific architecture and an emerging research direction in speech-capable large language models. In its original and most explicit sense, Speech ReaLLM was introduced as a real-time streaming automatic speech recognition system that marries “decoder-only” ASR with the RNN-T and teaches a multimodal LLM to generate after every input token received in real time, often emitting an empty response via a special BLANK token rather than waiting for endpointing [2406.09569]. In subsequent work, the term or closely related formulations have been used more broadly for speech reasoning LLMs that unify speech understanding, structured temporal inference, retrieval, explainability, speech generation, and full-duplex interaction, including speaker-attributed ASR, speech-to-speech conversion, and explainable speech emotion recognition [2604.03074] [2306.07944] [2509.24187].

## 1. Terminological scope and lineage

Across recent arXiv literature, “Speech ReaLLM” no longer refers to a single invariant model family. It has instead come to name a cluster of architectures that preserve LLM-style autoregressive reasoning while extending it to continuous audio, structured temporal decisions, or speech-native generation. The 2024 formulation is the canonical one: a streaming ASR system with decoder-only generation and BLANK-mediated real-time interleaving [2406.09569]. Later papers use the phrase more expansively for retrieval-augmented speech understanding, explainable SpeechLMs, or as a target notion for a “full Speech Reasoning LLM” [2306.07944] [2509.24187] [2604.03074].

| Paper | Use of “Speech ReaLLM” | Core emphasis |
|---|---|---|
| “Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time” [2406.09569] | Explicit model name | Streaming ASR without explicit end-pointing |
| “Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding” [2306.07944] | “Speech ReaLLM family of models” in the technical summary | Speech understanding with adapter and retriever |
| “Reasoning Beyond Majority Vote: An Explainable SpeechLM Framework for Speech Emotion Recognition” [2509.24187] | “Speech ReaLLM” in the technical summary | Transcript, emotion label, and rationale generation |
| “Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR” [2604.03074] | “paving the way toward a full Speech Reasoning LLM (Speech ReaLLM)” | Agentic multi-turn temporal reasoning |
| “Raon-Speech Technical Report” [2605.23912] | Raon-SpeechChat as “Speech ReaLLM” in the technical summary | Speech understanding, generation, and full-duplex conversation |

A common misconception is that Speech ReaLLM names only the 2024 streaming-ASR architecture. The literature shows a broader usage: some works retain the original emphasis on real-time interleaving, whereas others treat Speech ReaLLM as a broader program for speech-native reasoning, interaction, and generation [2406.09569] [2604.03074].

## 2. Decoder-only streaming ASR by teaching the flow of time

The original Speech ReaLLM architecture was proposed as the first “decoder-only” ASR architecture designed to handle continuous audio without explicit end-pointing [2406.09569]. Its central idea is to reproduce an RNN-T-like blank/non-blank alternation inside a decoder-only LLM stack. A streaming speech encoder, such as an Emformer or Conformer, consumes raw audio frame-by-frame or chunk-by-chunk and outputs a sequence of fixed-rate embeddings. These embeddings are interleaved with text token embeddings inside a stack of Transformer-decoder layers in a Llama-2 style decoder [2406.09569].

The model receives 80-dim log-Mel filterbank features every 20 ms, packs them into 240 ms chunks, and processes them with a 20-layer Streaming Conformer. The resulting speech embeddings are denoted $\{e_1,e_2,\dots,e_T\}$ with $e_t\in\mathbb{R}^d$, where $d=256$ in the reported experiments [2406.09569]. At each real-time step, after receiving $e_t$, the decoder is invoked immediately and generates zero or more text tokens until it predicts BLANK. BLANK is treated as just another token in the decoder’s vocabulary, so whenever no new words can yet be emitted, the model predicts BLANK and terminates the inner generation loop [2406.09569].

Formally, the decoding process is written as
$$
w_{t,k}=\arg\max_{w\in\mathcal V} P(w\mid h_{t,k-1}),
$$
with iteration continuing until $w_{t,k}=\text{BLANK}$ [2406.09569]. The significance of this construction is architectural rather than merely algorithmic: it allows a decoder-only LLM to behave like a real-time transducer while remaining within an autoregressive next-token framework. The paper explicitly frames this as a special case of a more general ReaLLM approach in which generation occurs after every input token received in real time, often producing an empty response [2406.09569].

## 3. Training, alignment, and adaptation to pretrained LLMs

Speech ReaLLM does not use the forward-backward marginalization of a conventional RNN-T. Instead, training relies on a fixed monotonic alignment “teacher” generated by an external CTC model [2406.09569]. For each utterance, the CTC teacher provides a frame-level alignment
$$
\alpha=[\alpha_1,\dots,\alpha_{T'}],\quad \alpha_t\in\{\text{word}_j\}\cup\{\text{BLANK}\},
$$
where each $\alpha_t$ corresponds to one 240 ms chunk. Speech embeddings and token embeddings are then interleaved according to $\alpha$, and the model is trained with a standard cross-entropy loss over the resulting sequence [2406.09569]:
$$
\mathcal{L}=-\sum_{n=1}^{T'+K}\ln P(\hat y_n\mid \mathbf h_n).
$$

This training strategy is important because it shows that a decoder-only LLM can be taught the timing structure of streaming ASR using fixed alignments plus cross-entropy, rather than a transducer-specific objective. The paper’s analysis argues that the decoder thereby learns to represent and reproduce the flow of time [2406.09569].

The same work also demonstrates compatibility with a large pretrained language model. A frozen Llama-2-7B decoder was adapted by inserting a small two-layer MLP that maps the 256-dim speech embeddings into the 4096-dim Llama embedding space, repurposing the underscore token “\_” as BLANK, and adding rank-16 LoRA adapters on all Transformer projection matrices. All original Llama-2 weights and the speech encoder were frozen; only the LoRA adapters and the embedding adapter MLP were trained. The system was fine-tuned on LibriSpeech (960 h) for 27 epochs with the same cross-entropy loss [2406.09569]. This is a narrow but consequential result: it shows that streaming ASR behavior can be induced in a pretrained 7B decoder without rearchitecting it into a conventional transducer.

## 4. Empirical characteristics of the original system

On LibriSpeech “test,” the reported 80M Speech ReaLLM achieved WERs of 3.0% and 7.4% in real time, without an external LM or auxiliary loss [2406.09569]. In the same comparison, a non-streaming Speech LLM of the same size achieved 4.8% and 8.0%, and an RNN-T achieved 3.6% and 9.4% [2406.09569]. The pre-trained Llama-7B adaptation achieved 4.7% and 9.1% [2406.09569]. These numbers establish that the streaming decoder-only formulation was competitive with same-scale streaming baselines while remaining close to a larger attention-based encoder-decoder reference.

| System | WER test clean | WER test other |
|---|---:|---:|
| Speech ReaLLM | 3.0% | 7.4% |
| Speech LLM (non-streaming) | 4.8% | 8.0% |
| RNN-T | 3.6% | 9.4% |
| + pre-trained Llama-7B | 4.7% | 9.1% |

Inference cost was reported in real-time factors. Speech ReaLLM achieved RTF 0.94 with beam size 4 and greedy RTF 0.79, while maintaining dev WER 2.7/7.6 and 2.8/7.8 respectively [2406.09569]. The paper also provides several ablations that clarify what the model learned. A non-streaming “Time-Aligned Speech LLM” trained with BLANK symbols, but still seeing all speech up front, reached Alignment Error Rate 4.9% and 6.5% and Length Error Rate 1.0% and 1.4% on dev clean/other, indicating that a decoder-only LLM can indeed learn and reproduce the passage of time via BLANK tokens [2406.09569].

A future-context ablation further showed that reducing the Conformer’s right context by 480 ms and delaying labels by two decoder tokens degraded WER from 2.7% and 7.6% to 4.4% and 9.1%, supporting the conclusion that future audio context is better handled in the encoder than by forcing the decoder to “look ahead” [2406.09569]. Length robustness was equally notable: on the longest 100 utterances in dev, Speech ReaLLM stayed at 2.6% and 7.4%, whereas the non-streaming Speech LLM rose to 17.8% and 22.0%, and the time-aligned variant to 17.3% and 17.4% [2406.09569]. This indicates that BLANK-mediated interleaving is not merely a latency device; it also changes the effective context-management behavior of the decoder.

## 5. Expansion toward speech reasoning, structured understanding, and explainability

Later work generalized the Speech ReaLLM idea from streaming transcription to broader speech reasoning. In “Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding,” the technical summary explicitly describes a “Speech ReaLLM family of models” built from three components: CTC-based blank-filtering, a Speech2Text adapter that maps compressed speech into the text embedding space of a frozen T5 (XXL), and, in the retrieval-augmented ReSLM variant, a Speech2Entity retriever that prepends relevant entities to the LLM input [2306.07944]. On Speech MultiWOZ, the reported dialog state tracking performance improved from 24.7% JGA for SLM without adapter or retrieval, to 28.4% with the Speech2Text adapter, and to 34.6% for ReSLM; WER improved from 11.5% to 9.2% to 8.5% across those variants [2306.07944]. Here, Speech ReaLLM is not primarily about streaming, but about aligning speech and language representations and handling rare entities through retrieval.

The same broadening appears in explainable speech emotion recognition. “Reasoning Beyond Majority Vote: An Explainable SpeechLM Framework for Speech Emotion Recognition” presents a two-stage pipeline in which a frozen Qwen2-Audio speech encoder and a LoRA-augmented Qwen2-Audio-7B-Instruct first produce a transcript and then emit both an emotion label and a concise natural-language rationale in a single autoregressive pass [2509.24187]. On MSP-Podcast v1.12, the reported open-form Macro-F1 rose from 30.11 to 36.14 under majority-vote scoring and from 56.78 to 66.78 under annotator-aware scoring when rationale supervision was added; human evaluation reported 69.1% overall acceptance of generated rationales [2509.24187]. In this usage, Speech ReaLLM denotes a reasoning-centric SpeechLM in which explanation is part of the generated target string.

A further extension is “Speaker-Reasoner,” an end-to-end Speech LLM with agentic multi-turn temporal reasoning for timestamped speaker-attributed ASR [2604.03074]. Rather than single-pass SOT decoding, the model performs a global turn over the entire audio to predict speaker count and gender summary, followed by local turns over time slices that jointly emit speaker IDs, gender tags, start and end timestamps, and transcriptions. Its reasoning state is summarized by
$$
R_t=f_\theta(R_{t-1},A_t,S_t),
$$
where the model attends over projected acoustic tokens for the current slice, previously generated tokens, and serialized speaker-aware cache entries [2604.03074]. On AISHELL-4 Eval, it reported DER 5.26%, CER 13.83%, cpCER 14.73%, and $\Delta cp$ 0.90%; on AliMeeting-Far test, DER 7.34%, CER 20.57%, cpCER 20.43%, and $\Delta cp$ –0.14% [2604.03074]. The paper states that these results pave the way toward a full Speech Reasoning LLM, explicitly abbreviating that target as Speech ReaLLM [2604.03074].

## 6. Speech-to-speech, full-duplex interaction, and the broader ReaLLM agenda

The most expansive interpretation of Speech ReaLLM concerns systems that both understand and generate speech while preserving text-side reasoning. Several 2025–2026 papers address this transition from streaming ASR to full speech interaction. “SimulS2S-LLM” proposes simultaneous inference for speech-to-speech translation using an offline-trained speech LLM, CIF-extracted boundary-aware speech prompt vectors, a wait-$k$ test-time policy, and incremental beam search over discrete output speech tokens [2504.15509]. On CVSS, it reported, for example, Es→En at ATD≈3.44 s with ASR-BLEU ≈ 26.3 versus StreamSpeech ≈ 22.9, and the abstract summarizes this as improving ASR-BLEU scores by 3 points at similar latency [2504.15509].

“PRIME-Speech” takes a different position: adding speech output should not overwrite the speech-to-text reasoning path of a strong frozen backbone [2606.30944]. Its frozen-backbone S2S conversion framework attaches a causal audio post-decoder synchronized with intermediate hidden states of the frozen backbone, uses mixed hidden-state, text, and audio-history conditioning, and introduces multi-token prediction to reduce the effective codec prediction rate [2606.30944]. The paper reports FLEURS BLEU 31.40 → 33.24 for S2T versus S2S under ASR evaluation, first-token-to-first-audio latency dropping from 1.07 s to 0.39 s with MTP $k=4$, and a failure mode in multi-turn dialogue where, without audio-cache reset, WER explodes > 100% by turn 5 [2606.30944]. This directly addresses a recurrent concern in Speech ReaLLM design: adding speech generation can degrade S2T behavior unless the backbone is protected.

“MOSS-Speech” pushes further toward a “true speech-to-speech” large language model without text guidance [2510.00499]. It combines a modality-based layer-splitting architecture with a frozen pre-training strategy on top of Qwen-3-8B, uses the first 32 layers for joint fusion, and then branches into separate text and speech heads [2510.00499]. The paper reports 84.87 on spoken StoryCloze, 90.32 on zh-StoryCloze, and 67.19/69.53 on MMLU/CMMLU in pre-training evaluation, while in supervised fine-tuning it reports LLaMA-QA 77.33/63.67 for S→T/S→S and UTMOS 4.37 [2510.00499]. This suggests a shift from text-guided speech systems toward speech-native generation that still preserves competitive text performance.

The most directly full-duplex formulation in the supplied literature is Raon-Speech and Raon-SpeechChat. Raon-Speech transforms a pretrained Qwen3-VL-8B-Instruct backbone into a 9.1 B SpeechLM with an AuT speech encoder, Mimi codec, a speech-generation expert, residual code predictor, and speaker encoder; Raon-SpeechChat replaces the non-causal encoder with a causal streaming Voxtral encoder and reaches 9.8 B parameters [2605.23912]. The report states that Raon-Speech trains on 1.38M hours of highly curated English and Korean speech and text datasets through speech modules alignment, end-to-end SpeechLM pre-training with knowledge distillation, and multi-task preference optimization-based post-training, while Raon-SpeechChat continues training on 119K hours of time-aligned real and synthetic dialogue data [2605.23912]. Across 42 benchmarks, the report gives English Libri-clean WER 1.44, Libri-other WER 2.89, VoiceBench SpokenQA 76.79, and MMLU-Pro 64.05 for Raon-Speech, and on FDB v1.0 it reports for Raon-SpeechChat Candor TOR 0.213 for pause handling, backchannel frequency 0.081, smooth turn-take TOR 0.832, and user interruption TOR 0.980 [2605.23912]. In the technical summary, this full-duplex extension is explicitly described as “Speech ReaLLM” [2605.23912].

A broader implication of these later works is that Speech ReaLLM has evolved from a narrowly defined real-time ASR mechanism into a research program organized around four persistent constraints: preserving a pretrained backbone’s text-side capabilities, maintaining temporal fidelity under streaming or multi-turn conditions, handling structure beyond plain transcription such as speaker identity or rationale generation, and producing speech output without reintroducing a rigid ASR→LLM→TTS bottleneck [2406.09569] [2604.03074] [2606.30944] [2605.23912].

Source: https://www.emergentmind.com/topics/speech-reallm