Spoken Language Reasoning

• Spoken language reasoning is the capacity of computational systems to perform structured inference over spoken audio while generating context-aware responses.
• It integrates advances in speech recognition, reinforcement learning, and multimodal modeling to tackle the modality reasoning gap between spoken and text-based inputs.
• Emerging architectures, such as TARS and dual-brain models, use reward alignment and interleaved processing to enhance accuracy across diverse benchmarks.

Spoken language reasoning refers to the capacity of computational systems—principally LLMs and speech LLMs—to perform structured, stepwise, or inference-driven reasoning directly over spoken input, and to generate explainable, accurate, and listener-appropriate responses through speech. This domain synthesizes advances in speech recognition, linguistic theory, reinforcement learning, and multimodal modeling to address the inherent challenges of real-time, audio-centric, and context-rich communication. Empirical work has demonstrated a persistent and significant ‘modality reasoning gap’ between the reasoning ability of models on text versus speech, motivating new architectures, benchmarks, training algorithms, and evaluation standards to close this divide (Wang et al., 9 Jan 2026).

1. The Modality Reasoning Gap: Definition and Evidence

The modality reasoning gap denotes the consistent shortfall in reasoning performance observed when identical LLM backbones are fed spoken (audio) input compared to clean, tokenized text. Modern speech LLMs typically use a three-stage architecture: a frozen speech encoder, a lightweight projection layer to embed acoustic features in the LLM token space, and a decoder-only text LLM backbone (Wang et al., 9 Jan 2026). On benchmarks such as MMSU and OBQA, the same speech LLM achieves considerably lower reasoning accuracy with speech than with text.

This gap arises even with accurate initial speech-text alignment, due to two principal compounding effects:

• Representational drift: As representations propagate through the Transformer stack, small input mismatches expand into diverging hidden state trajectories ($$H^{(l)}_{\text{speech}} \not\approx H^{(l)}_{\text{text}}$$ for $l=1...L$), undermining subsequent reasoning.
• Behavior deviations: The final outputs from speech-conditioned inference ($y_{\text{speech}}$) often depart semantically from the intended chain-of-thought and answers predicted by text-conditioned inference ($y_{\text{text}}$).

Quantification of the gap uses the Modality Recovery Rate (MRR): $$\text{MRR}(\pi_\theta) = \frac{\mathbb{E}_{q\in\mathcal{D}}[\mathcal{S}(y_{\text{speech}})]}{\mathbb{E}_{q\in\mathcal{D}}[\mathcal{S}(y_{\text{text}}^{\text{base}})]} \times 100\%,$$ where $\mathcal{S}(\cdot)$ is a task-level score, e.g., QA accuracy. Typical pre-alignment MRR values are 91–93%, confirming a nontrivial degradation in spoken reasoning (Wang et al., 9 Jan 2026).

2. Architectures and Benchmarking for Spoken Reasoning

End-to-End and Cascade Systems

• End-to-end speech LLMs receive audio waveforms, encode them, and generate text or speech outputs within a unified model (e.g., Qwen2.5-Omni-7B, Phi-4-Multimodal-Instruct-7B) (Wang et al., 9 Jan 2026, 2505.15000).
• Cascade systems first transcribe audio via ASR (e.g., Whisper-large-v3), then apply a text LLM or math-specialized LLM to the transcript. These typically outperform end-to-end LLMs on arithmetic and knowledge-intensive spoken tasks (2505.15000).

Reasoning-Oriented Benchmark Suites

• MMSU: 5,000 audio QA triplets, spanning perceptual, semantic, phonological, and paralinguistic reasoning—systematically constructed from linguistic subfields and covering 47 granular tasks (Wang et al., 5 Jun 2025).
• Spoken-MQA: Over 2,700 audio math QA problems, segmented into arithmetic, single/multi-step contextual, and knowledge reasoning. Strict verbal disambiguation ensures relevance to speech input, not just textual parsing (2505.15000).
• VoiceBench/VERA/WavBench/SpeechR: Benchmarks targeting the comparison of reasoning accuracy under strictly controlled speech versus text conditions, and extending to naturalistic dialogue, paralinguistics, normative, and multi-step procedural inference (Li et al., 12 Feb 2026, Lin et al., 30 Sep 2025, Yang et al., 4 Aug 2025).

Performance assessment consistently shows:

Model	Reasoning Accuracy (Speech)	Text Baseline	Gap	Source
Qwen2.5-Omni-7B	61.5–70.1% (MMSU)	67.9–75.8%	5–8%	(Wang et al., 9 Jan 2026)
FT-Phi4-MM-6B	56–89% (SMQA)	~90%	1–19%	(2505.15000)
GPT-4o-Audio	Up to 90.7% (contextual)	92%+ (text)	<5%	(2505.15000)

Cascade systems may reach up to 81% on Spoken-MQA overall, while end-to-end models lag except when heavily finetuned (2505.15000).

3. Algorithmic Innovations for Neural Reasoning in Speech

State-of-the-art approaches address the reasoning gap using three broad categories:

a. Trajectory/Reward Alignment (TARS)

TARS (Trajectory Alignment for Reasoning in Speech) is an RL framework that minimizes representational and behavioral divergence by incorporating two dense alignment rewards:

• Representation alignment: Layer-wise cosine similarity between $\bar h^{(l)}_{\text{speech}}$ and $\bar h^{(l)}_{\text{text}}$, averaged across layers.
• Behavior alignment: Cosine similarity between semantic embeddings of speech- and text-conditioned model outputs.

The total reward,

$$R_{\text{total}} = R_{\text{base}} + \alpha R_{\text{repr}} + \beta R_{\text{beh}},$$

with base reward reflecting task accuracy and format. TARS achieves MRR of 98.9–100.5%, closing nearly the entire gap without degrading ASR ability (Wang et al., 9 Jan 2026).

b. Streamed and Interleaved CoT (SHANKS, STITCH, Mini-Omni-Reasoner, Mind-Paced Speaking)

• SHANKS enables simultaneous hearing and unspoken chain-of-thought reasoning by chunking streaming input (typically 4s per chunk), updating an incremental context, and conditionally triggering interruptions or tool calls before user speech has concluded. This yields up to 37% higher “interruption accuracy” versus baseline (Chiang et al., 8 Oct 2025).
• STITCH alternates generation of reasoning and speech output chunks. Reasoning windows ($N_{\rm reason}$ tokens) are scheduled during audio playback, allowing internal thought to progress alongside speech synthesis without increased latency. On math QA, this approach matches full CoT accuracy while minimizing pre-speech delay (Chiang et al., 21 Jul 2025).
• Mini-Omni-Reasoner implements token-level “thinking-in-speaking,” interleaving 2 spoken response tokens with 8 reasoning tokens repeatedly. This yields zero extra decoding latency and produces concise, semantically aligned spoken output (Xie et al., 18 Aug 2025).
• Mind-Paced Speaking (MPS) employs a dual-brain architecture: a Formulation Brain emits CoT reasoning segments, and an Articulation Brain generates spoken response segments in low-latency synchronization. MPS achieves nearly 93% accuracy on Spoken-MQA with negligible added wait (Wu et al., 10 Oct 2025).

c. Decoupled Reasoning and Speech (Think–Verbalize–Speak, Slot-Filling Reasoning, Modular SWMs)

• Think–Verbalize–Speak separates high-fidelity reasoning from its speech rendering. The “Think” LLM produces full stepwise reasoning, which is then verbalized by a trained summary model (ReVerT), significantly improving speech naturalness and listener comprehension with minimal accuracy loss (Woo et al., 19 Sep 2025).
• Slot Filling as Reasoning decomposes SLU into multi-step CoT (ASR, span selection, justification, JSON extraction), with hybrid models trained to support both direct and CoT-driven output. Explicit reasoning steps increase slot-filling F1 by up to 11% in medium-scale LLMs (Hacioglu et al., 22 Oct 2025).
• Speech World Model (SWM) adopts a cognitive DAG with modules for world model, affect, speech act, pragmatics. Posterior sampling, instruction-tuned LLMs, and counterfactual interventions yield interpretable, causally-grounded reasoning over spoken input (Zhou et al., 5 Dec 2025).

4. Analysis: Failure Modes, Paralinguistic Reasoning, and Multimodal Expansion

Failure Modes

Comprehensive benchmarks (VERA, WavBench, SpeechR, MMSU) show that:

• Shortfalls concentrate in: multi-step logic, math, knowledge-oriented reasoning, and paralinguistic tasks (e.g., sarcasm, emotion-grounded inference) (Wang et al., 5 Jun 2025, Li et al., 12 Feb 2026, Yang et al., 4 Aug 2025).
• Error typology: Native streaming models overproduce fluent but incorrect conclusions; cascades introduce grounding errors; end-to-end systems exhibit off-target/ refusal spikes (Lin et al., 30 Sep 2025).

Paralinguistic and Multimodal Reasoning

• Emotional and paralinguistic integration remains weak. Best open-source models achieve ∼39% accuracy on phonology and paralinguistics, versus 88% on semantic reasoning (MMSU) (Wang et al., 5 Jun 2025).
• IEAT (Injected Emotional-Attribution Thinking) fuses emotion and cause embeddings into the LLM's reasoning trace, boosting emotional reasoning on HumDial benchmarks (Wang et al., 8 Jan 2026).
• Models using explicit visual context (VRSLU, SilVar) combine image-based CA and audio with stepwise reasoning, yielding improved SLU and VQA outcomes (Wu et al., 24 Nov 2025, Pham et al., 2024).
• SpeechR and WavBench further decouple factual, procedural, and normative reasoning, exposing particularly low performance on subjective or pragmatic inferences (e.g., scam detection, moral judgment). SpeechR shows GPT-4o at 89% accuracy in procedural reasoning but only ~50% for normative tasks (Yang et al., 4 Aug 2025, Li et al., 12 Feb 2026).

5. Metrics, Task Taxonomies, and Practical Implications

Metrics

• Accuracy: Proportion of correct answers under forced-choice or generative scoring.
• MRR (Modality Recovery Rate): Retained reasoning performance under speech relative to text input (Wang et al., 9 Jan 2026).
• Latency: Token/frame count or second elapsed until audio output—central for real-time dialogue (Wu et al., 10 Oct 2025, Lin et al., 30 Sep 2025).
• Coherence and Chain-of-Thought (CoT) Quality: LLM-as-judge scores on correctness, logical relevance, and coherence (Yang et al., 4 Aug 2025, Li et al., 12 Feb 2026).
• Speech Suitability: Human/Lexical metrics for naturalness, word count, Flesch Reading Ease, parse-tree depth, non-vocalizable symbol count (Woo et al., 19 Sep 2025).
• Specialized task metrics: Frame parsing accuracy (intent + slot), BLEU/Cosine similarity for reasoning explanations (Wu et al., 24 Nov 2025).

Taxonomy of Reasoning Tasks

Category	Typical Example	Speech Benchmark
Factual	"Who won the race?"	SpeechR, MMSU
Procedural/Math	"What is 37+48?"	Spoken-MQA, WavBench-Pro
Semantic	"What is the referent of 'that'?"	MMSU (Deixis, Polysemy)
Paralinguistic	Sarcasm, speaker profiling	MMSU, SpeechR
Normative	"Is this SMS a scam?"	SpeechR
Multimodal	SLU with image, VQA	VRSLU, SilVar

6. Identified Challenges and Research Directions

Key persistent challenges:

• Parsing verbalized symbolic content: Speech LLMs show strong bias toward LaTeX/symbolic input and perform poorly on spoken mathematical expressions, with up to 17% performance drop (2505.15000).
• Representational drift: Layer-wise divergence in hidden states requires alignment via dense reward signals (Wang et al., 9 Jan 2026).
• Latency trade-offs: Real-time speech necessitates interleaved or chunked reasoning, as full CoT computation up-front incurs unacceptable delays (Wu et al., 10 Oct 2025, Chiang et al., 21 Jul 2025).
• Acoustic-paralinguistic integration: Low-level prosodic and emotion cues are only partially captured in current architectures (Wang et al., 5 Jun 2025, Yang et al., 4 Aug 2025).
• Causal and interpretable reasoning: Modular world models with explicit, causally-grounded inference chains provide interpretability and facilitate intervention/testing (Zhou et al., 5 Dec 2025).

Promising directions include:

• Reinforcement alignment: Policy optimization targeting trajectory-level representation and behavior similarity (TARS) (Wang et al., 9 Jan 2026).
• Multistream/simultaneous reasoning: SHANKS, STITCH, and “token-level thinking-in-speaking” (Chiang et al., 8 Oct 2025, Chiang et al., 21 Jul 2025, Xie et al., 18 Aug 2025).
• Integrated paralinguistic/object grounding: Datasets and models combining audio, visual, and profile-based context with explicit CoT (Wu et al., 24 Nov 2025, Pham et al., 2024).
• Automated data curation and continuous learning: Scalable pipelines to generate rare linguistic phenomenon and balance semantic, phonological, paralinguistic objectives (Wang et al., 5 Jun 2025).

7. Synthesis and Impact

Recent interdisciplinary advances have transformed spoken language reasoning from shallow ASR or factual lookup into deep, stepwise inference within real-world, audio-centric, and multi-modal contexts. While state-of-the-art techniques (e.g., TARS, MPS, SHANKS, Mini-Omni-Reasoner) approach text-model reasoning accuracy, systematic gaps remain—especially for mathematical, procedural, and paralinguistic tasks. Closing the modality reasoning gap entails refined architecture (modular, dual-brain, chunked reasoning), optimized reward alignment, and genre-spanning, semantically verified benchmarks. These insights form the basis for next-generation spoken AI agents that are not only conversationally fluent but also rigorous, trustworthy, and contextually aware reasoners (Wang et al., 9 Jan 2026, Wang et al., 5 Jun 2025, Xie et al., 18 Aug 2025, Wu et al., 10 Oct 2025, Chiang et al., 21 Jul 2025, Lin et al., 30 Sep 2025).