Five-Level Speech Understanding Framework

• The Five-Level Speech Understanding Framework is a hierarchical model decomposing speech comprehension from basic acoustic mapping to AGI-level integration.
• It details specific tasks, benchmarks, and evaluation metrics across levels, supporting advancements in ASR, paralinguistic analysis, and pragmatic inference.
• Empirical findings reveal that while lower levels achieve human-like performance, higher levels expose significant gaps in emotional and contextual understanding.

The Five-Level Speech Understanding Framework explicates the hierarchical evolution of computational systems for speech perception and comprehension, ranging from shallow surface mapping to full integration of implicit, emotional, and pragmatic knowledge. This framework is embedded as a unifying scaffold in multiple recent lines of research, notably the SAGI roadmap for superhuman speech understanding (Bu et al., 2024), the BoSS (Beyond-Semantic Speech) capability hierarchy (Wang et al., 23 Jul 2025), and the HPSU benchmark for human-level perception (Li et al., 28 Nov 2025). Across these resources, the five-level structure systematically decomposes speech understanding into increasingly complex and cognitively rich strata, each characterized by specific input–output mappings, capability requirements, representative tasks, evaluation metrics, and current research limitations.

1. Formal Hierarchy of Levels

The five-level scheme is instantiated in multiple forms but exhibits tight structural correspondence across leading works. The levels and their primary objectives can be summarized as follows:

Level	Core Objective	Representative Mapping
1	Surface acoustic and/or text mapping	$f_1$: waveform $\to$ text or attributes
2	Low-level acoustic or paralinguistic perception	$f_2$: waveform $\to$ paralinguistic features
3	Non-semantic or affective comprehension	$f_3$: waveform $\to$ emotion/context labels
4	Specialist/abstract acoustic reasoning	$f_4$: waveform $\to$ expert/domain inference
5	Integrated, AGI-level or pragmatic inference	$f_5$: waveform $\to$ generalized output/task

In the SAGI roadmap, Level 1 encompasses ASR and language identification, Level 2 extends to prosody and acoustic feature tracking, Level 3 models non-semantic/social/affective cues, Level 4 targets medical and forensic tasks requiring domain expertise, and Level 5 unifies these into an open-ended Speech AGI paradigm (Bu et al., 2024). The BoSS framework parallels this, building from command recognition (L1) to human-like social interaction (L5), operationalized by the inclusion of distinct "BoSS dimensions" (explicit semantics, contextual dynamics, affective cues, implicit semantics) at each stage (Wang et al., 23 Jul 2025). HPSU aligns these strata empirically, starting from speaker attribute recognition and progressing through paralinguistic, emotional, and pragmatic/subtextual reasoning (Li et al., 28 Nov 2025).

2. Tasks, Benchmarks, and Evaluation Metrics

Each level introduces distinct tasks and standardized benchmarks to capture the requisite processing capabilities:

Level 1 is measured by standard metrics such as Word Error Rate (WER) for ASR, K-way accuracy for language identification, and attribute accuracy for speaker characteristics. Datasets include LibriSpeech, Europarl-ST, CosyVoice, and VCTK (Bu et al., 2024, Li et al., 28 Nov 2025). Human labeling serves as the gold standard, but proxy measurements using Whisper-v3 (2.44% WER) and Qwen2-Audio (4.63% WER) are common where human WER cannot be obtained (Bu et al., 2024). HPSU reports human accuracy on speaker attributes at 80.3%, with top models trailing substantially (Li et al., 28 Nov 2025).

Level 2 advances to binary or categorical classification of acoustic events—volume modulation, pitch band distinctions, and binaural localization. Performance is typically at ceiling for humans but remains near random for open-source speech LLMs (e.g., Qwen2-Audio: 48.96% volume, 50.00% pitch; SALMONN: ~50%) (Bu et al., 2024). BoSS similarly restricts L2 to shallow contextual dynamics and limited dialogue slot-tracking, evaluated by slot and task-completion rates (Wang et al., 23 Jul 2025).

Level 3 encompasses emotion recognition, speaker age/gender, scene classification, and emotion-conditioned translation. Datasets span RAVDESS, MS-SNSD, AIR-Bench, and CosyVoice. Accuracy is at 85.5% for humans on paralinguistic/emotional content (HPSU), but ranges from 49–62% for state-of-the-art open-source models (Li et al., 28 Nov 2025).

Level 4 targets specialized domains: COVID-19 cough detection, cough type and severity on COUGHVID, complex emotion shifts, nonverbal behavior, and vocal/text mismatches. Human performance is moderate (e.g., COVID-19 risk detection: 60.6%), with model performance significantly below random for the most challenging medical tasks (Bu et al., 2024). BoSS L4 integrates affective and implicit semantics into end-to-end modeling, evaluated by interpretation accuracy in emotion-rich tasks (Wang et al., 23 Jul 2025).

Level 5 is benchmarked by holistic or AGI-scale metrics such as coaching or detective reasoning ratings, deep intent/subtext detection, and scene comprehension. HPSU composite scores are 86.9% for humans versus 62–64% for top LLMs (Li et al., 28 Nov 2025). BoSS L5 measures the preservation of mutual information and relevance (e.g., maximizing $\to$0 under a KL-bound) and social responsiveness (Wang et al., 23 Jul 2025).

3. Methodological and Theoretical Foundations

All frameworks ground level distinction in the structure and preservation of information beyond pure textual semantics. In BoSS, explicit semantics ($\to$1), affective cues ($\to$2), contextual dynamics ($\to$3), and implicit semantics ($\to$4) are integrated into a multidimensional observation vector $\to$5 (Wang et al., 23 Jul 2025). Optimal hypothesis selection at each timestep is formalized via a cognitive relevance objective: $\to$6 where $\to$7 is cognitive effect and $\to$8 is processing effort (per Relevance Theory) (Wang et al., 23 Jul 2025). This objective is implemented within a Hidden Markov Model (HMM), with Viterbi decoding over temporal hypotheses.

In end-to-end SLMs, waveform encoding ($\to$9) produces embeddings $f_2$0 that respect all BoSS dimensions, with a LLM $f_2$1 and TTS decoder $f_2$2 ensuring that mutual information between input $f_2$3 and LLM embedding $f_2$4 is maximized, as formalized by KL-based bounds: $f_2$5 Alignment between speech encoding and output decoding distributions is necessary to prevent loss of paralinguistic and pragmatic content (Wang et al., 23 Jul 2025).

4. Empirical Performance and Model Limitations

Systematic evaluation across these levels, as reported in HPSU, BoSS, and the SAGI roadmap, reveals that current LLM-based models substantially underperform compared to human annotators, especially above Level 3. For instance, while Level 1 (surface ASR and attribute extraction) is near saturation for targeted ASR tasks, open-source models struggle with non-standard audio (e.g., medical or lyrics transcription) and with paralinguistic and non-semantic content (Levels 2–4) (Bu et al., 2024, Li et al., 28 Nov 2025).

The following table summarizes human and model accuracy, averaged across levels (from HPSU (Li et al., 28 Nov 2025)):

Framework Level	Human (%)	Gemini 2.5 Pro (%)	Qwen-2.5-Omni (%)
1. Speaker Attributes	80.3	67.8	57.6
3. Emotion Recognition	85.5	61.0	53.0
4. Emotion/Nonverbal	85.6	53.9	55.6
5. Semantic Inference	86.9	62.2	63.7
Overall Avg	87.3	62.6	60.0

Common failure cases include confusion among similar accents, ASR dropouts in noise, emotion misclassification in ambiguous prosody, and significant deficits in detecting subtext or sarcasm. In SAGI evaluation, even leading models such as Qwen2-Audio and GPT-4o are unable to approach human performance in medical speech diagnosis or zero/few-shot inference, with system refusal or near-chance performance being frequent (Bu et al., 2024).

5. Annotation Pipelines and Data Fusion

The HPSU benchmark introduces a three-stage, semi-automatic annotation process to balance scale and reliability in highly variable real-world spoken language (Li et al., 28 Nov 2025). This involves:

• Data collection and preprocessing with audio/video denoising and ASR for transcript generation.
• Multimodal cross-validation using audio, text, and vision embeddings to hypothesize speaker and emotional attributes; consistency is measured via cross-model cosine similarity.
• Hierarchical fusion with QWQ inference and expert verification. Human triage ensures at least triple-agreement, achieving consensus in over 81% of cases.

This cross-modal fusion yields datasets in which annotation quality and context capture surpass purely text- or audio-based pipelines, providing a firmer empirical foundation for level-based benchmarks.

6. Principal Challenges and Research Directions

Key limitations across reports include:

• Paralinguistic collapse: Speech LLMs frequently discard low-level acoustic information, evidenced by encoder embeddings with high intra-category cosine similarity, rendering them insensitive to gender and emotion when varied in style (Bu et al., 2024).
• Instruction sensitivity: Systems exhibit marked prompt-sensitivity; even minor perturbations or shifts from text to speech prompts can cause performance degradation (Bu et al., 2024).
• Data imbalance and scarcity: Specialist acoustic tasks and emotion-rich dialogues are under-represented in pretraining corpora, impeding model generalization to higher levels (Bu et al., 2024, Li et al., 28 Nov 2025).
• End-to-end alignment: Maintaining BoSS dimensions through SLM architectures without semantic bottlenecking remains an unsolved challenge (Wang et al., 23 Jul 2025).

Recommended directions for future research include reinforcement of BoSS-relevant data, design of acoustic encoders that resist text-centric collapse, joint multi-modal objectives, scaling up audio-aware LLMs, and exploration of self-supervised or symbolic approaches to abstract acoustic knowledge (Bu et al., 2024, Wang et al., 23 Jul 2025).

7. Synthesis and Comparative Context

The Five-Level Speech Understanding Framework now constitutes a de facto standard for benchmarking and conceptualizing progression toward superhuman or human-level machine speech understanding. SAGI, BoSS, and HPSU collectively provide a unified gradation of abilities, from surface mapping through paralinguistic, affective, and ultimately pragmatic-social reasoning. Each level imposes precise functional mappings, data and architecture requirements, and model selection criteria (e.g., maximizing cognitive relevance $f_2$6, mutual information preservation). While substantial progress has been made at the lowest levels, human-machine parity remains elusive for affective, contextual, and pragmatic sub-components, and integration of all dimensions is required for AGI-level spoken communication (Bu et al., 2024, Wang et al., 23 Jul 2025, Li et al., 28 Nov 2025).