SpeechQualityLLM: Unified Speech Quality Assessment

• SpeechQualityLLM is a multimodal LLM architecture that integrates speech and text processing to deliver comprehensive assessments of speech quality and pronunciation.
• It employs instruction tuning and joint audio-text representations to generate detailed numeric ratings and natural language feedback using advanced deep learning techniques.
• The system demonstrates robust performance on MOS, L2 pronunciation grading, and multi-aspect diagnostic reporting, supporting applications from CAPT to synthetic speech detection.

SpeechQualityLLM refers to a family of LLM–centric architectures designed to assess speech quality and pronunciation at scale, unifying classic Mean Opinion Score (MOS) prediction, multidimensional perceptual scoring, and conversational rationales within a single, instruction-driven, multimodal framework. These systems leverage joint speech–text representation and instruction tuning to yield rubric-aligned numeric ratings and explanatory feedback, and have been evaluated for both general speech quality assessment and specialized scenarios such as L2 pronunciation grading and multidimensional diagnostic reporting (Parikh et al., 20 Jan 2026, Monjur et al., 9 Dec 2025).

1. System Architecture and Instruction Tuning

SpeechQualityLLM centers around an audio encoder–decoder design, with the encoder typically based on a variant of Whisper-large (for raw waveform ingestion and log-mel spectrogram embedding) and the text decoder built on a GPT-style transformer backbone, such as Qwen-7B or Llama-3.1-8B. Variable-length audio is converted to a series of spectrogram frames $X = [x_1, ..., x_T] \in \mathbb{R}^{T \times F}$, downsampled to $A = E_a(X) \in \mathbb{R}^{L \times d}$ via Conv+Transformer stacks. These audio embeddings are ranged along with discrete prompt embeddings $P = [p_1,...,p_M]$ and decoded in an interleaved fashion, yielding the output distribution

$$P_\theta(Y \mid A, P) = \prod_{n=1}^N P_\theta(y_n \mid y_{<n}, A, P)$$

where $Y = [y_1, ..., y_N]$ denotes the output tokens (Parikh et al., 20 Jan 2026, Monjur et al., 9 Dec 2025).

Instruction tuning is performed on diverse speech tasks (ASR, translation, captioning), followed by Direct Preference Optimization (DPO) to prefer human-aligned completions. In specific instantiations, only the LoRA adapters on query/key projections or a limited subset of encoder parameters are updated to promote parameter efficiency (Monjur et al., 9 Dec 2025).

In pronunciation assessment applications, additional layers or adapters may be used to align pre-trained audio encoder representations into the LLM's native latent space, with prompt strategies combining explicit rubrics, sentence-level targets, and context sentences (Parikh et al., 20 Jan 2026, Fu et al., 2024).

2. Multi-Aspect Evaluation Paradigms and Prompt Design

SpeechQualityLLM generalizes beyond scalar MOS or accuracy regression by supporting multi-aspect, rubric-driven rating via natural-language prompting. For L2 pronunciation, the model is prompted to provide numeric (0–10) scores for multiple rubrics—including accuracy (phonemic correctness), fluency (pausal and rhythm coherence), prosody (intonation, stress, rhythm), and completeness (proportion of utterance realized)—returning JSON, tabular, or textual outputs (Parikh et al., 20 Jan 2026): $A = E_a(X) \in \mathbb{R}^{L \times d}$4

For MOS and perceptual dimensions on corpora such as NISQA, prompts are templated for both numeric and categorical tasks: | Task Type | Prompt Pattern | | -------------- | -------------------------------------------------------------- | | MOS-numeric | "Rate overall quality on 1–5." | | Dim-numeric | "How noisy/color/timing/... on 1–5?" | | Multi-dim | "Give MOS and all four dims in one reply." | | Explanatory | "Explain main artifacts, then give MOS." |

A separate set of templates supports natural-language, diagnostic or listener-profile–conditioned queries, allowing the model to justify ratings, enumerate artifacts, and simulate rater variability (Monjur et al., 9 Dec 2025).

3. Datasets, Evaluation Protocols, and Metrics

Benchmarking SpeechQualityLLM systems exploits large-scale, human-annotated corpora covering a breadth of quality signals:

• Speechocean762: L2 English, 5,000 rival utterances rated 0–10 on accuracy, fluency, prosody, completeness (Parikh et al., 20 Jan 2026).
• NISQA: MOS and four dimensions (noisiness, coloration, discontinuity, loudness) with tens of thousands of utterances (Monjur et al., 9 Dec 2025).
• QualiSpeech: 11 low-level aspects (e.g., noise, distortion, speed, continuity, naturalness) and interval-based structured descriptions (Wang et al., 26 Mar 2025).
• SpeechEval: Multilingual, 32k clips annotated for assessment, pairwise comparison, improvement suggestion, and deepfake detection (Wang et al., 16 Oct 2025).

Quantitative evaluation includes:

• Tolerance-based match for rubric ratings:

$$\mathrm{MatchRate}_i(k) = \frac{1}{N} \sum_{n=1}^N \mathbf{1}(\lvert s_{i,n}^{(m)} - s_{i,n}^{(h)}\rvert \le k)$$

• Correlation coefficients (PCC, SRCC) for numeric alignment:

$$r_i = \frac{\sum_n (s_{i,n}^{(m)} - \bar{s}_i^{(m)})(s_{i,n}^{(h)} - \bar{s}_i^{(h)})}{\sqrt{\sum_n (s_{i,n}^{(m)}-\bar{s}_i^{(m)})^2} \sqrt{\sum_n (s_{i,n}^{(h)}-\bar{s}_i^{(h)})^2}}$$

• Regression metrics: MAE, RMSE, MSE for MOS and subdimension prediction (Monjur et al., 9 Dec 2025).
• A/B Test Accuracy: predictive accuracy on forced-choice sample pairs (Chen et al., 27 Jan 2025).
• BLEU, ROUGE, sacreBLEU: overlap of generated diagnostic text with template-based or human explanations (Monjur et al., 9 Dec 2025, Chen et al., 27 Jan 2025).
• Interval and detection metrics: precision, recall, and temporal IoU for localizing annotated artifacts (Wang et al., 26 Mar 2025).

Task-specific evaluation highlights robust model–human agreement within ±2 points (tolerance) but reveals biases in low-score undercalling and rubric granularity, especially in completeness and error localization (Parikh et al., 20 Jan 2026, Wang et al., 26 Mar 2025).

4. Empirical Performance, Limitations, and Error Analysis

SpeechQualityLLMs consistently match or exceed classical regression models in mainline MOS estimation, with full-reference, finetuned variants achieving $\mathrm{MAE} \approx 0.41$, $r=0.86$ (NISQA), close to the $\mathrm{MAE} \sim 0.4$ of perceptual metrics like PESQ (Monjur et al., 9 Dec 2025). For L2 assessment, tolerance-based accuracy for accuracy/flunecy/prosody within ±2 is up to 89.5%, but exact rubric match rates are substantially lower, demonstrating a lack of fine-grained discrimination (Parikh et al., 20 Jan 2026).

Key limitations identified are:

• Central-value bias: strong tendency to cluster predictions in the mid to upper range, rarely assigning low/outlier scores.
• Granularity and low-score underprediction: failure to assign appropriately low ratings for severely impaired or erroneous samples, with utterances rated extremely low by humans almost never receiving comparable scores from the model.
• Rubric-specific misalignment: e.g., completeness often poorly matched between model and annotators (Parikh et al., 20 Jan 2026).
• Limited phonetic/error sensitivity: over-reliance on global acoustic cues obscures detailed phonemic or suprasegmental analysis, substantiated by case studies where major mispronunciation errors are overlooked.

In explanatory or natural-language dimensions, finetuned SpeechQualityLLMs are able to enumerate key artifact types and provide justified rationales with BLEU scores (explanatory tasks) up to 54.8 (sacreBLEU) (Monjur et al., 9 Dec 2025), but outputs can default to formulaic style without targeted instruction tuning and corpus diversity (Wang et al., 26 Mar 2025).

5. Recommendations, Enhancements, and Future Directions

Model improvement strategies are grounded in the analysis of bias, granularity, and interpretability issues:

• Prompt refinement: Explicitly instruct the model to attend to low-end scoring and specific error types, e.g., “If fewer than 50% of phonemes are correct, assign Accuracy ≤ 4” (Parikh et al., 20 Jan 2026).
• Score calibration: Apply mapping functions $A = E_a(X) \in \mathbb{R}^{L \times d}$0, via logistic or isotonic regression on a subset of human ratings, to minimize expected calibration error

$A = E_a(X) \in \mathbb{R}^{L \times d}$1

• Phonetic feature integration: Add an auxiliary phoneme posterior layer $A = E_a(X) \in \mathbb{R}^{L \times d}$2, or an auxiliary phoneme-level loss

$A = E_a(X) \in \mathbb{R}^{L \times d}$3

to make fine-grained errors more salient (Parikh et al., 20 Jan 2026).

• Leveraging multitask, multi-prompt corpora: Incorporate rich, structured datasets annotating low-level aspects, e.g., noise/interruption intervals and structured distortion descriptions (Wang et al., 26 Mar 2025), and train on multi-lingual, multi-domain resources (Wang et al., 16 Oct 2025).

Further directions include enhanced segment-level supervision, multitask continual learning with ASR, TTS, emotion, and diarization (Chen et al., 27 Jan 2025), and deployment of profile-conditioned or listener-adaptive prompting to reflect subjective perceptual variation (Monjur et al., 9 Dec 2025, Wang et al., 16 Oct 2025). Scalability is achievable by using LLMs as pseudo-raters for annotation in under-resourced domains (Cumlin et al., 8 Aug 2025).

6. Applications and Perspectives

SpeechQualityLLMs demonstrate utility in:

• Computer-Assisted Pronunciation Training (CAPT): Enabling rapid, large-scale, rubric-based scoring of L2 learner speech, with applicability to low-resource language contexts where manual annotation is scarce (Parikh et al., 20 Jan 2026).
• VoIP, Telephony, and Streaming Quality Monitoring: Replacing or supplementing MOS predictors such as PESQ/POLQA in automated pipelines, while giving practitioners flexible, rationale-rich QA interfaces (Monjur et al., 9 Dec 2025).
• Synthetic Speech and Deepfake Detection: Structured, chain-of-thought–based explanation and binary/continuous classification for real vs. synthetic speech (Wang et al., 16 Oct 2025).
• ASR and TTS System Comparison: Enabling interpretable benchmarking and A/B testing via forced-choice rationale generation and numeric scoring (Chen et al., 27 Jan 2025).
• Profile-Conditioned and Task-Adaptive Assessment: Simulating variability among raters (e.g., audiophile, hearing-impaired) and enabling detailed feedback for system improvement (Monjur et al., 9 Dec 2025).
• Cross-Lingual and Multi-Domain Evaluations: Extending automatic scoring and rich description to new languages and speech domains given suitable adaptation corpora (Wang et al., 16 Oct 2025, Cumlin et al., 8 Aug 2025).

SpeechQualityLLM systematically elevates speech assessment from scalar, opaque scoring to a framework combining numeric reliability, multidimensional feedback, and natural-language justification, with demonstrable accuracy competitive with domain-specific regressors and ready integration into speech-driven pipelines (Parikh et al., 20 Jan 2026, Monjur et al., 9 Dec 2025, Wang et al., 16 Oct 2025).