SpeechLMScore Evaluation

• SpeechLMScore is a reference-free, data-driven metric that quantifies speech quality and severity using unsupervised language modeling on discretized acoustic units.
• It extracts latent features via models like HuBERT, quantizes these into acoustic tokens, and employs an LSTM-based language model to compute speech naturalness.
• Experiments demonstrate its robustness with strong correlations to human ratings and minimal sensitivity to noise in clinical and spontaneous speech settings.

SpeechLMScore is a reference-free, data-driven metric for evaluating the quality, severity, or naturalness of speech using unsupervised LLMing on learned acoustic units. Unlike traditional approaches requiring either reference speech, transcripts, or hand-crafted features, SpeechLMScore quantifies how likely a speech signal is to conform to models of healthy, typical speech as learned by self-supervised representation learning and acoustic unit LLMing (AULM). This approach is motivated by the need for robust, ecologically valid evaluation—particularly in settings such as speech disorder assessment or spontaneous speech—all without reliance on manual annotation or reference data.

1. Core Principles and Methodology

SpeechLMScore exploits the capacity of self-supervised speech models, such as HuBERT-BASE-LS960H, to extract latent, temporally aligned representations from raw audio. The full computational chain involves three main stages:

1. Representation Extraction: Each input utterance $x_t$ is processed through a pretrained model (e.g., HuBERT) to yield a sequence of feature vectors $h_t = \text{HuBERT}(x_t)$; each $h_t$ encodes rich acoustic and phonetic information.
2. Quantization to Discrete Units: The continuous representations $h_t$ are discretized by k-means clustering or a similar partitioning mechanism, producing a sequence $d_t \in \{1, \dots, K\}$, where $K$ is the vocabulary of acoustic tokens ("units").
3. Unit LLMing and Scoring: An autoregressive LLM (typically an LSTM) is trained on large-scale healthy speech corpora (such as LibriLight) to estimate unit token probabilities:

$p(d_t \mid d_{<t})$

The scoring of a new utterance is defined via the negative average log-probability (i.e., perplexity):

$$\hat{x} = \exp\left( - \frac{1}{T} \sum_{t=1}^T \log p(d_t \mid d_{<t}) \right)$$

Here, lower perplexity (higher log-probability) indicates a speech pattern that conforms more closely to healthy, typical speech; higher values suggest greater deviation and thus more severe impairment.

This entirely reference-free process allows for automatic evaluation without transcripts or pathology labels, leveraging only large-scale healthy-speech pretraining and unsupervised inference.

2. Motivation and Distinction from Traditional Methods

Conventional speech severity evaluation typically relies on:

• Hand-crafted acoustic features (e.g., measures of shimmer, jitter, $F_0$ variation, Harmonics-to-Noise Ratio, Cepstral Peak Prominence).
• Reference-based metrics, often requiring the existence of a transcript or canonical speech sample, such as Phoneme Error Rate (PER) obtained via automatic speech recognition (ASR) alignment.

These approaches face notable limitations:

• Limited generalization: Such systems may capture dataset-specific artifacts or require domain-specific tuning.
• Application constraints: Many require controlled prompts, careful manual transcription, and are insensitive to the characteristics of spontaneous or pathological speech.
• Noise sensitivity: Hand-crafted features and ASR-based metrics can degrade sharply in noisy or ecologically realistic settings.

By contrast, SpeechLMScore's unsupervised design sidesteps both reference speech and hand labeling, instead leveraging data-driven regularities in continuous and spontaneous speech data. The methodology is motivated by the well-evidenced observation that automatic "naturalness" scores from learned models correlate strongly with manual severity assessments.

3. NKI-SpeechRT Dataset and Experimental Setup

To comprehensively evaluate SpeechLMScore and its robustness in clinical settings, the NKI-SpeechRT dataset is introduced, building on the NKI-CCRT corpus. This dataset comprises:

• Speaker population: 55 speakers, primarily native Dutch, recorded at up to five treatment-related time points (pre- and post-interventions) while reading a standardized story ("De vijvervrouw" by Godfried Bomans).
• Subjective ratings: Human severity (intelligibility) scores from recent SLP graduates, plus additional subjective noisiness scores.
• Ecological validity: Includes samples from diverse time points, capturing a wide range of natural and pathological speech.

The dataset provides the necessary foundation for systematically comparing reference-free and reference-based models under realistic, variable, and noisy conditions.

4. Quantitative Performance and Comparative Evaluation

Performance is evaluated chiefly by correlating SpeechLMScore with human listener ratings. Key findings include:

• NKI-SpeechRT Dataset:
  • SpeechLMScore achieved a Pearson correlation of $r = 0.3834$, outperforming all traditional non-intrusive acoustic metrics: HNR ($r = -0.2999$), WADA SNR ($r = -0.2852$), and others.
• NKI-OC-VC Dataset:
  • SpeechLMScore correlation reached $r = 0.6895$, very close to the best reference-based method (PER, $r = -0.9155$), despite requiring no references.
• Noise Robustness:
  • Correlation with subjective noise scores is negligible: $r = 0.0305$, $p = 0.6741$ for SpeechLMScore, indicating almost complete invariance to noise—a sharp contrast to features like WADA SNR.

These results demonstrate that SpeechLMScore narrows the performance gap between reference-based and reference-free methods and is significantly less sensitive to real-world noise and recording conditions.

5. Robustness to Noise and Pathways Toward Ecological Applicability

A primary requirement for clinical and field deployment is robustness to background noise and signal artifacts. SpeechLMScore exhibits minimal correlation with subjective noise ratings in both subjective tests and quantitative analyses. This is attributed to its dependence on learned linguistic structure and temporal context (as encoded in self-supervised representations and long-context LMs), which are less perturbed by background or mic-channel noise than low-level spectral features.

A plausible implication is that SpeechLMScore is eminently suitable for large-scale, real-world deployment, including in unscripted, spontaneous speech or settings with highly variable acoustic quality.

6. Implications, Limitations, and Future Directions

Implications and broader applications:

• Clinical utility: SpeechLMScore offers a scalable, effective tool for triaging, continuous monitoring, and generalization across diverse speaker populations, especially in the absence of reference speech or detailed manual annotation.
• Automatic assessment expansion: The methodology can be extended to multilingual or cross-domain settings, and is compatible with further advances in self-supervised modeling and acoustic unit discovery.

Limitations and avenues for improvement:

• Performance gap to reference-based models: Although narrowing, there remains a quantitative gap compared to reference-based methods in some datasets (e.g., PER).
• Representation dependence: Performance is contingent on the quality and domain-match of the pretrained unit LLM. Retraining or fine-tuning on larger or target-specific healthy data may provide further improvements.
• Potential for interpretability gains: Integration with phonetic posteriorgrams or alternative unit representations could facilitate interpretability and more granular source attribution for errors or severity phenomena.

Future research directions:

• Exploration of improved self-supervised representations, such as wav2vec or WavLM, and model distillation for increased robustness and generality.
• Application to broader speech assessment scenarios, including spontaneous conversation, multilingual datasets, or combined naturalness and intelligibility metrics.
• Clinical translation: Prospective studies to determine performance longitudinally and in direct clinical workflows.

SpeechLMScore thus represents a reference-free, robust, and empirically validated approach to automatic speech severity evaluation, with demonstrated superiority over conventional acoustic features and resilience against real-world acoustic variability (Halpern et al., 1 Oct 2025).