Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings

Abstract: The variations in vocal effort range (e.g. whisper, soft, neutral, loud, shout) alter production and speech acoustics, reducing intelligibility and limiting the robustness of any subsequent speech technology. Classification is challenging since effort lies on a continuum, adjacent categories are easily confused, and labeled data remain scarce. Prior SSL approaches with wav2vec2, HuBERT, and AST improve performance on the AVID corpus but still suffer from boundary errors. In this study, we introduce WavLM for the first time in vocal effort classification and benchmark it against wav2vec2 and HuBERT. To address data scarcity, we conduct a systematic study of augmentation strategies, covering RIR convolution, additive noise, time masking, speed perturbation, band-limiting, MixUp, and CutMix. Augmentation consistently improves WavLM, with gains ranging from +0.6% to +1.8% absolute. We further propose Gaussian-neighbor soft labels, which further reduce near-boundary confusions by modeling the vocal effort continuum. Our best system, WavLM-BASE with gradual unfreezing, augmentation, and Gaussian-neighbor soft labels, achieves 78.2% mean accuracy, establishing a new state-of-the-art on AVID.

• The paper demonstrates that combining WavLM with advanced soft-labeling yields a mean accuracy of 78.22% in classifying vocal effort on non-calibrated recordings.
• It systematically evaluates mix-based data augmentation methods, such as MixUp, to reduce boundary confusion in adjacent vocal effort categories.
• Gradual unfreezing of transformer layers enhances model resilience, offering practical benefits for applications in speaker identification and forensic linguistics.

Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings

Problem Landscape and Motivation

Vocal effort classification, encompassing categories from whisper to shout, has critical impacts on speech production, intelligibility, and downstream speech technology robustness. The task is inherently challenging due to the perceptual continuum of vocal intensity—adjacent categories (e.g., loud and very loud) are particularly susceptible to confusion and annotator disagreement, compounded by the scarcity of well-labeled datasets. Conventional ASR methods do not directly address this continuum, and boundary errors persist with prior state-of-the-art SSL models such as wav2vec2 and HuBERT. This paper conducts the first systematic evaluation of WavLM for vocal effort classification, addressing limitations of prior approaches, and investigates diverse data augmentation and soft-labeling strategies to reduce boundary confusion and improve model robustness.

Methodological Framework

Dataset and Preprocessing

Experiments utilize the AVID corpus, which provides 10,000 utterances across four vocal effort categories (soft, normal, loud, very loud) from 50 English speakers. The focus is on non-calibrated speech recordings, where amplitude cues are normalized, requiring models to infer vocal effort from spectral and temporal features. Evaluation employs 10-fold group cross-validation and mean accuracy as the primary metric.

SSL Backbone Models

Three transformer-based SSL encoders are considered: Wav2Vec2-Base, HuBERT-Base, and WavLM-Base, all pre-trained on large speech corpora with masked prediction objectives. WavLM-Base is shown to outperform others by more than 1% absolute accuracy, with gradual unfreezing of transformer layers during fine-tuning to mitigate overfitting and enhance adaptation to effort-related cues.

Data Augmentation Techniques

A comprehensive suite of waveform-level augmentations is evaluated:

• RIR Convolution: Simulates reverberant environments.
• Additive Noise: Introduces Gaussian noise.
• Time Masking: Random temporal silence spans.
• Speed Perturbation: Modifies speaking rate without RMS normalization.
• Band-Limiting: Applies frequency filtering.
• MixUp/CutMix: Generates interpolated and segmented composite utterances, extending data beyond discrete class labels.

The augmentation’s impact is visualized in log-mel spectrograms for an utterance under each transformation.

Figure 1: Log-mel spectrograms of an utterance demonstrating the spectral effects of various augmentation methods.

MixUp and CutMix differ in global versus localized blending, respectively, as seen in the spectrogram overlays.

Figure 2: Spectrograms comparing source utterances, CutMix segment replacement, and MixUp global interpolation.

Soft-Labeling Strategies

Recognizing the continuum nature of vocal effort, three soft-label approaches are proposed:

• Label Smoothing: Uniform redistribution of probability mass.
• Gaussian-Neighbor Soft Labels: Smears distribution preferentially toward neighboring effort categories, controlled by variance.
• Mix-Based Soft Labels: Interpolates Gaussian-neighbor distributions for augmented utterances.

The probability distributions for each approach are illustrated for the “loud” category.

Figure 3: Probability distributions for hard, smoothed, and Gaussian-neighbor labels in the loud category.

KL-divergence loss is used in place of traditional cross-entropy to promote uncertainty calibration and boundary sensitivity.

Experimental Results and Analysis

Baseline SSL Comparison

WavLM-Base achieves 75.24% mean accuracy (std: 1.47), outperforming wav2vec2 by 7.7% and HuBERT by 1.11%, validating its superior prosodic representation for vocal effort discrimination. Gradual unfreezing further stabilizes fine-tuning.

Data Augmentation Impact

All augmentations yield performance improvements (+0.6–1.8% absolute). RIR convolution and time masking are particularly effective, but Mix-based methods, especially MixUp, demonstrate the strongest gains, consistent with the continuous-label nature of vocal effort.

Soft-Label Regularization Efficacy

Label smoothing outperforms hard labels (+1.7%), while Gaussian-neighbor soft labels provide larger accuracy gains (77.32%, +2.1% over hard labels). Combined with MixUp (α=0.6), the mean accuracy reaches 78.22%—the highest and most stable result reported for AVID. Increased MixUp sharpness (α=0.8) degrades performance, indicating optimal balance between label ambiguity and discriminability.

Confusion matrices contrast baseline WavLM and the best system, showing reduced errors between adjacent categories and improved class balance.

Figure 4: Confusion matrices for VE-ID—best system (MixUp + Gaussian-neighbor soft labels) substantially reduces boundary confusions.

Implications and Future Directions

The proposed combination of WavLM, targeted augmentations, and Gaussian-neighbor soft labels reshapes vocal effort classification robustness in naturalistic, non-calibrated data. Strong numerical results (78.22% mean accuracy) demonstrate reliable performance improvements, particularly in mitigating errors along the vocal effort continuum. Practically, these advancements enable more nuanced modeling in speaker identification, forensic linguistics, and affective computing—scenarios where vocal effort variations are both common and consequential for system reliability.

Theoretically, Gaussian-neighbor soft labels provide a principled mechanism for integrating ordinal structure and perceptual proximity into model training, and mix-based augmentations produce richer inter-category examples that reflect real-world ambiguities. The gradual unfreezing of SSL backbones should be considered in low-resource adaptation scenarios.

Future research will likely explore extension to larger, more diverse datasets, multi-modal signals, and integration within end-to-end frameworks for naturalistic team-based communications (e.g., Fearless Steps APOLLO), with expectations for further gains through more sophisticated continuum-aware learning strategies.

Conclusion

This study systematically advances vocal effort classification leveraging WavLM, diverse augmentation, and specialized soft-label regularization. The methodology achieves state-of-the-art results on AVID, provides a framework for robust speaker-based vocal effort identification, and has broad implications for the transferability of continuum-aware SSL approaches in speech applications.