Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR

Abstract: Modern automatic speech recognition (ASR) systems excel at transcribing lexical content but often omit nonverbal vocalizations (NVs), such as laughter, breaths, coughs, and cries, that carry conversational and affective information. Modeling NVs in ASR is challenging because NV annotations are sparse and highly long-tailed, with frequent categories such as breaths and laughter dominating rarer events such as cries and coughs. We study three data-centric strategies for improving low-resource NV recognition: (1) a two-stage curriculum that first maps all NV events to a generic token and then fine-tunes on target categories; (2) inter-token transfer from high-resource events, such as laughter and breath, to rare events, such as crying; and (3) voice-conversion augmentation with class balancing. Experiments show that shared acoustic structure across vocal events can be exploited to improve rare-category detection while preserving lexical ASR quality.

• The paper proposes a two-stage curriculum learning approach that pretrains with a generic NV token before fine-tuning for specific non-verbal event detection, achieving up to a 9.7 point F1 improvement for rare events.
• It employs inter-token knowledge transfer by using abundant NVs like breath and laugh to boost the detection performance of rarer events such as cry, improving cry F1 from 32.1 to 69.0 with limited samples.
• The study demonstrates that combining class balancing with voice conversion augmentation effectively addresses long-tail NV data challenges while reducing WER without harming lexical accuracy.

Effective Modeling of Non-Verbal Vocalizations in ASR

Introduction

The paper "Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR" (2607.01563) conducts a systematic analysis of data-centric methods for integrating non-verbal vocalizations (NVs), such as laughter, breath, cough, and cry, into end-to-end automatic speech recognition (ASR) frameworks. In contemporary ASR systems prioritized for lexical transcription, NVs are typically omitted, resulting in transcripts inadequate for affective computing, conversational AI, and speech pathology. The principal challenge stems from the highly long-tailed and imbalanced distribution of NVs, leading to both scarce and dominant NV categories in training data.

Challenges in Non-Verbal Event ASR

Direct integration of NV tokens into ASR exposes two critical complications. First, rare events like crying or sighing have insufficient labeled data for reliable supervised modeling. Unlike rare words, these NVs lack structured representations and canonical phone sequences, complicating their detection. Second, class imbalance—exemplified by breath comprising over 90% of NV samples in typical corpora—causes standard training objectives to ignore rare events, further suppressing recognition performance for underrepresented NV types.

Figure 1: Sample distribution of non-verbal sound events before and after class balancing.

Data-Centric Strategies

The authors introduce three key strategies to tackle NV scarcity and imbalance: (1) a two-stage curriculum learning framework, (2) inter-token knowledge transfer leveraging acoustic and physiological similarity between NV types, and (3) class balancing combined with synthetic voice conversion (VC) augmentation. Each strategy is designed to maximize rare event learning efficiency without sacrificing overall ASR performance.

Two-Stage Curriculum Learning

The first stage of curriculum learning collapses all NVs into a generic <NV> token, training the network to distinguish speech from non-speech vocal activity across the full dataset. After learning this NV acoustic manifold, fine-tuning restores fine-grained NV labels (e.g., <cry>, <laugh>, <cough>). NV token embeddings are bootstrapped using the generic <NV> embedding, establishing an informative initialization for rare events.

Figure 2: Two-stage curriculum learning pipeline, training first on a generic <NV> token before specific NV category specialization.

Empirically, the two-stage approach produces substantial F1 gains for rare categories, with the greatest impact observed at extreme data scarcity—up to a 9.7 point F1 improvement in cry detection with 400 target samples. This confirms that decoupling general NV detection from fine-grained event classification is highly effective under limited supervision.

Inter-Token Knowledge Transfer

Acoustic and physiological overlap among NVs invites transfer learning. The model leverages high-resource NVs (breath, laugh) as an "acoustic scaffold" during rare-category training, substantially improving the detection of related low-resource NVs (e.g., cry). Inclusion of both laugh and breath tokens can more than double cry F1 relative to using only rare event data, demonstrating pronounced transfer efficiency due to shared respiratory and laryngeal features.

Class Balancing and Voice Conversion Augmentation

For datasets dominated by a single NV class, class balancing—via upsampling rare NV samples—ensures that rare event detection loss is emphasized. The addition of VC augmentation, specifically zero-shot conversion to new reference speakers, multiplies speaker diversity for otherwise speaker-limited rare events. Notably, augmentation alone fails without corresponding class balancing; their combination yields the only meaningful improvement for long-tail NV categories.

Figure 3: Effectiveness of voice-conversion augmentation (VC) for rare NV categories, with and without class balancing.

Experimental Results

The study benchmarks the enhanced NV-ASR system against Whisper-D, a Whisper-v2-large-based NV-annotated recognizer. Despite comprising only 200M parameters (vs. 1.55B in Whisper-D), the system achieves strictly superior F1 across all NV categories in the curated evaluation. Substantial improvements are documented for both common (laugh, cough) and rare (cry, sigh) categories. Importantly, the integration of NV recognition in the ASR model also reduces the WER below the baseline, indicating that NV modeling is not in conflict with lexical recognition performance.

In ablation studies, the two-stage curriculum strategy, inter-token transfer, and class balancing with VC augmentation all provide statistically significant boosts to rare NV event F1, particularly under data-poor and long-tailed training regimes. Notably, inter-token transfer with breath and laugh tokens improves cry F1 from 32.1 to 69.0 using only 400 annotated cry samples, an efficiency otherwise unattainable through direct data collection.

Theoretical and Practical Implications

The findings redefine how ASR systems can be extended beyond lexical transcription to incorporate paralinguistic information critical for conversational and affective speech interfaces. The clear empirical gains for rare NVs using generic acoustic NV pretraining and transfer support a general strategy for emerging NV taxonomies: collect high-quality generic NV labels and modest target-category samples, then leverage related abundant tokens for efficient rare event modeling. The results underscore that voice conversion is strictly beneficial only when loss weighting ensures rare-class relevance—not as an isolated augmentation technique.

These principles—two-stage coarse-to-fine learning, physiological transfer, and synthetic balancing—can scale to larger and more diverse NV inventories, including future applications in clinical speech diagnostics, emotion recognition, and robust AI dialogue. The approach yields a template for rapidly extending state-of-the-art ASR to new paralinguistic event classes using limited annotation effort.

Future Directions

The authors highlight limitations including the fixed seven-category taxonomy and lack of event localization evaluation. Broader taxonomies, support for subtle non-verbal cues, and position-aware tagging frameworks remain important next steps. The demonstrated techniques suggest that as new datasets and event inventories emerge, future NV-ASR models can rapidly adopt and effectively recognize novel non-verbal vocalizations, underpinning more expressive and human-aligned speech AI systems.

Conclusion

Through data-centric methodology—curriculum learning, inter-token transfer, and speaker-diversifying augmentation—a unified NV-aware ASR system can achieve high-fidelity recognition across both abundant and rare non-verbal vocalizations without adversely affecting WER. The approach enables scalable extension of ASR for rich conversational and affective content, providing a foundation for practical deployment in diverse speech technology applications.