NaturalVoices: Expressive Speech Dataset

• NaturalVoices is a large-scale, spontaneous speech dataset offering over 5,000 hours of richly annotated podcast audio for VC, ASR, and TTS research.
• It employs an automated, multi-stage pipeline for segmentation, transcription, diarization, emotion labeling, and environmental sound annotation to ensure precise metadata.
• The dataset supports advanced research in expressive speech synthesis, nonverbal vocalization modeling, and robust performance in noisy, real-world conditions.

NaturalVoices (NV) refers to a family of large-scale, spontaneous, and richly annotated speech datasets, processing pipelines, and methodological paradigms for modeling, converting, and synthesizing naturalistic human speech. The NaturalVoices initiative addresses the limitations of acted and scripted corpora by providing podcast-scale data with fine-grained annotations for voice conversion (VC), expressive speech synthesis, and automatic speech recognition (ASR), emphasizing nonverbal vocalizations (NVs), emotion, and environmental context. The NV resource suite comprises over 5,000 hours of spontaneous conversational audio, modular open-source annotation pipelines, and benchmarks for robust, emotion-aware VC and TTS systems (Du et al., 31 Oct 2025, Salman et al., 2024).

1. Dataset Composition and Annotation Structure

NaturalVoices includes large, spontaneous podcast recordings annotated for speaker identity, transcript, speech quality, demographic properties, categorical and continuous emotion, and nonverbal and background events (Du et al., 31 Oct 2025, Salman et al., 2024). The central corpus comprises:

• Scale and source: 5,049 hours of segmented speech from 6,790 Creative Commons podcast episodes; more than 2,670 unique speakers.
• Recording conditions: "In-the-wild" noise, multiple speakers/overlaps, and varied sampling rates (44.1 kHz: 77.3%; 16 kHz: 14.5%; 48 kHz: 4.8%).
• Segment structure: 99.3% of segments are 1–10 s (median ≈ 6.67 s). Segment-level statistics include speaker count, gender balance (≈54.7% male utterances), and speech/music discrimination.
• Annotation pipeline: Modular and open-source [github.com/Lab-MSP/NaturalVoices]—document-level ASR (Faster-Whisper), diarization (pyannote-audio), segment-level speaker mapping, speech quality (PESQ, STOI, SI-SDR, MOS, SNR via WADA-SNR, DNSMOS Pro), demographics, categorical (anger, sadness, happiness, neutral) and attribute-based emotion (valence, arousal, dominance via WavLM+regressor), and environmental event tags (AST; 500+ classes).

Extensive filtering options (e.g., single speaker only, SNR > 30 dB, segment length, emotion balance) are supported for custom subset extraction (Du et al., 31 Oct 2025).

2. Pipeline Architecture and Annotation Methodologies

The NV processing pipeline implements multi-stage, automatic annotation:

• Segmentation/Transcription: Initial splitting with Faster Whisper into coherent 4–6 s units, refined using silence detection, followed by phone-level alignment with Montreal Forced Aligner (Salman et al., 2024).
• Speech vs. Music Filtering: 1D temporal convolutional network trained on radio-broadcast distinction.
• Speaker Diarization/Clustering: pyannote.audio v2.1 to identify local speakers, then global speaker assignment via timestamp merging and reference segment selection.
• Demographics: Transformer-based gender and age prediction (Felix 2023 model; majority-vote aggregation).
• Emotion Labeling: WavLM backbone with LoRA PEFT adapters and MLP head, trained with cross-entropy loss for four-way categorization and MSE regression for continuous attributes ($$[\hat{a},\hat{v},\hat{d}]=W_{\mathrm{reg}}h(x)+b_{\mathrm{reg}}$$).
• SNR Estimation: WADA-SNR computes SNR in dB by modeling Laplacian mixture in short-time amplitude derivatives.
• Sound Event Annotation: Audio Spectrogram Transformer (AST, pre-trained on AudioSet) for 500+ event tags.

The annotation process is fully automated, yielding each segment's transcript, high-precision phone alignment, speaker ID, demographic properties, SNR, emotion (categorical + continuous), and acoustic event tags.

3. Nonverbal Vocalizations (NV) and Expressive Events

NaturalVoices provides extensive support for modeling nonverbal vocalizations and expressive events:

• NV Categories: NVs are operationalized as event-level vocal/oral actions lacking canonical phone sequences—breaths, laughter, sighs, swallows, smacks, coughs, cries—added as sequence tokens in ASR/VC tasks (Yang et al., 2 Jul 2026).
• Occurrence and Distribution: "Breath" dominates (90%+), followed by laughter (≈9%), with the remainder <2% each (e.g., cries, coughs are <1%).
• Annotation: Inline transcript tags (e.g., “<laugh>”), frame-level spoken-noise (SPN) supervision, and auxiliary NV category labels.
• Data-centric modeling: Three strategies enhance NV modeling in both ASR and voice conversion:
  1. Two-stage curriculum: Train with all NVs collapsed to a generic <NV> tag ($L_{\mathrm{stage1}}$), then fine-tune on specific NV categories ($L_{\mathrm{stage2}}$), initializing specific NV token embeddings from the generic scaffold.
  2. Inter-token transfer: For rare NVs, incorporate auxiliary high-resource NV samples during fine-tuning and exploit shared acoustic structure.
  3. Voice-conversion augmentation with class balancing: Rare NV utterances are augmented via Seed-VC, targeting per-category sampling weights $w_c = \min(\alpha / p_c, w_{\max})$ to achieve class balance (Yang et al., 2 Jul 2026).

• Evaluation: Sentence-level F1, precision/recall per NV, and WER for lexical ASR. Demonstrated gains: e.g., "cry" F1 up to 69.0 with both inter-token transfer and sufficient rare NV samples.

The NV ontological approach is designed for extensibility as new nonverbal categories and expressive events are identified.

4. Benchmarks, Evaluation Metrics, and Model Performance

NaturalVoices underpins robust large-scale benchmarks for VC and expressive speech synthesis:

• Objective VC Evaluation: Speaker similarity (SV acceptance rate, SECS; e.g., SV Acc ≈ 0.95–0.98), intelligibility (CER, WER via Whisper or w2v2), emotion transfer metrics (ECA, EECS), MOS (quality, speaker similarity, and emotion).
• Data scale effects: Greater training data yields improvements in SV Acc and MOS, though increases in segment spontaneity may elevate WER.
• Robustness: Wide SNR coverage enables training VC models that generalize across noisy, real-world acoustic environments (Salman et al., 2024).
• Comparison to baselines: VC models trained on NaturalVoices achieve substantial gains in speaker similarity (e.g., S2S SV 71.10%→93.65%) vs. scripted datasets (VCTK) (Salman et al., 2024).
• Expressive/Emotion-balanced conversion: Emotion-balanced subsets (e.g., 340 h with 85 h per category) enable evaluation of emotional voice conversion accuracy and fine-grained emotion embedding transfer.
• Limitations: Current VC architectures tuned on acted, scripted speech degrade when scaled to fully spontaneous NV data, highlighting a key benchmark role for NV (Du et al., 31 Oct 2025).

5. Applications and Research Use Cases

NaturalVoices is positioned as a foundational resource for multiple research domains:

• Expressive and Emotional VC/TTS: Enables cross-speaker identity/emotion transfer and fine-grained controllability using both categorical and continuous emotion attributes; supports the synthesis of laughter, sighs, fillers, and other NVs directly within speech streams (Du et al., 31 Oct 2025, Cho et al., 15 Mar 2026).
• Noisy-to-Noisy and Robust Speech Modeling: Trains models for environments with realistic background sounds, high variability in SNR, and overlapping speakers.
• Dialogue and Conversational Systems: Naturalistic cues such as hesitations, turn-taking, and nonverbal events are explicitly annotated and encoded.
• Semi/Self-Supervised Learning: Weak and automatic labels enable the use of NV in self-training, semi-supervised speaker diarization, ASR/SER, and style transfer without manual annotation (Salman et al., 2024).
• Best Practices: Researchers are advised to filter according to task-specific SNR, segment length, speaker count, and emotion distribution, and to retrain vocoders on matching subsets to prevent domain mismatch.

6. Methodological Advances and Implications for Future Systems

The NV approach has precipitated several methodological innovations:

• Curriculum learning and acoustic scaffolds: General NV representation scaffolds, built via two-stage curricula, enable low-resource NV category bootstrapping, accelerating rare event recognition and synthesis (Yang et al., 2 Jul 2026).
• Shared acoustic features and augmentation: Systematic cross-transfer and augmentation leverage structural similarity among respiratory/laryngeal NVs.
• Rich metadata and conditioning: The combination of categorical labels, continuous attributes, prosodic statistics, and sound events empowers multi-objective and prompt-based speech generation and VC.
• Extensible open-source pipeline: Modular annotation tools allow adaptation, re-annotation, and filtering, supporting reproducible benchmarking and custom subset creation.

This suggests that future expressive VC and ASR/TTS systems should natively integrate NV tokens, adopt data-centric class rebalancing and augmentation strategies, and implement curriculum-driven learning scaffolds for both NV and emergent expressive categories.

7. Limitations and Prospects

NaturalVoices exposes challenges distinct from traditional corpora:

• Spontaneity and Domain Shift: VC models trained on acted/read speech frequently underperform on spontaneous material, indicating the necessity of in-domain training for applications in real-life dialogue, podcast, and emotional interaction contexts (Du et al., 31 Oct 2025).
• Weak Labeling: Demographic and emotion metadata are "weakly-supervised," necessitating methods that accommodate label uncertainty or re-annotation protocols.
• NV granularity and overlap: Current resources do not adequately model overlapping NV+speech events (e.g., laughing while talking) due to annotation and segmentation challenges (Cho et al., 15 Mar 2026).
• Scaling and model degradation: Certain model architectures (e.g., diffusion-based) may exhibit degraded performance at full NV scale due to noise and label variability.

Potential research opportunities include prompt-guided VC using NV metadata, prosody-aware anti-spoofing, end-to-end spontaneous TTS, joint segmentation for overlapping NV modeling, and semi-supervised NV mining from additional in-the-wild data.

---

References:

• "NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion" (Du et al., 31 Oct 2025)
• "Towards Naturalistic Voice Conversion: NaturalVoices Dataset with an Automatic Processing Pipeline" (Salman et al., 2024)
• "Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR" (Yang et al., 2 Jul 2026)
• "Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations" (Cho et al., 15 Mar 2026)