NonverbalTTS: Nonverbal Vocal Synthesis

• NonverbalTTS is a framework for modeling and synthesizing nonverbal vocalizations using detailed taxonomies and specialized NV tokens for precise, context-aware control.
• It leverages extensive datasets and benchmarks like NVBench and NonVerbalSpeech-38K to evaluate NVV controllability with metrics such as F1 scores and normalized tag distances.
• The system architecture incorporates token-based control, prosodic modulation, and instruction-prompt encoders to achieve fine-grained NVV insertion and expressive synthetic speech.

NonverbalTTS (NVTTS) refers to the modeling, synthesis, and evaluation of nonverbal vocalizations (NVVs) and paralinguistic events—such as laughter, sighs, coughs, gasps, sniffs, and conversational interjections—within text-to-speech (TTS) and related speech generation systems. NVTTS systems aim to achieve not only lexical fidelity but also accurate, contextually situated, and controllable production of NVVs, thereby enhancing the expressivity, emotional resonance, and naturalness of synthetic speech.

1. Taxonomies and Data Resources

The advancement of NVTTS is underpinned by the development of datasets and taxonomies capturing a broad inventory of nonverbal events. Taxonomic granularity is critical for enabling fine-grained controllability and objective benchmarking.

• Taxonomies: State-of-the-art resources such as NVBench establish 6 super-categories and 45 fine-grained NVV types, covering respiratory, throat/physiological, laughter/crying spectra, oral/miscellaneous, and emotional vocalizations, with each NVV represented as a tuple $(t, s)$ (type and onset index) (Xue et al., 17 Apr 2026). NV-Bench further highlights functional class groupings—vegetative, affect bursts, and conversational grunts—yielding 14 canonical NV categories for multi-lingual analysis (Ni et al., 16 Mar 2026).
• Datasets: Notable open resources include:
  • NonverbalTTS: 17 h English corpus (10 NV types, 8 emotion classes) drawn from VoxCeleb and Expresso with manual validation and a multi-annotator fusion protocol (Borisov et al., 17 Jul 2025).
  • NonVerbalSpeech-38K: 38,718 samples (~131 h, 10 NV categories) across English and Chinese from diverse real-world sources, automatically annotated by a wav2vec 2.0-based detector (Ye et al., 7 Aug 2025).
  • NVSpeech: 48,430 manual and 174,179 automatic word-level annotated Mandarin utterances spanning 18 paralinguistic categories (Liao et al., 6 Aug 2025).

The proliferation of high-quality, richly annotated, and taxonomically comprehensive datasets is a precondition for NVTTS system scaling and cross-lingual generalization.

2. System Architectures and Control Mechanisms

NVTTS systems operationalize NVV generation by explicitly extending the TTS control interface, model architectures, and embedding strategies.

• Token-based Controllability: The dominant approach augments TTS vocabularies $V' = V_{lex} \cup V_{para}$, where $V_{para}$ are special NV tokens (e.g., "[Laughter]", "[Sigh]") with dedicated learned embeddings interleaved within lexical sequences. This allows for arbitrary positional control at inference (Liao et al., 6 Aug 2025, Borisov et al., 17 Jul 2025, Zhou et al., 25 May 2026).
• Prosodic and Fine-Grained Modulation: Certain systems encode additional NV attributes—style, duration, repetition—via composite embeddings $e_{nv} = e_s + e_d + e_c$ (style, duration, count), offering prosodic and frequency control (Zhou et al., 25 May 2026).
• Instruction-Prompt Encoders: Modern zero-shot TTS backbones such as CosyVoice2 optionally deploy instruction-prompt encoders that process style/NV tags as additional control signals, mapping to context-sensitive style embeddings (Liao et al., 6 Aug 2025).
• APIs for Control Strength: Interface specifications expose a control_strength parameter to modulate cross-attention weights on NV tokens, directly affecting event duration or intensity $$( e^{nv}_{para} = (1+\lambda_{control})E(token_{para}) )$$ (Liao et al., 6 Aug 2025).

These mechanisms jointly enable explicit context-aware NVV insertion, flexible duration/exaggeration control, and support for complex expressive phenomena.

3. Training Objectives, Losses, and Optimization

NVTTS systems typically employ standard TTS loss formulations, adapted to the expanded input space:

• Main Objectives: Flow-matching or diffusion-based reconstruction loss $L_{flow}$, L1/L2 spectrogram loss $L_{recon}$, and, optionally, cross-entropy over next-token prediction (for token models) (Liao et al., 6 Aug 2025, Borisov et al., 17 Jul 2025).
• Paralinguistic Regularization (optional): Classifier-based cross-entropy loss $L_{para}$ on NV tokens may be introduced, but main acoustic objectives often suffice for correct NV rendering (Liao et al., 6 Aug 2025).
• Emotion/Context Losses: Additional losses include emotion classification (for joint affective-NV modeling) and explicit duration prediction (Zhou et al., 25 May 2026).
• Optimization Regimes: Fine-tuning on mixed human-validated and auto-labeled data, large-scale batching, staged curriculum (from high-quality to larger auto-labeled corpora), AdamW optimization, and warmup/cosine learning-rate schedules are standard practice (Liao et al., 6 Aug 2025).

Ablations across architectures reveal that the removal of explicit NV tags dramatically degrades NV generation, while emotion tags may have only marginal impact on NV-specific metrics (Borisov et al., 17 Jul 2025).

4. Objective and Subjective Evaluation Protocols

Rigorous benchmarking of NVTTS systems is enabled by multi-axis protocols distinguishing general speech quality from NVV-specific performance.

• Naturalness & Intelligibility: Conventional metrics include WER/CER (with/without NV tokens), DNSMOS (non-intrusive perceptual quality), and speaker similarity (embedding-based SIM/SSIM).
• NVV Controllability: Precision, recall, and F1 for correct NVV type and placement, with placement error (Normalized Tag Distance, NTD) computed as:

  $$\mathrm{NTD} = \frac{1}{|\mathcal{U}_{TP}|} \sum_{u \in \mathcal{U}_{TP}} \frac{|s_p^{(u)} - s_g^{(u)}|}{L_u}$$

  where $s_p^{(u)}$ and $V' = V_{lex} \cup V_{para}$0 are predicted and ground-truth onset indices, and $V' = V_{lex} \cup V_{para}$1 is transcript length (Xue et al., 17 Apr 2026).

• Instruction Alignment: The Paralinguistic Character Error Rate (PCER), restricted to NV tokens, assesses alignment to intended NV instructions (Ni et al., 16 Mar 2026):

  $V' = V_{lex} \cup V_{para}$2

• Acoustic Fidelity: Distributional gap via Fréchet Audio Distance (FAD) and Fréchet Distance (FD) on PANNs features measure statistical proximity of synthetic NVVs to real ones.
• Perceptual Salience: Subjective Likert-scale ratings (e.g., NMOS for naturalness, PE for perceptual NV salience) and instruction-following (IMOS) are correlated with objective metrics ($V' = V_{lex} \cup V_{para}$3 for IMOS vs. PCER; each 10% PCER drop yields $V' = V_{lex} \cup V_{para}$4 0.4 IMOS gain) (Ni et al., 16 Mar 2026).

Benchmarks such as NVBench and NV-Bench standardize these protocols across 45- and 14-type NVV inventories respectively (Xue et al., 17 Apr 2026, Ni et al., 16 Mar 2026), enabling direct cross-system and per-NVV category comparison.

5. System Benchmarks and Comparative Findings

Extensive benchmarking exposes decoupling between naturalness, intelligibility, and NVV controllability:

• System Comparison: Tag-based systems (e.g., ElevenLabs, CosyVoice2, Bark, Orpheus) display higher NVV controllability (F1 up to 0.73; NTD as low as 0.009), while certain prompt-based systems excel in naturalness and overall quality but not NVV instruction following (Xue et al., 17 Apr 2026).
• Strong Performers: For Mandarin, NV-CV3 achieves lowest PCER ($V' = V_{lex} \cup V_{para}$5), while NV-FlexiVoice excels at acoustic fidelity (FAD = 0.29, FD = 2.72). Ground-truth audio attains PCER = 9.38% and SIM = 0.781 (Ni et al., 16 Mar 2026).
• Trade-offs: High overall speech quality (low WER, high DNSMOS) does not guarantee NVV controllability. NVV salience remains challenging for low-SNR oral cues and long-duration affective NVVs, with PE often below 1.5 regardless of base system (Xue et al., 17 Apr 2026).
• Fine-Grained Control: Embedding NVV duration/repetition metadata yields large eMOS gains (4.20 vs. baseline 3.85) and increased emotion recognition accuracy (up to 98.3% for sadness) with only minor nMOS decrement (Zhou et al., 25 May 2026).

Results from continuous and discrete control studies indicate inventory coverage and precise temporal alignment remain primary bottlenecks in current systems (Ni et al., 16 Mar 2026, Ye et al., 7 Aug 2025).

6. Research Challenges and Future Directions

Open problems and promising avenues include:

• Long-Tail NVV Modeling: Under-represented events (yawn, gasp, certain oral cues) are particularly challenging due to data sparsity and poor acoustic distinctiveness (Ye et al., 7 Aug 2025, Xue et al., 17 Apr 2026).
• Alignment Robustness: Misalignments in word- or token-level timestamps (from ASR or forced alignment) propagate as tag-placement errors, impacting normalized tag distance and instruction following (Ye et al., 7 Aug 2025).
• Unified and Multi-Modal Pipelines: There is interest in joint architectures predicting both lexical and NVVs in a single pass, as well as multimodal extensions harnessing video or facial cues for automatic NVV detection and boundary refinement (Ye et al., 7 Aug 2025, Cho et al., 15 Mar 2026).
• Perceptual Metrics: Correlation between objective metrics (PCER, FAD/FD) and human/LLM ratings is encouraging, yet bespoke NVV-aware metrics that encompass pragmatic appropriateness and prosodic naturalness are still evolving (Xue et al., 17 Apr 2026, Ni et al., 16 Mar 2026).
• Scalability and Cross-Lingual Generalization: Fully automatic pipelines (e.g., NonVerbalSpeech-38K) and cross-lingually robust NV detectors are enabling larger, more diverse corpora and advancing generalization across language boundaries (Ye et al., 7 Aug 2025).
• Semi-Supervised and Data Augmentation: Research into semi-supervised NVV mining, augmentation for rare NVVs, and style transfer across emotional and prosodic dimensions are active focal points (Cho et al., 15 Mar 2026, Zhou et al., 25 May 2026).

Unified evaluation standards and open-access datasets are accelerating reproducible research, but qualitative advances will depend on improved fusion of linguistic, affective, and acoustic control and on closing the gap in NVV appropriateness, naturalness, and coverage across social and communicative functions.