Vocalize: Multifaceted Voice Research
- Vocalize is a broad concept that encompasses phonetic normalization, nonverbal vocalization, and interactive voice systems as structured operations in audio research.
- Research employs sound-oriented symbolic encodings and inline paralinguistic tokenization to integrate lexical and non-lexical vocal signals.
- Empirical studies reveal practical benefits, from improved clustering and accuracy metrics to enhanced joint transcription of speech and nonverbal cues.
Vocalize denotes a family of operations in contemporary audio, speech, HCI, and computational linguistics research rather than a single method. In current usage, it may mean converting words into a sound-oriented code, recognizing or generating non-verbal human vocalizations, inserting paralinguistic events into ASR or TTS token streams, building interactive voice systems that operate beyond lexical language, or deploying voice-centered applications such as gamified competitions and conversational platforms (Nayak et al., 2020, Baird et al., 2022, Ivsic et al., 2024, Teskeredzic et al., 28 Jul 2025). The common thread is that vocal output is treated as a structured signal carrying phonetic, affective, social, and interactional information.
1. Terminological scope
The term has at least four technically distinct senses in the cited literature.
| Context | Meaning of “vocalize” | Representative source |
|---|---|---|
| Phonetic normalization | Convert words into a Soundex-based representation | (Nayak et al., 2020) |
| Non-verbal vocalization modeling | Recognize or generate laughs, gasps, cries, coughs, breathing, and related events | (Baird et al., 2022) |
| Voice interaction beyond language | Use vocal sound, gesture, and bodily presence as the interaction substrate | (Ivsic et al., 2024) |
| Proper noun platform | End-to-end gamified voice competitions over WhatsApp | (Teskeredzic et al., 28 Jul 2025) |
In "Linguistic Classification using Instance-Based Learning," the phrase “we vocalize each word” does not mean vowel restoration or a grapheme-to-phoneme system; it means transforming each word into a sound-oriented code using Soundex, specifically a 4-character long Soundex score used for clustering and classification (Nayak et al., 2020). By contrast, ExVo, voc2vec, MNV-17, NonverbalTTS, and NVSpeech use vocalization to denote non-lexical human sounds such as vocal bursts, nonverbal vocalizations, and paralinguistic cues (Baird et al., 2022, Koudounas et al., 22 Feb 2025, Mai et al., 19 Sep 2025, Borisov et al., 17 Jul 2025, Liao et al., 6 Aug 2025).
A recurrent misconception is that vocalization research is reducible to speech recognition or text-to-speech. The literature instead treats vocalization as a broader object: standalone emotional bursts, inline transcript tokens, spatially distributed vocal interaction, and phonetic normalization are all legitimate uses of the term. This suggests that “vocalize” is best understood as an umbrella concept spanning representation, interpretation, synthesis, and interaction.
2. Vocal bursts and non-verbal corpora
A central research program concerns non-verbal human vocalizations as primary learning targets. The ICML 2022 Expressive Vocalizations Workshop and Competition uses the Hume Vocal Bursts dataset, containing 59,201 vocalizations, 36:47:04 total audio, and 1,702 speakers from 4 countries, annotated for the intensity of 10 expressed emotions and organized into three tracks: ExVo-MultiTask, ExVo-Generate, and ExVo-FewShot (Baird et al., 2022). The benchmark formalized vocal bursts such as laughs, gasps, and cries as a distinct affective-audio domain.
Mandarin work has pushed this further toward integrated speech understanding. MNV-17 introduces a 7.55-hour performative Mandarin speech dataset with 2,444 sentence-level samples, 17 distinct and well-balanced classes of common NVs, and 49 speakers; its reported max/min class ratio is 2.7, and its split is speaker-independent (Mai et al., 19 Sep 2025). The dataset is explicitly designed for joint semantic transcription and nonverbal vocalization recognition, rather than sound-event tagging alone.
Representation learning has also become domain-specific. voc2vec is pretrained on 10 open-source non-verbal vocalization datasets totaling 125.19 hours, 68,067 samples, and 6.67 s average sample duration, and the best variant, voc2vec-ls, reports +5% average UAR, +2% average accuracy, and +4% average macro F1 over the next-best-performing model across six downstream datasets (Koudounas et al., 22 Feb 2025). This supports the claim that non-verbal human vocalization is not well covered by speech-only or generic audio pretraining.
3. Representation strategies
Current research uses two major representation regimes: sound-oriented symbolic encodings and inline paralinguistic tokenization.
The first regime appears in multilingual lexical work. The Soundex-based paper uses vocalization as phonetic normalization: words are converted into 4-character Soundex scores, clustered with DBSCAN using epsilon 0.0375 and minimum samples 10, yielding 16 unique clusters, and classification then combines Levenshtein distance on strings with Euclidean distance on Soundex values in a custom Linguistic Distance Metric (Nayak et al., 2020). Its reported average clustering coefficient of 0.48 illustrates that “vocalize” can denote a computational sound-based encoding of words rather than acoustic generation.
The second regime inserts vocal events directly into transcript space. NVSpeech builds a Mandarin pipeline in which paralinguistic cues are represented as inline decodable tokens such as [Laughter], [Breathing], [Cough], [Crying], [Uhm], [Confirmation-en], and [Question-ah]. Its manually annotated subset, NVSpeech, contains 48,430 utterances, 76 hours, 18 classes, and 1,578 speakers; the automatically labeled corpus expands this to 174,179 audio-transcription pairs and 573.4 hours (Liao et al., 6 Aug 2025). The representational claim is that vocalizations belong inside the transcript rather than beside it.
MNV-17 adopts the same general principle for expressive ASR. The task is framed as unified sequence prediction in which lexical content and NV labels occupy the same output stream, and “each NV label was treated as a single character” in CER computation (Mai et al., 19 Sep 2025). On the benchmark, Qwen2.5-Omni achieves 3.60% joint CER, while strict NV recognition accuracy reaches 57.29% for both SenseVoice and Qwen2.5-Omni (Mai et al., 19 Sep 2025). This is not ordinary sound event detection; it is mixed lexical-paralinguistic transcription.
4. Generation and personalization
Generation research treats vocalization as controllable output rather than only a recognition target. In ExVo-Generate, the baseline trains separate emotion-specific generators with MSG-GAN on mel spectrograms, inverts them with fast Griffin-Lim, and evaluates with both FID and HEEP. The reported baseline FID scores range from 4.81 to 8.27 depending on emotion, and the combined generation score is (Baird et al., 2022). The dual metric is important: distributional similarity and human-perceived emotional precision are not interchangeable.
Personalized non-speech vocalization synthesis adopts a TTS-style factorization. "Synthesizing Personalized Non-speech Vocalization from Discrete Speech Representations" uses pretrained HuBERT features quantized by a 100-cluster k-means model, run-length encodes the resulting unit stream into “pseudo-phonemes,” and conditions a FastSpeech2-inspired acoustic model plus HooliGAN vocoder on pseudo-phonemes and speaker identity (Hsu, 2022). Using the ExVo train split of 571 crowd-sourced speakers and 12 hours of data, the system reports HEEP 0.79 for Amusement versus 0.49 for the ExVo baseline, and 0.51 for Awe versus 0.46 for baseline; the paper’s strongest diagnostic conclusion is that recording-condition heterogeneity is the major obstacle for NSV modeling (Hsu, 2022).
Text-aligned vocalization synthesis has likewise become explicit. NonverbalTTS introduces a 17-hour open-access English corpus annotated with 10 NV types and 8 emotion categories, where NVs are inserted directly into transcripts using tags such as [laugh] and [sigh] (Borisov et al., 17 Jul 2025). Fine-tuning CosyVoice-300M on this data yields results comparable to CosyVoice2; the best reported overall NV Jaccard is 0.80 for NVTTS-no-emotion, and removing NV tags reduces overall to 0.76, indicating that explicit textual NV tags are essential for controllable vocalization generation (Borisov et al., 17 Jul 2025).
Dialogue generation extends vocalization into multi-stage speech planning. SLIDE first uses an LLM to generate spoken dialogue text, then converts it to phoneme sequences, predicts phoneme durations with a two-tower transformer, and finally conditions a dialogue speech LLM on the resulting spoken phoneme sequence to generate discrete audio units (Lu et al., 1 Jan 2025). On Fisher, this reduces perplexity from 1228.82 to 421.29 and raises Meaningfulness MOS from 1.52 to 4.08, while preserving high Naturalness MOS (Lu et al., 1 Jan 2025). Here, to vocalize is to realize semantically coherent text as temporally structured conversational speech.
5. Interactive and application-specific systems
Some systems treat vocalization as interaction rather than transcription or synthesis alone. Transhuman Ansambl is an interactive singing-voice interface consisting of sixteen bespoke virtual singers arranged in a circle, each with a loudspeaker, RGBW LED ring, ultrasonic distance sensor, custom electronics, and central control in Max/MSP with Arduino Mega (Ivsic et al., 2024). In live performance mode, the system measures volume, pitch, and voice attack from a human singer and answers with samples drawn from a custom dataset of over one hundred samples organized across two groups of eight singers and three main vocal techniques: falsetto, belting, and musical phrasing (Ivsic et al., 2024). In installation mode, proximity values from 1 to 10 modulate output. The project’s theoretical framing is explicitly “voice beyond language.”
The proper noun "Vocalize" denotes a different class of system: a marketing-oriented platform for gamified voice competitions over WhatsApp (Teskeredzic et al., 28 Jul 2025). A campaign defines a target phrase and a target image contour; users send voice notes, and the platform scores them on keyword correctness and waveform-shape similarity. For keyword scoring, the paper gives the normalized Levenshtein formula
where is the Levenshtein distance and is the length of the longer string (Teskeredzic et al., 28 Jul 2025). For shape scoring, both the target contour and the user waveform are reduced to 40 temporal segments, RMS values are computed, and similarity is based on a dot product (Teskeredzic et al., 28 Jul 2025). Deployed at WeAreDevelopers, KulenDayz, GOTO Chicago, and Web Summit, the system collected 6,321, 1,257, 1,216, and 3,662 voice recordings respectively, with substantial lead-conversion and repeat-participation rates (Teskeredzic et al., 28 Jul 2025). This is a vocalization platform in the applied, production-oriented sense.
6. Evaluation, temporal dynamics, and safety
Evaluation practice is heterogeneous because vocalization is heterogeneous. ExVo combines acoustic distributional similarity and human affective judgment; MNV-17 uses joint CER and strict exact-match NV accuracy; NonverbalTTS uses WER, speaker similarity, EMO-SIM, DNSMOS, and NV Jaccard; SLIDE separates naturalness and meaningfulness; and Transhuman Ansambl relies largely on observational and reflective evidence (Baird et al., 2022, Mai et al., 19 Sep 2025, Borisov et al., 17 Jul 2025, Lu et al., 1 Jan 2025, Ivsic et al., 2024). One clear implication is that no single metric adequately covers vocalization quality.
Temporal organization is itself an evaluable property. In daylong home recordings of infant-caregiver interaction, both infant and caregiver vocalizations are clustered in time, with positive successive IEI correlations: for infants, standardized coefficients range from 0.23 to 0.30, and for adults from 0.29 to 0.33, all at (Ritwika et al., 2 May 2025). After controlling for previous IEI, receiving a social response predicts that the individual will vocalize again sooner than they would have in the absence of a response, leading the authors to describe infant-caregiver interaction as an “interpersonal foraging” process (Ritwika et al., 2 May 2025). This broadens vocalization research from signal content to multi-scale temporal dynamics.
Safety has become a distinct concern as TTS systems acquire stronger content fidelity. "Synthetic Voices, Real Threats" evaluates five commercial LALM-based TTS systems with HARMGEN, a suite of five attacks—Concat, Shuffle, Read, Spell, and Phoneme—designed to induce models to vocalize harmful content (Chen et al., 14 Nov 2025). The paper reports that under at least one attack, all five models that initially refused a majority of hate-speech prompts were compelled to synthesize 100% of them, and that deepfake detection on GPT-4o-mini-audio achieved accuracy of no more than 25% with EER over 75% (Chen et al., 14 Nov 2025). Proactive moderation detects 57–93% of attacks, but not all (Chen et al., 14 Nov 2025). This suggests that vocalization systems must be treated as cross-modal safety problems, not merely as benign renderers of text.
Across these literatures, vocalize no longer denotes only the act of making sound with the voice. It names a research space in which voice can be encoded, tokenized, clustered, transcribed, generated, personalized, spatialized, gamified, and moderated. The term therefore links phonetic abstraction, expressive non-verbal communication, interactive voice design, and deployment-scale speech systems into a single, technically diverse field.