Sound-Symbolic Onomatopoeia

• Sound-symbolic onomatopoeia is a linguistic phenomenon where word sounds mirror sensory impressions, enabling non-arbitrary mappings between phonetics and perception.
• It drives advances in environmental sound synthesis, neural audio parameterization, and expressive motion generation by linking discrete phonemes to continuous signals.
• Annotated datasets and neural models show high reliability in capturing nuanced acoustic features, fueling progress in multimodal AI research.

Sound-symbolic onomatopoeia refers to words or word forms whose phonological structure imitates or evokes the sensory (usually auditory, occasionally kinesthetic or visual) features of the concepts they denote. Unlike conventional sign–referent relations, where form–meaning mappings are largely arbitrary, sound-symbolic and onomatopoeic terms exploit non-arbitrary phonetic patterns that cue listeners to core acoustic parameters—enabling flexible cross-modal grounding and intuitive human–machine interaction. This phenomenon underpins multilingual iconic lexica (particularly in Japanese and Korean), informs the design of audio and motion synthesis systems, and provides a critical testbed for probing the capabilities and limitations of large-scale multimodal neural models.

1. Sound-Symbolic Onomatopoeia: Foundations and Motivation

Sound-symbolic onomatopoeia emerges from the iconic mapping between speech sounds and physical events or sensory impressions. Traditional symbolic language (e.g., “whistle,” “door slam”) captures event categorization but is not well suited for direct control over psychoacoustic dimensions (pitch, timbral sharpness, envelope, time–frequency structure). In contrast, onomatopoeic words encode such microstructural auditory details in their phonological form—Japanese “pii–” or “buu-buu” manifesting sustained, resonant, or abrupt auditory textures.

Psychoacoustic studies (Lemaitre & Rocchesso 2014; Sundaram & Narayanan 2006) have demonstrated that humans naturally employ onomatopoeia to convey graded qualities such as roughness, sharpness, or impact, with listeners and sound designers exploiting these forms as intuitive “sketches” for specifying or recalling complex environmental sounds (Okamoto et al., 2020). In computational settings, sound-symbolic onomatopoeia enables fine-grained, text-driven control of generative models, supporting both exploratory creativity and practical audio–visual content generation.

2. Large-Scale Datasets and Annotation Methodologies

The RWCP-SSD-Onomatopoeia dataset exemplifies a systematic approach to pairing environmental audio events with human sound-symbolic labels (Okamoto et al., 2020). Drawing from the RWCP-SSD corpus—comprising 9,722 high-fidelity (<2s, 48 kHz/16bit) recordings spanning 105 sound-event types—the annotation campaign collected 155,568 (sound, word) pairs via crowdsourcing. Each audio clip was independently labeled by at least five annotators, each producing three unique katakana strings transcribed into phonemic representations.

Sound categories in the RWCP taxonomy comprise: (1) crash/impact (e.g., “gasha,” “baka”), (2) human-operation with identifiable materials (e.g., whistle “pii–,” telephone ring “rii–rii–”), and (3) human-operation with unidentifiable materials (e.g., sawing “girigiri–”). The protocol involved:

• Task I: Workers submit onomatopoeic transcriptions and a self-reported confidence score $c \in \{1,\ldots,5\}$.
• Task II: For $c\geq4$, other workers assign an acceptance score $a \in \{1,\ldots,5\}$.

Summary statistics for a 15-class subset show high annotator reliability: mean($c$)$\approx$4.2, mean($a$)$\approx$4.5, both with low variance and positive correlation ($r\approx 0.7$), and over 80% of scores at level $\geq3$. Whistle sounds were most reliably labeled (mean($c$)$\approx$4.8, mean($a$)$\approx$4.9).

3. Mathematical Formulations & Embeddings Linking Phonetic and Perceptual Features

Future generative models can operationalize sound-symbolic onomatopoeia by mapping discrete phoneme/mora sequences to continuous audio or motion parameters through neural architectures. Illustrative mapping functions (Okamoto et al., 2020) include:

• Fundamental frequency: $$f_0(t) = g(\mathrm{phoneme\ sequence}, \mathrm{mora\ timing})$$
• Spectral envelope: $$S(\omega) = h(\{F_1, F_2, \ldots\}, \mathrm{sibilant\ index})$$
• Temporal envelope: $$A(t) = \sum_{m=1}^M \alpha_m\,\mathrm{rect}((t-\tau_m)/\Delta_m)$$

Neural implementations may define:

$$z = \mathrm{Encoder}(P), \quad \{ f_0(t), S(\omega,t), A(t) \} = \mathrm{Decoder}(z)$$

where $P$ denotes the phoneme sequence.

For expressive control beyond audio, the Sakamoto system (Okamura et al., 2023) deterministically maps an onomatopoeic string to a $\mathbb{R}^{43}$ vector encoding bipolar adjectival impressions (e.g., “light–dark,” “sharp–mild”) via phoneme and syllable decompositions:

$$f(w) = \frac{1}{N+M}\left( \sum_{i=1}^N v_{p_i} + \sum_{j=1}^M v_{s_j} \right)$$

where $v_{p_i}, v_{s_j} \in \mathbb{R}^{43}$, and $w$ is the onomatopoeic word.

4. Experimental Paradigms and Evaluation in Multimodal AI

Onomatopoeia serves both as a vehicle for and a target of computational psycholinguistic tests. In “With Ears to See and Eyes to Hear” (Loakman et al., 2024), classic shape and magnitude symbolism tasks (Kiki–Bouba, Mil–Mal) and iconicity rating benchmarks probe the capacity for LLMs and vision–LLMs (VLMs) to internalize sound-symbolic correspondences.

Stimuli include DALL·E 3-generated images (“spiky”/“rounded”; “tiny”/“huge”) and pseudoword pairs contrasting in hypothesized sound–shape or sound–size mappings. Prompt engineering distinguishes standard from informed settings—the latter provide task-relevant context on sound symbolism. Evaluated models include GPT-4-vision, Gemini Pro, and several open-source LLaVA and LLaMA variants across a spectrum of parameter counts.

Key statistical measures:

• Raw agreement with human judgments (percentage)
• Cohen’s $\kappa$ and Fleiss’s $\kappa$ for inter-annotator reliability
• Spearman’s $\rho$ and Pearson’s $r$ for rank and linear correlation in iconicity rating tasks

LLM and VLM performance on shape symbolism (standard Kiki–Bouba) approaches human reliability (GPT-4 $\kappa = 0.76$ with informed prompt, human Fleiss’s $\kappa = 0.731$). Maximum agreement for magnitude symbolism (Mil–Mal, Zil–Zal informed prompt) reaches 93% ($\kappa = 0.86$) for GPT-4. Iconicity ratings scale with model size and instruction-tuning, e.g., GPT-4 $\rho = 0.537$ (Nov '23), LLaMA-2 13B $\rho = 0.379$ (Loakman et al., 2024).

5. Applications: Environmental Sound Synthesis, Expressive Generation, and Motion Control

Sound-symbolic onomatopoeia provides a direct interface for generative models in several domains:

• Environmental sound synthesis: Neural architectures (e.g., conditional WaveNet or GANs) condition on sound-symbolic word embeddings to synthesize nuanced acoustic scenes, with the RWCP-SSD-Onomatopoeia dataset providing the first large-scale, confidence-validated training resource (Okamoto et al., 2020).
• Neural audio parameterization: Onomatopoeic input enables the prediction of detailed time–frequency trajectories ($S(\omega,t)$), yielding direct controllability for psychoacoustic parameters.
• Dance and motion generation: The “Dance Generation by Sound Symbolic Words” framework (Okamura et al., 2023) adapts the AI Choreographer pipeline, replacing music features with 43-dimensional Sakamoto system embeddings. Autoregressive transformer models synthesize 3D skeleton motion sequences responsive to phonetic impressions. Cross-lingual generalization is supported, with the pipeline applicable to onomatopoeia in diverse scripts and even invented vocalizations.

Table: Principal Datasets and Methods

Dataset/Method	Modalities	Scale	Core Application
RWCP-SSD-Onomatopoeia	Audio+text	155,568 pairs	Environmental sound synthesis
Sakamoto System (Doizaki et al.)	Text (phonology)	$\mathbb{R}^{43}$ axis	Cross-lingual sound-symbolic vectorization
Onomatopoeia–dance pairs	Text+motion	44 samples	Dance/choreography generation

6. Emergence and Modeling of Sound Symbolic Reasoning in Large Neural Models

Neural models trained on multimodal corpora exhibit emergent sensitivity to sound symbolism through several learning pathways (Loakman et al., 2024):

• Grapheme–phoneme statistical regularities: Token embeddings encode quasi-phonetic information, even in the absence of explicit audio exposure.
• Visual–textual co-occurrence: Pretraining on image–caption pairs (e.g., “the dog goes woof”) grounds onomatopoeic forms in concrete visual and event contexts.
• Prompt-based task adaptation: Providing models with minimal context (“sound symbolism tasks”) can sharply increase agreement with human judgements.
• Implicit “hearing”: GPT-4 reliably generates plausible onomatopoeic lists for events described textually or visually, reflecting acquisition of internalized “soundscape models.”

A plausible implication is that, while no true auditory representations are formed, pattern-matching over graphemic, statistical, and visual co-occurrence cues suffices for nontrivial sound–meaning mapping in LLMs and VLMs. Generalization beyond English orthography and pseudophonotactics, however, remains open, with further potential in explicit audio–text alignment and extension to non-Latin scripts.

7. Open Challenges and Research Directions

Several outstanding technical and linguistic challenges persist:

• Discrete-to-continuous mapping: Translating character-level onomatopoeia into high-dimensional acoustic or motion feature spaces remains nontrivial, especially with limited supervised data.
• Annotation variability and universality: The same onomatopoeic form can describe subtly different events; language-specific lexica (e.g., Japanese richness) raise questions for cross-linguistic generalization (Okamoto et al., 2020).
• Multimodal model limits: Current VLM/LLM performance depends strongly on task context and model scale; absence of audio grounding and potential contamination in pretraining corpora limit interpretability and generalization (Loakman et al., 2024).
• Evaluative metrics: Beyond confidence/acceptance scores, robust listening and viewing tests are needed to assess generation fidelity under onomatopoeic control.
• Scaling and creative applications: Larger, cross-modal paired corpora and more advanced generative models (e.g., diffusion transformers) could support real-time, multimodal creative systems for audio, motion, and texture synthesis (Okamura et al., 2023).

Sound-symbolic onomatopoeia thus provides a unique and technically fertile paradigm for crossmodal grounding, dataset curation, and the development of expressive, language-agnostic neural generators, with ongoing implications for the study of computational phonosemantics, creative AI, and human–machine interaction.