Communicating Sound Through Natural Language

Abstract: Natural language is widely used to describe, prompt, and control audio systems, but rarely serves as the representation carrying audio itself. We introduce lexical acoustic coding (LAC), a framework in which pre-trained LLM sender and receiver agents transmit sound through natural language. Under fixed system prompts, the agents write their own analysis and synthesis code, communicating only through a lexical sentence, shared vocabulary, and optional symbolic music structure. The sender analyzes an input waveform into interpretable, non-learned acoustic descriptors, quantizes each with a feature-specific interval vocabulary, and verbalizes the lexical code as English. The receiver parses the sentence back into lexical-acoustic constraints and renders a waveform through closed-loop refinement. The transmitted text serves as both a rich caption and as the transport representation itself. We frame LAC as a finite-rate lossy quantizer, exposing trade-offs between vocabulary size, rate, and fidelity. Experiments on short sounds and symbolic music transfer show that plain text preserves measurable acoustic structure while remaining interpretable, editable, and native to LLM-mediated communication.

• The paper introduces a novel method where audio is encoded into structured English text, enabling sound to be transmitted via natural language.
• It employs deterministic DSP feature extraction with lossy quantization and closed-loop refinement to achieve around 84% post-synthesis reconstruction accuracy.
• Its approach provides human-readable, semantically editable audio transport that bridges descriptors, codecs, and captions for interpretable sound design.

Communicating Sound Through Natural Language: Lexical Acoustic Coding

Introduction and Motivation

The paper addresses the underexplored paradigm of using natural language—not as a peripheral or metadata modality, but as the primary representational and transport mechanism for audio content. The core proposal is "lexical acoustic coding" (LAC), where pre-trained LLMs, instantiated as sender and receiver agents, transmit audio solely through structured, lossily quantized, human-readable lexical codes encoded in English sentences. The central question is whether a sufficiently structured acoustic vocabulary can carry enough information for an LLM-mediated text channel to reconstruct audio with meaningful fidelity, interpretability, and editability.

Methodology

Lexical Acoustic Coding Framework

LAC consists of a one-time vocabulary sharing/setup phase and a per-sound communication phase:

• Vocabulary Setup: Agents agree on acoustic features ($\mathcal{F}$), feature-specific lexical alphabets ($\mathcal{A}_i$), and forward/inverse mappings ($E_i$, $R_i$, $I_i$) associating real-valued descriptor intervals to lexical terms. The vocabulary includes the feature types, interval bins, and English labels, defined either by humans or LLM agents.
• Analysis and Quantization (Sender): Each input sound is decomposed deterministically (using non-learned, standard DSP features such as RMS energy, spectral centroid, inharmonicity, etc.) into a $d$-dimensional vector. Each coordinate is independently quantized into a lexical symbol corresponding to its value interval (e.g., "mid-power," "warm," "clipped").
• Sentence Verbalization: The ordered lexical code is synthesized into a fluent English sentence which must unambiguously and injectively encode the entire lexical feature vector.
• Decoding (Receiver): The LLM parses the sentence, retrieves the lexical code, and inverts each label (via the agreed vocabulary) to a representative numeric value and interval, which parameterize a deterministic hybrid renderer to produce an audio reconstruction. The renderer employs a combination of harmonic, modal, and noise-synthesis layers.
• Closed-Loop Refinement: Decoder parameters (a compact control vector) are iteratively refined in a closed loop by re-extracting features from the rendered output and adjusting synthesis controls to minimize violations of the lexical constraints, optimizing for accuracy in lexical bin assignment.

Theoretical Framing

LAC is formalized as a finite-rate, lossy vector quantizer in feature space, with the transmitted sentence defining a Markovian channel: $S \to L \to \widetilde{S}$, where $S$ is the source, $L$ is the lexical state, and $\widetilde{S}$ the reconstruction. The quantization bottleneck is upper bounded by $\mathcal{A}_i$0 bits per sound, with the data processing inequality capping mutual information between source and reconstruction. Extreme bin counts ($\mathcal{A}_i$1, typical $\mathcal{A}_i$2) cap the payload at around several hundred bits per sound.

Empirical Evaluation

Dataset and Protocol

A new evaluation corpus of approximately 300 tracker modules (~3700 short samples, durations sub-2s), provides diverse, isolated timbral events. Symbolic musical structure is encoded separately in ABC-like notation; LAC is evaluated as the carrier solely for timbral (acoustic) attributes, decoupling musical content from sound characterization.

Reconstruction and Analysis

• Feature Family Ablation: Lexical-bin reconstruction accuracy is evaluated by cumulatively adding temporal, spectral, harmonic, and psychoacoustic feature classes. Pre-synthesis (direct inversion) recovers 100%; post-synthesis (after rendering and feature re-extraction) plateaus at approximately 74% accuracy, reflecting renderer approximability, not channel loss.
• Closed-Loop Refinement: Post-synthesis accuracy is further improved (to ~84%) by iterative optimization of the synthesis controls, at the cost of throughput (longer decoding time per sound). Diminishing returns are observed above 256 re-renderings per sample.
• Qualitative Results: Reconstructions preserve salient acoustic structure—envelope, periodicity, and timbral mass—even when sample-wise correspondence is weak. Listening examples reveal high interpretability and semantic editability via the text channel.

Comparative Positioning

LAC is situated in the auditory design space between descriptors, codecs, and captions. Unlike neural codecs (Défossez et al., 2022, Zeghidour et al., 2021), LAC is human-readable, fully amenable to LLM-native semantic editing, and exposes all transmission parameters in physically and perceptually meaningful axes—a trait absent in prior latent-token-based pipelines (Borsos et al., 2022, Agostinelli et al., 2023). Unlike captions, LAC's sentences are not just summaries but are invertible carriers supporting structured, information-preserving communication.

Limitations

The current system is intentionally not a waveform or speech codec. It does not transmit phone-level, speaker ID, lyrics, arrangement, or melody. LAC’s expressive bandwidth is bounded by the granularity and completeness of the lexical vocabulary and quantization; it is not designed for time-varying (non-stationary) signals, sustained notes, or arbitrary text-to-audio generation. Musical symbolic transfer is implemented using ABC notation, but unconstrained natural language descriptions struggle to encode complete arrangements. The feature design is not optimized, and the decoder renderer is deterministic and modular, not end-to-end learned.

Implications and Outlook

LAC sets a precedent for overt, interpretable, and semantically editable audio transport through plain text. Its design makes it particularly compatible with agentic and structured workflows involving LLMs: humans and machines can both read and alter the transmitted content, and LLMs can natively generate, audit, or manipulate sound by text. This opens new directions for AI-native interoperable formats, creative music workflows (timbral “remixing” via text edits), and semantic communications leveraging shared priors and language conventions (Jiang et al., 2023).

Risks include possible misuse for imitation/spoofing—making audio generation pipelines more transparent also lowers the barrier to creation of plausible forgeries if the sentence vocabulary is sufficiently rich.

Theoretically, LAC advances the notion of using language as a finite-rate, semantic-aware transport interface, suggesting new lossy source coding schemes that are human- and agent-aligned rather than purely, or even primarily, distortion-optimized.

Prospects for Future Work

• Extending LAC to time-varying and long-form audio via sliding-window or hierarchical sentence structures.
• Optimizing vocabulary design for maximal information rate per lexical token.
• Enriching the renderer to bridge the gap between lexical bin accuracy and perceptual fidelity, including hybrid approaches that combine LAC with neural codecs.
• Evaluating LAC-based communication in multi-agent, interactive, or conversational settings for music, sound design, or real-time control.

Conclusion

The paper establishes lexical acoustic coding as a viable, interpretable, and semantically controllable protocol for audio transport via natural language (2605.08750). By formalizing a pipeline where LLMs communicate not just about sound, but through sound rendered as text, it opens new spaces for agentic reasoning, manipulation, and co-creation of audio content that is both auditably structured and perceptually meaningful. LAC is particularly compelling for controllable, interpretable audio applications in the foundation model era, but its practical impact will depend on future advances in vocabulary engineering, rendering fidelity, and time-varying structure support.