Lexical Acoustic Coding (LAC)

• Lexical Acoustic Coding (LAC) is a framework that jointly encodes acoustic descriptors into structured natural language, enabling sound to be communicated as text.
• The system employs a sender–receiver architecture that deterministically maps multidimensional acoustic features to interpretable lexical codes for downstream tasks.
• Empirical findings across dialog act classification, speaker diarization, and emotion recognition highlight domain-dependent benefits and limitations of fusing lexical and acoustic modalities.

Searching arXiv for papers on “lexical acoustic coding” and closely related “lexico-acoustic” formulations. Lexical Acoustic Coding (LAC) denotes a class of representations and systems in which lexical information and acoustic information are jointly encoded for downstream inference, or, in the strictest recent usage, a framework that communicates audio entirely through natural language by quantizing acoustic descriptors into a structured lexical code (Rossi et al., 9 May 2026). The term itself is formalized in “Communicating Sound Through Natural Language” (Rossi et al., 9 May 2026), but conceptually related formulations appear earlier in dialog act classification, speaker diarization, escalation detection, spoken-word recognition, lexical access, and unsupervised lexical learning from raw speech, where lexical and acoustic cues are fused, aligned, or mutually constrained (Ortega et al., 2018). This broader record suggests that LAC names both a specific 2026 framework and a more general design pattern: the use of lexical structure as an explicit carrier, organizer, or complement for acoustic information.

1. Definition and scope

In its formal 2026 sense, Lexical Acoustic Coding is a framework in which pre-trained LLM sender and receiver agents transmit sound through natural language (Rossi et al., 9 May 2026). The sender analyzes an input waveform into a low-dimensional vector of handcrafted acoustic descriptors, quantizes each descriptor using a feature-specific lexical alphabet, and verbalizes the resulting lexical code as English; the receiver parses the sentence back into lexical-acoustic constraints and renders a waveform through closed-loop refinement (Rossi et al., 9 May 2026). The transmitted text is simultaneously a caption-like description and the transport representation itself (Rossi et al., 9 May 2026).

The paper frames LAC as a finite-rate lossy quantizer in feature space, with the source waveform $S$, lexical code $L$, and reconstruction $\widetilde{S}$ linked by the Markov chain $S \to L \to \widetilde{S}$ (Rossi et al., 9 May 2026). The maximum number of lexical states is

$|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$

with worst-case bit budget

$$B_{\max} := \log_2 |\mathcal{L}| = \sum_{i=1}^d \log_2 |\mathcal{A}_i|,$$

and the information bottleneck

$$I(S;\widetilde{S}) \le I(S;L) = H_{\mathrm{Sh}}(L) \le \log_2 |\mathcal{L}|$$

(Rossi et al., 9 May 2026).

A broader, retrospective use of the term is conceptually supported by earlier work. In “Lexico-acoustic Neural-based Models for Dialog Act Classification” (Ortega et al., 2018), the authors do not use the term LAC, but their lexico-acoustic model combines lexical and acoustic utterance-level encodings into a joint representation for classification. The supplied interpretation explicitly identifies this as “a neural instantiation of what you might call lexical–acoustic coding” (Ortega et al., 2018). A similar retrospective reading applies to multimodal speaker diarization via word embeddings plus MFCC aligned at word time steps (Park et al., 2018), and to acoustic–lexical feature concatenation for escalation detection (Zhou et al., 2021). This suggests that LAC can denote either a specific text-as-audio-transport protocol or a family of multimodal encoding strategies in which lexical and acoustic modalities are jointly represented.

2. Historical lineages before the formal term

Before “Lexical Acoustic Coding” was introduced as a named framework, several distinct research lines already instantiated its core idea in different technical forms.

One lineage concerns multimodal classification in spoken dialogue. The dialog act work of Ortega and Vu builds a lexico-acoustic model with a lexical encoder and an acoustic encoder whose outputs are concatenated for dialog act prediction (Ortega et al., 2018). The lexical branch constructs utterance representations from 300-dimensional word embeddings using a CNN, then applies an LSTM with attention over context utterances to obtain a context-aware lexical representation $\mathbf{l}_t$; the acoustic branch extracts 13-dimensional MFCCs with 25 ms frame length and 10 ms frame shift, applies a CNN over frames, and produces an utterance-level acoustic representation $\mathbf{a}_t$; the joint code is

$\mathbf{z}_t = [\mathbf{l}_t ; \mathbf{a}_t].$

This is an explicit late-fusion lexico-acoustic representation (Ortega et al., 2018).

A second lineage concerns speaker segmentation and diarization. In the seq2seq diarization system of Meng et al., each word token is aligned with a 13-dimensional MFCC vector averaged over that word’s duration, transformed and concatenated with a lexical embedding, and passed into a GRU encoder (Park et al., 2018). The input at word time step $L$0 is

$L$1

and the encoder state $L$2 can be read as a joint lexical–acoustic code (Park et al., 2018). Here the fusion is early rather than late, and the task is not semantic classification but speaker-change prediction and diarization.

A third lineage comes from human speech processing and lexical access. “Phonological (un)certainty weights lexical activation” models spoken-word recognition as a mapping from graded acoustic evidence to probability distributions over lexical items (Gwilliams et al., 2017). The acoustic-weighted model defines lexical probability as

$L$3

thereby propagating subphonemic acoustic uncertainty to lexical activation (Gwilliams et al., 2017). In a different but related tradition, the Italian lexical access model based on Stevens’ framework represents words as hierarchical arrangements of distinctive features recovered from landmarks and acoustic cues (Benedetto et al., 2021). That work does not use the LAC label, but the supplied interpretation characterizes it as “essentially an implementation of a lexical acoustic code in Stevens’ sense” (Benedetto et al., 2021).

A fourth lineage concerns learning lexical structure directly from raw acoustic output. “CiwGAN and fiwGAN” forces a generator to hide recoverable lexical or featural codes in generated waveforms, using an InfoGAN-style mutual-information objective (Beguš, 2020). In that setting, discrete latent variables act as compact lexical acoustic codes that determine word-like outputs and sometimes support productive recombination (Beguš, 2020).

Taken together, these strands indicate that LAC has been approached as late fusion, early fusion, probabilistic lexical weighting of acoustic evidence, distinctive-feature coding from landmarks, and discrete latent coding of lexical content in raw speech. The 2026 framework differs chiefly in making natural language itself the transport representation (Rossi et al., 9 May 2026).

3. Formal architecture of text-based LAC

The 2026 LAC framework is architecturally sender–receiver and explicitly deterministic at the feature-coding layer (Rossi et al., 9 May 2026). Given a mono waveform $L$4, the sender computes

$L$5

with $L$6 in the implementation (Rossi et al., 9 May 2026). The features are grouped into temporal, spectral, harmonic, psychoacoustic Bark-band, and psychoacoustic non-Bark families (Rossi et al., 9 May 2026). The temporal family includes rms_energy, crest_factor_db, zero_crossing_rate, log_attack_time, attack_slope_db_s, temporal_centroid, and decay_time_s; the spectral family includes spectral_centroid_hz, spectral_flatness, spectral_rolloff_hz, spectral_flux, spectral_kurtosis, spectral_entropy, and spectral_irregularity; the harmonic family includes f0_hz, harmonic_noise_ratio_db, inharmonicity, tristimulus_1/2/3, and odd_even_harmonic_ratio; there are 24 Bark-band features and two additional psychoacoustic features, sharpness_acum and roughness (Rossi et al., 9 May 2026).

Each feature $L$7 has a feature-specific lexical alphabet $L$8, and the shared vocabulary is

$L$9

where $\widetilde{S}$0 maps a numerical value to a lexical label, $\widetilde{S}$1 maps a lexical label to a numeric interval, and $\widetilde{S}$2 maps a lexical label to a representative value (Rossi et al., 9 May 2026). The lexical code is

$\widetilde{S}$3

A deterministic verbalizer $\widetilde{S}$4 maps $\widetilde{S}$5 to a sentence $\widetilde{S}$6,

$\widetilde{S}$7

under the requirement that every lexical term be preserved in recoverable form and that the inverse parser satisfy

$\widetilde{S}$8

(Rossi et al., 9 May 2026).

At the receiver, parsing gives

$\widetilde{S}$9

Each label is inverted via

$S \to L \to \widetilde{S}$0

For finite intervals, the representative is the midpoint

$S \to L \to \widetilde{S}$1

which is minimax-optimal within the bin; for $S \to L \to \widetilde{S}$2, the representative is the geometric mean

$S \to L \to \widetilde{S}$3

(Rossi et al., 9 May 2026). The decoded targets are assembled into $S \to L \to \widetilde{S}$4, a deterministic seed $S \to L \to \widetilde{S}$5 is computed, and a hybrid renderer produces

$S \to L \to \widetilde{S}$6

Closed-loop refinement re-analyzes

$S \to L \to \widetilde{S}$7

and optimizes

$S \to L \to \widetilde{S}$8

with derivative-free Powell search (Rossi et al., 9 May 2026).

The vocabulary is deliberately interpretable. The RMS alphabet includes “whisper”, “hushed”, “mid-power”, “forceful”, and “thunderous”; log_attack_time includes “onset-undetected”, “snap-onset”, “swift-onset”, “moderate-onset”, “gradual-onset”, and “creeping-onset”; Bark-band labels combine level words such as “silent”, “trace”, “faint”, “present”, “strong”, “dominant”, and “overwhelming” with band-specific nouns such as “rumble” and “air”; f0_hz is partitioned into 288 log-frequency bins with labels such as “lumen sol crown” (Rossi et al., 9 May 2026). The result is a code that is both machine-invertible and explicitly human-readable.

4. Modal fusion variants across tasks

Outside the 2026 text-transport formulation, the most important LAC distinction is how lexical and acoustic streams are aligned and fused.

In dialog act classification, the fusion is late and utterance-level. The lexical model uses word2vec-initialized 300-dimensional embeddings, filter widths 3, 4, and 5 with 100 feature maps per width, ReLU, dropout 0.5, utterance-wise max pooling, and an LSTM with output attention over context utterances (Ortega et al., 2018). The acoustic model uses 13 MFCCs extracted with openSMILE from 25 ms frames with 10 ms shift, a one-layer CNN whose filters span all 13 MFCC dimensions and 5 frames, 100 feature maps, ReLU, and max-pooling of size $S \to L \to \widetilde{S}$9 (Ortega et al., 2018). The final representation is the concatenation $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$0, optimized by cross-entropy with ASGD, initial learning rate 0.11 decayed by 10% every 2000 updates, and mini-batch sizes 50 for MRDA and 150 for SwDA (Ortega et al., 2018).

In speaker diarization, fusion is early and word-synchronous. Each word is represented by a lexical embedding and a 13-dimensional MFCC vector averaged over the word’s time span; these are linearly transformed, concatenated, and fed to a 256-unit GRU encoder (Park et al., 2018). The decoder is a 256-unit GRU with attention, trained to reconstruct words plus speaker turn tokens #A and #B (Park et al., 2018). A grouping-based loss resolves the speaker-label permutation issue by taking the minimum of losses under original and flipped #A/#B assignments (Park et al., 2018). The decoding stage then produces speaker-turn vectors over sliding windows of 32 words with overlap 31, aligns window-level labelings by possible flipping to reduce Hamming distance, and applies majority vote per word (Park et al., 2018).

In escalation detection, the fusion is feature-level concatenation with a shallow backend classifier. The acoustic stack is WebRTC-VAD with noise reduction mode 2, MFCC extraction in librosa with 25 ms Hamming windows, 10 ms hop, 256 mel filters, frequency range 50–8000 Hz, pre-emphasis factor 0.97, and a ResNet-18 plus global average pooling trained first on emotional datasets and then used as a representation extractor (Zhou et al., 2021). The lexical side uses manual Dutch transcripts, optionally translated to English, encoded by the multilingual Universal Sentence Encoder from Sentence-BERT, producing a 768-dimensional sentence embedding (Zhou et al., 2021). The fusion vector is explicitly stated as 1280-dimensional and is fed to a linear SVM (Zhou et al., 2021).

In speech emotion recognition, no multimodal fusion is implemented, but the paper’s findings are directly relevant to LAC because it compares strictly unimodal acoustic and lexical pipelines under matched conditions (Combei, 6 Sep 2025). The acoustic side uses frozen SSL speech encoders such as wav2vec2-xls-r-2b with temporal average pooling and a three-hidden-layer MLP; the lexical side uses Whisper-large-v3 transcripts, frozen text encoders such as DeBERTa or BERT-Large with token average pooling, and the same MLP (Combei, 6 Sep 2025). The paper explicitly leaves multimodal architectures and trainable modality-specific weights as future work (Combei, 6 Sep 2025). This suggests that, in this domain, LAC design must decide not only how to fuse modalities but whether fusion is necessary for a given dataset.

5. Empirical findings and domain dependence

Empirical results across these papers show that the value of lexical–acoustic coding is strongly task- and condition-dependent.

For dialog act classification, the lexical-only model achieves 84.1 on MRDA and 73.6 on SwDA, the acoustic-only model 67.8 and 50.9, and the lexico-acoustic model 84.7 and 75.1, respectively (Ortega et al., 2018). Acoustic features therefore improve overall accuracy modestly, with larger gains on SwDA (+1.5 absolute) than on MRDA (+0.6) (Ortega et al., 2018). The paper’s deeper analysis identifies three cases where acoustic features help: when a dialog act has sufficient training data, when lexical information is limited, and when strong lexical cues are absent (Ortega et al., 2018). On MRDA, utterances consisting only of “Right” or only of “Yeah” show improved F1 for Statement under the lexico-acoustic model; removing the question mark from transcripts increases the improvement of LAM over LM from 0.6% to 1.1%, and Question accuracy on formerly punctuated utterances becomes 50.2 for LAM versus 46.6 for LM (Ortega et al., 2018). These results explicitly tie the benefit of acoustic coding to lexical ambiguity and impoverished transcripts.

For speaker diarization with reference transcripts, the lexical-only seq2seq model yields DER 28.02 on Fisher and 27.89 on Switchboard, whereas the lexical-plus-MFCC model yields 24.26 and 22.44 (Park et al., 2018). Word-level DER improves from 16.42 to 12.32 on Fisher and from 12.40 to 8.56 on Switchboard (Park et al., 2018). With ASR transcripts, however, the lexical-plus-MFCC system degrades to 50.95 DER on Switchboard ASR while lexical-only yields 38.64, though both still outperform LIUM at 66.57 (Park et al., 2018). The paper links this to ASR quality: high WER implies high DER, and low WER is necessary but not sufficient for low DER (Park et al., 2018). A plausible implication is that LAC systems with word-synchronous acoustic averaging are especially sensitive to transcript and alignment noise.

For escalation detection, acoustic transfer learning dominates. MFCC yields 0.675 UAR, MFCC+VAD 0.710, MFCC+TE 0.690, and MFCC+VAD+TE 0.721 on the development set (Zhou et al., 2021). Emotion-pretrained acoustic representations raise MFCC+VAD to 0.810, and adding text leaves the score at 0.810 (Zhou et al., 2021). The paper also reports that English textual embeddings alone reach approximately 45% UAR qualitatively, far below acoustic-only performance (Zhou et al., 2021). Here LAC contributes modestly without pretraining and not at all once the acoustic encoder becomes strong, indicating a domain in which prosodic and affective cues dominate the lexical contribution.

For speech emotion recognition on MELD, the lexical-only system using Whisper plus DeBERTa Layer 19 reaches 51.5% WF1 on test, whereas the best acoustic system reaches 49.3% WF1 (Combei, 6 Sep 2025). On the development set, DeBERTa Layer 19 reaches 51.73% WF1 with ASR transcripts and 60.9% with manual transcripts (Combei, 6 Sep 2025). The paper’s central claim is that lexical coding alone can match or surpass acoustic-only models on this content-rich conversational dataset (Combei, 6 Sep 2025). At the same time, the acoustic side remains strong and denoising generally degrades both modalities, suggesting that paralinguistic cues and transcript fidelity must be handled carefully (Combei, 6 Sep 2025). This is the clearest case in the supplied material where lexical information can be dominant.

For text-as-transport LAC, the feature-family ablation shows that pre-synthesis lexical accuracy rises monotonically and reaches 100% when all feature groups are included; post-synthesis lexical accuracy reaches approximately 84% with refinement and approximately 72–74% without refinement (Rossi et al., 9 May 2026). Bark bands contribute the largest improvement, temporal and psychoacoustic features add little further improvement, and increasing Powell search evaluations improves lexical-bin match at the cost of runtime (Rossi et al., 9 May 2026). These figures isolate distortion due not to the text channel but to the renderer.

6. Linguistic, cognitive, and generative interpretations

Several papers in the record use structures that can be interpreted as LAC in a stronger representational sense, not merely as multimodal feature fusion.

The spoken-word recognition study (Gwilliams et al., 2017) models lexical activation as weighted by both phonological certainty and lexical frequency early in a word, and by frequency alone later. At the second phoneme, acoustic-weighted predictors significantly improve fit to left superior temporal gyrus activity, with $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$1, $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$2, while switch-based predictors do not; at the sixth phoneme, switch-based predictors are significant with $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$3, $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$4, and acoustic-weighted predictors are only marginal with $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$5, $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$6 (Gwilliams et al., 2017). The paper thereby defines a dynamic lexical acoustic code in probabilistic form: early lexical representations preserve graded phonological uncertainty, later ones approximate discrete cohort selection.

The Italian lexical access model (Benedetto et al., 2021) provides a different cognitive formalization. Words are represented as hierarchical bundles of distinctive features, and acoustic landmarks serve as anchors for feature recovery (Benedetto et al., 2021). The code can be written as

$|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$7

where each landmark type and feature bundle is inferred from acoustic cues (Benedetto et al., 2021). The feature inventory includes articulator-free and articulator-bound features such as Vowel, Glide, Cons, Cont, Son, Strid, Lips, Blade, Body, Ant, Dist, Lat, High, Low, Back, Round, ATR, CTR, Nasal, Spread glottis, Constr glottis, Stiff, and Slack (Benedetto et al., 2021). This is not a neural fusion model; it is a hypothesis about the representational interface between acoustics and lexical access.

The GAN-based lexical learning work (Beguš, 2020) offers a generative version of LAC. ciwGAN uses one-hot categorical codes, fiwGAN binary featural codes, and both maximize a mutual-information term between latent code and generated waveform through an InfoGAN extension of WaveGAN (Beguš, 2020). The objective is

$|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$8

with $|\mathcal{L}| = \prod_{i=1}^d |\mathcal{A}_i|,$9 and code variables either categorical $$B_{\max} := \log_2 |\mathcal{L}| = \sum_{i=1}^d \log_2 |\mathcal{A}_i|,$$0 or binary $$B_{\max} := \log_2 |\mathcal{L}| = \sum_{i=1}^d \log_2 |\mathcal{A}_i|,$$1 (Beguš, 2020). In the 8-word fiwGAN, 3 binary features encode up to $$B_{\max} := \log_2 |\mathcal{L}| = \sum_{i=1}^d \log_2 |\mathcal{A}_i|,$$2 classes; in the full TIMIT experiment, 13 features encode up to $$B_{\max} := \log_2 |\mathcal{L}| = \sum_{i=1}^d \log_2 |\mathcal{A}_i|,$$3 possible codes for 6,229 lexical types (Beguš, 2020). The paper reports innovative outputs such as “start” from training on “suit” and “dark,” arguing that phonetic and phonological representations learned by the network can be productively recombined (Beguš, 2020). This suggests a form of LAC in which lexical structure is not merely fused with acoustics but embedded in a compact, manipulable discrete code recoverable from audio.

These lines of work support different interpretations of LAC: as a multimodal engineering pattern, as a probabilistic interface between acoustic evidence and lexical hypotheses, as a distinctive-feature code anchored by landmarks, and as a compact latent representation that induces lexical structure in waveforms. The formal 2026 framework adds a further twist by making the code itself ordinary English (Rossi et al., 9 May 2026).

7. Limitations, misconceptions, and open directions

A common misconception is that LAC necessarily refers to a single established architecture. The record does not support that view. Only the 2026 paper uses “Lexical Acoustic Coding” as a formal name (Rossi et al., 9 May 2026). Earlier work uses terms such as “lexico-acoustic model,” “word embeddings + MFCC,” “acoustic-lexical information fusion,” or feature-based lexical access (Ortega et al., 2018). It is therefore more precise to distinguish the named LAC framework from conceptually related lexical–acoustic coding strategies.

A second misconception is that combining lexical and acoustic information is uniformly beneficial. The evidence is mixed. Acoustic cues help dialog act classification especially when lexical cues are weak or ambiguous (Ortega et al., 2018); they improve diarization with clean transcripts but can hurt when ASR transcripts are noisy (Park et al., 2018); lexical cues help escalation only modestly and not once acoustic transfer learning becomes strong (Zhou et al., 2021); lexical-only models can surpass acoustic-only models for SER on MELD but would not be expected to work well on datasets like RAVDESS where the same lexical content is spoken with multiple emotions (Combei, 6 Sep 2025). The balance between lexical and acoustic coding is thus domain-dependent.

The text-based LAC framework has its own limitations. It is designed for short, isolated, non-speech sounds, not as a general-purpose speech or music codec (Rossi et al., 9 May 2026). Experiments with overlapping-window descriptions were fragile, exact waveform recovery is not intended, and the framework depends on prompt design and LLM behavior (Rossi et al., 9 May 2026). Symbolic music transfer separates structure from timbre and remains basic (Rossi et al., 9 May 2026). These constraints delimit the current meaning of LAC in its strict formal usage.

Across the broader literature, several future directions recur. Dialog act work points to context-aware acoustic modeling, richer acoustic features beyond MFCCs, more advanced multimodal fusion, pretraining, and robustness to ASR errors (Ortega et al., 2018). Diarization work highlights multiple ASR hypotheses, extension beyond two speakers, overlap handling, richer speaker embeddings, and confidence-weighted fusion (Park et al., 2018). SER work points to multimodal architectures with trainable modality-specific weights, SER-aware denoising, and ASR error correction tailored to emotion (Combei, 6 Sep 2025). The Italian lexical-access program calls for probabilistic lexical decoding over hierarchical feature bundles and cross-language validation of landmark universals (Benedetto et al., 2021). The text-transport LAC framework suggests improved vocabularies, better binning schemes, more structured parsing and decoding, time-varying representations, and hybrid symbolic–continuous models (Rossi et al., 9 May 2026).

In that sense, LAC is best understood not as a settled paradigm but as an evolving family of representational strategies organized around a single question: how lexical structure and acoustic structure can be encoded together, whether for inference from speech, generation of speech-like forms, or communication of sound itself.