Disentangled Speech Tokenization

• Disentangled speech tokenization is a method that separates semantic content (what is said) from acoustic features (how it is said) to enable independent manipulation.
• Architectures like Residual Vector Quantization and dual-stream designs leverage dedicated codebooks and loss functions to optimize semantic alignment and acoustic detail extraction.
• This approach underpins robust applications such as voice conversion, emotion transfer, and multimodal integration with large language models.

Speech tokenization with semantic–acoustic disentanglement refers to the process of converting continuous speech signals into discrete token sequences such that tokens separately (and explicitly) encode (1) semantic/linguistic content (“what is said”) and (2) acoustic/prosodic/emotional detail (“how it is said”). This paradigm is foundational for speech LLMs, neural codecs, and multimodal AI systems, enabling controllable generation, high-fidelity synthesis, robust downstream understanding, and seamless integration with LLMs. The core technical challenge lies in extracting and quantizing information such that semantic and acoustic streams are statistically and functionally disentangled, supporting independent manipulation and robust transfer across diverse applications.

1. Motivation and Core Principles

The motivation for semantic–acoustic disentanglement arises from the polyphonic nature of speech signals: spoken audio jointly encodes lexical (word-level) content, phonetic structure, prosody, speaker identity, and emotional nuances. Conventional tokenizers based on self-supervised learning (SSL) or codec-style quantization typically entangle these factors, impeding fine-grained control and degrading performance on tasks such as voice conversion, emotion transfer, zero-shot TTS, and multimodal LLM integration.

Disentangled tokenization is premised on the following principles:

• Factorization: Semantic tokens should robustly encode abstracted linguistic units (phonemes, words, sentences) invariant to speaker and prosody, while acoustic tokens should capture the remaining detail: speaker traits, intonation, timbre, background noise, and affect (Jung et al., 9 Jul 2025, Chen et al., 19 Oct 2025, Zhang et al., 14 Jan 2026).
• Modularity for Control: Separate token streams enable targeted manipulation (e.g., swap only the “how” tokens for voice transfer or anonymization) (Jung et al., 9 Jul 2025, Khurana et al., 18 Jun 2025, Wizadwongsa et al., 15 Jun 2026).
• Alignment with LLMs: Discrete semantic tokens, when well-aligned to text units, allow for plug-and-play speech/text fusion in LLMs (Jo et al., 20 Jun 2025, Ahasan et al., 2024, Jiang et al., 15 Mar 2025, Song et al., 29 May 2026).
• Compression: Both channels must be sufficiently information-dense to enable high-fidelity reconstruction and interpretation at low bitrate (Chen et al., 19 Oct 2025, Wizadwongsa et al., 15 Jun 2026, Jung et al., 9 Jul 2025).

2. Architecture and Methodological Variants

2.1 RVQ and Layered Quantization

Residual Vector Quantization (RVQ) underpins many modern disentangled tokenizers. Generally, the architecture consists of (a) a self-supervised semantic encoder (often HuBERT or WavLM), (b) a sequence of codebooks applied hierarchically (with the first capturing semantic content, subsequent codebooks quantizing residuals to capture acoustic information), and (c) a decoder (neural vocoder or neural codec) that reconstructs waveform or spectrogram from concatenated token embeddings (Jung et al., 9 Jul 2025, Khurana et al., 18 Jun 2025, Zhang et al., 2023, Zhang et al., 14 Jan 2026, Chen et al., 19 Oct 2025).

• Semantic codebook (e.g., $z_0$): Trained to minimize cross-entropy with pseudo-labels from a HuBERT/Whisper teacher. Dimensionally, $K_0 = 1024$, $D = 256$ is typical.
• Acoustic residual codebooks (e.g., $z_1$, $z_2$): Trained via reconstruction and, increasingly, via explicit distillation from speaker/prosody models (e.g., ECAPA-TDNN embeddings), with $K_1 = 512$, $D = 256$ per codebook (Jung et al., 9 Jul 2025, Zhang et al., 14 Jan 2026).
• Disentanglement mechanism: Only the first codebook receives semantic loss, while acoustic codebooks are optimized for orthogonal (e.g., speaker or style) objectives.

2.2 Factorized and Dual-Stream Designs

Recent work introduces explicit parallel streams (or factorized bottlenecks) for hierarchical disentanglement beyond semantic–acoustic, such as lexical, phonetic, and acoustic streams (Khurana et al., 18 Jun 2025, Chen et al., 19 Oct 2025, Jiang et al., 15 Mar 2025):

Model	Token Streams	Disentanglement Enforcement
HAC (Khurana et al., 18 Jun 2025)	Acoustic / Phonetic / Lexical	Dual distillation: HuBERT (phonemes), LaBSE (words)
SAC (Chen et al., 19 Oct 2025)	Semantic / Acoustic	Frozen semantic encoder, split losses
DSA-Tokenizer (Zhang et al., 14 Jan 2026)	Semantic / Acoustic	Dual CTC (ASR) vs. Mel restoration (recon.)

Each stream may have codebooks of differing depth/rate, e.g., 7 RVQ for acoustic, 1 for phonetic, 1 for lexical (Khurana et al., 18 Jun 2025).

2.3 Contextual and Multimodal Distillation

Models such as DM-Codec (Ahasan et al., 2024) and UniCodec (Jiang et al., 15 Mar 2025) integrate contextual signals (from LLMs, e.g., BERT or ELECTRA) as direct supervision for semantic/lexical token streams, while reserving lower layers for phonetic/acoustic encoding. Weighted distillation losses enforce stratified information mapping, and group-wise VQ is used for global (speaker/style) vs. local (semantic/prosodic) tokenization. This yields unified or tri-partite token streams conducive to downstream multimodal generation and robust, prosody-aware speech modeling.

3. Training Objectives and Disentanglement Losses

Training regimes for disentangled speech tokenization typically combine:

• Reconstruction loss: Waveform or spectrogram MSE/L1/L2 to drive overall fidelity.
• Semantic (alignment) loss: Cross-entropy/distance between semantic code embeddings and SSL teacher (e.g., HuBERT output) (Zhang et al., 2023, Jung et al., 9 Jul 2025).
• Acoustic distillation loss: L2/cosine on acoustic code embeddings to match external prosody or speaker embeddings (e.g., ECAPA-TDNN) (Jung et al., 9 Jul 2025).
• Adversarial/Feature-matching loss: Hinge or LSGAN on the reconstructed audio for perceptual quality (Khurana et al., 18 Jun 2025, Chen et al., 19 Oct 2025).
• Codebook commitment loss: As in VQ-VAE, to stabilize codebook updates (Chen et al., 19 Oct 2025, Khurana et al., 18 Jun 2025).
• Contextual loss (in some models): Cosine or contrastive loss against LLM embeddings for higher-level semantic alignment (Ahasan et al., 2024, Jo et al., 20 Jun 2025).
• Disentanglement/orthogonality (in some models): Indirectly enforced via careful loss routing (only semantic codebook supervised by content loss, only acoustic codebooks by speaker/timbre/prosody objectives) (Jung et al., 9 Jul 2025, Zhang et al., 14 Jan 2026).

4. Token Representation, Frame Rate, and Practical Encoding

Token rates are determined by the stride of the encoder network (typ. 20 ms or 40 ms, i.e., 50 Hz or 25 Hz). Each frame yields a semantic token and 1–8 acoustic tokens, depending on model configuration (Jung et al., 9 Jul 2025, Zhang et al., 14 Jan 2026, Chen et al., 19 Oct 2025). For hierarchical or groupwise VQ, global tokens (e.g., speaker/style continuous embedding) may be provided as well (Jiang et al., 15 Mar 2025, Huang et al., 31 Jan 2026).

• Token stream formation: At inference, semantic tokens are extracted per frame, and the residual encoder outputs a vector of acoustic token indices. The sequence length matches the number of audio frames.
• Rate/fidelity trade-off: Higher frame rate or more codebooks produce higher audio fidelity but increase sequence length, placing demands on downstream sequence models and transmission bitrates (Jo et al., 20 Jun 2025, Zhang et al., 14 Jan 2026, Jung et al., 9 Jul 2025).
• Frame alignment and pooling: Architectures such as LM-SPT (Jo et al., 20 Jun 2025) and Kanade (Huang et al., 31 Jan 2026) employ variable frame rates, pooled or adaptive stride, and pooling to align token sequence length to LLM context windows.

5. Empirical Evaluation and Information Metrics

Empirical evaluation of semantic–acoustic disentanglement is multifaceted. Key reported metrics include:

• Reconstruction Quality: PESQ, UTMOS, MUSHRA, ViSQOL, SI-SDR, STFT/Mel distortion (Jung et al., 9 Jul 2025, Khurana et al., 18 Jun 2025, Chen et al., 19 Oct 2025, Zhang et al., 14 Jan 2026).
• Speech Recognition (ASR): WER (Word Error Rate), CER (Character Error Rate) on tokenized/reconstructed speech (Jung et al., 9 Jul 2025, Wizadwongsa et al., 15 Jun 2026, Chen et al., 19 Oct 2025).
• Disentanglement Probes:
  • Speaker Verification (EER/Accuracy, SIM, F0Corr): Speaker and prosody separation (high classification on acoustic tokens, low on semantic; vice versa for content).
  • Word/Phoneme Probing (ABX, PNMI, CKA, MI): Layer-wise cluster analysis and cross-modal alignment to assess semantic versus phonetic encoding (Shi et al., 11 Mar 2026, Khurana et al., 18 Jun 2025, Zhang et al., 2023).
  • Voice/Emotion Conversion: MOS and similarity on cross-utterance style/content swap tasks (Jung et al., 9 Jul 2025, Zhang et al., 14 Jan 2026, Yang et al., 27 Jun 2025).
  • Downstream LLM / Multimodal Retrieval: Recall@1, cross-modal CKA, SLM performance on content and speaker/affect tasks (Jung et al., 9 Jul 2025, Song et al., 29 May 2026).

Consistently, multi-stream or hierarchical models yield state-of-the-art reconstruction while enabling a trade-off between content retention and style flexibility—e.g., DSA-Tokenizer achieves near-perfect separation in disentanglement probing, with semantic tokens yielding WER ≈6.3% and chance-level speaker classification, and vice versa for acoustic tokens (Zhang et al., 14 Jan 2026).

6. Downstream Applications and Impact

The semantic–acoustic disentangled paradigm powers a wide spectrum of applications:

• Speech Coding: Low-bitrate, high-fidelity, robust speaker and emotion preservation (Jung et al., 9 Jul 2025, Chen et al., 19 Oct 2025, Jiang et al., 15 Mar 2025).
• Voice Conversion/Anonymization: Swap semantic and acoustic streams to independently modulate content and style (Jung et al., 9 Jul 2025, Zhang et al., 14 Jan 2026, Khurana et al., 18 Jun 2025).
• Emotion and Speaker Recognition: Encode emotion and speaker identity in acoustic tokens for classification and downstream adaptive synthesis (Jung et al., 9 Jul 2025, Jiang et al., 15 Mar 2025).
• Controllable and Zero-Shot TTS: Plug-and-play synthesis by supplying semantic tokens from text and acoustic tokens (or global embedding) from prompts, supporting prosody transfer and speaker adaptation (Kim et al., 2024, Lee et al., 2024, Zhang et al., 14 Jan 2026).
• Multimodal/Multilingual LLMs: Feed semantic tokens to LLMs and acoustic tokens to affect or speaker modules for advanced language+speech reasoning (Jung et al., 9 Jul 2025, Jo et al., 20 Jun 2025, Song et al., 29 May 2026).
• Audio-LLM Generalization: Extension to non-speech (music, ambient audio) by combining semantically rich and acoustically selective representations (Song et al., 29 May 2026).

7. Future Directions and Open Challenges

Active areas of research and identified limitations include:

• Teacher Model Dependence: Most pipelines rely on frozen SSL encoders (e.g., HuBERT, WavLM) and acoustic proxies (e.g., ECAPA-TDNN); interest is growing in end-to-end co-training and alternative supervisors (Jung et al., 9 Jul 2025, Zhang et al., 14 Jan 2026).
• Fine-Grained Hierarchical Disentanglement: Extending and subdividing acoustic streams by prosody, emotion, environment, and multi-scale (syllabic, word, utterance) attributes (Jung et al., 9 Jul 2025, Khurana et al., 18 Jun 2025, Jiang et al., 15 Mar 2025).
• Token Rate Compression and Stability: Designing frame-pooling and reduction for efficient LLM integration without semantic collapse (Jo et al., 20 Jun 2025, Song et al., 26 Sep 2025).
• General Audio and Universal Tokenizers: Broadening the audio interface to non-speech domains (music, scenes). Mechanisms such as SAE and SAP show promise in this area (Song et al., 29 May 2026).
• Noise-Robust and Stable Tokenization: Architectures such as StableToken address the fragility of semantic token streams to noise, augmenting with multi-branch/voting schemes to ensure consistent input for speech LLMs (Song et al., 26 Sep 2025).
• Cross-Modal Consistency and Semantic Alignment: Ensuring that “semantic tokens” truly align with text semantics, not just phonetic forms, via distillation from LLMs and contrastive objectives (Shi et al., 11 Mar 2026, Ahasan et al., 2024, Jo et al., 20 Jun 2025).

A plausible implication is that unified, highly disentangled tokenization will underpin new generations of speech and multimodal models, yielding robust, interactive systems capable of precise, controllable, and generalizable audio reasoning at scale.