Discrete Audio Token Generation

Updated 6 November 2025

Discrete audio token generation is the process of converting audio signals into sequences of discrete symbols using quantization and clustering techniques.
It employs methods such as residual vector quantization for acoustic tokenization and self-supervised clustering for semantic token generation to balance fidelity and coherence.
This approach enables high-fidelity audio synthesis, scalable compression, and integration into multimodal language models for diverse real-world applications.

Discrete audio token generation refers to the process of transforming audio signals—typically speech, music, or general sound—into sequences of discrete symbols ("tokens"), enabling audio to be processed, modeled, and generated using techniques originally developed for language modeling. This paradigm underpins advances in high-fidelity audio generation, large multimodal language modeling, and efficient audio compression. Discrete audio tokens can encode fine-grained acoustic signals (acoustic tokens), higher-level phonetic and semantic structures (semantic tokens), or both in hybrid systems.

1. Motivations and Foundations

The core motivation for discrete audio token generation is to exploit the success of next-token prediction (autoregressive) models—especially Transformers—in domains beyond natural language. By representing audio as a token sequence, language modeling frameworks become directly applicable, facilitating coherent generative modeling of long audio contexts, interactive audio synthesis, and unified handling of multi-modal tasks.

However, unlike text, where tokenization is deterministic and context-independent, audio presents unique challenges: tokenizing signals must balance the preservation of local acoustic fidelity, long-range structural or semantic coherence, and the ability to reconstruct perceptually natural sound. This balance often requires distinct token types and careful engineering of quantization schemes, vocabulary size, and hierarchical modeling within token generation pipelines (Borsos et al., 2022).

2. Tokenization Methodologies

2.1 Acoustic Token Generation

Acoustic tokens focus on capturing frame-level, low-level waveform or spectro-temporal structure to facilitate high-fidelity reconstruction. The dominant methodology employs neural audio codecs with residual vector quantization (RVQ), such as SoundStream, EnCodec, and RVQ-GANs:

A neural encoder (usually a convolutional or hybrid CNN/RNN) reduces the audio to slow frame-rate embeddings.
RVQ hierarchically quantizes each embedding through multiple codebooks. For each frame, the embedding is iteratively quantized, yielding a tuple of discrete indices. The number of codebooks and their size set the bitrate and information capacity; e.g., $Q$ layers with $N$ entries each gives $Q$ tokens/frame (Borsos et al., 2022, Shechtman et al., 10 Oct 2024).
The tokens can be flattened in row-major order for language modeling or sequence learning.

Acoustic token systems achieve ViSQOL scores of 3.3–3.9 at 2–6 kbps, supporting nearly perceptually transparent speech at extremely low bitrates (e.g., 150–300 tokens/s at 3 kbps) (Shechtman et al., 10 Oct 2024).

2.2 Semantic Token Generation

Semantic tokens capture high-level content, such as phonetics or word-like structure. They are extracted from self-supervised learning (SSL) models—e.g., HuBERT, w2v-BERT, WavLM—by clustering the hidden representations:

Intermediate-layer embeddings are extracted at a lower temporal rate (e.g., 25 Hz).
$k$ -means clustering is performed on these embeddings (vocabulary size often 1k–2k), yielding discrete token indices.
Tokens from different SSL model layers may be used in parallel and, empirically, carry different information: earlier layers emphasize low-level acoustics; deeper layers encode semantics (Mousavi et al., 15 Jun 2024).

Semantic tokens are highly effective for capturing linguistic content (low ABX error rates, high discriminative-task accuracy), but perform poorly when used alone for fine-grained speech synthesis (ViSQOL $\approx$ 1.1–1.4) (Borsos et al., 2022).

2.3 Hybrid and Hierarchical Schemes

Hybrid tokenization combines both classes. In hierarchical modeling (as in AudioLM), semantic tokens are generated and modeled first to establish long-term structure, with acoustic tokens conditioned on them to recover local detail:

Stage 1: Model semantic token sequence
Stage 2: Autoregressively generate coarse/fine acoustic tokens, conditioned on semantics (Borsos et al., 2022, Lee et al., 25 Jun 2024).

This structure permits decoupling global structure from surface realization, supporting both long-term coherence and high signal fidelity.

Token Type	Captures	Bitrate	Fidelity	Structure
Acoustic	Signal details, timbre	High/Med	High	Poor
Semantic	Phonemes, words, events	Low/Med	Low	Strong
Hybrid	Both	Med	High	Strong

3. Quantization, Modeling, and Training Paradigms

3.1 Quantization Techniques

RVQ is the standard quantizer for acoustic tokens, enabling hierarchically scalable bitrate and fidelity. Semantic tokens rely on post-hoc clustering (k-means or PQ) of SSL representations, sometimes over multiple layers with learned attention to combine their importance for a given task (Mousavi et al., 15 Jun 2024).

Innovations such as query-based global compression (learnable queries via transformers) (Yang et al., 14 Apr 2025) and BPE/token-aggregation of discrete sequences (Shen et al., 2023, Dekel et al., 8 Jun 2024), further compress and regularize tokenization, addressing sequence length and learning stability.

3.2 Language Modeling and Sequence Learning

Audio tokens are modeled using autoregressive, decoder-only Transformers. For hierarchical/hybrid schemes, conditional independence assumptions (e.g., semantics $\perp$ past acoustics $|$ past semantics) justify splitting sequence modeling into stages, each maximizing the likelihood over the relevant token stream (Borsos et al., 2022):

$\prod_{t=1}^{T'}p(h_{t} | h_{<t})$

Sequence flattening strategies ensure appropriate serialization for autoregressive sampling, maintaining distinction among different quantizer layers in acoustic modeling.

3.3 Training Objectives

Acoustic codecs: Minimize waveform reconstruction (L $_1$ , L $_2$ ), adversarial, and multi-scale spectral losses.
Semantic tokenizers: Cluster SSL representations; occasionally use masked prediction or supervised audio tagging objectives when event-level semantics are desired (Tian et al., 21 May 2025).
Hybrid systems: Joint objectives balance fidelity and latent semantic structure, or perform staged training.

Auxiliary losses may enforce codebook diversity, semantic disentanglement (speaker vs. content), and adaptation to downstream language modeling (Yang et al., 14 Apr 2025).

4. Trade-offs, Challenges, and Solutions

4.1 Fidelity vs. Structure

Acoustic tokens optimize for reconstruction but have weak long-term or structural modeling. Semantic tokens foster coherence, syntax, and semantic richness but are incapable of detailed resynthesis. Hybrid models, particularly those using multi-stage generation or hierarchical LM (as in AudioLM), achieve both high ViSQOL and low WER, as verified by both automatic and subjective evaluation (Borsos et al., 2022).

4.2 Efficiency and Sequence Compression

High-fidelity codecs generate very long token sequences, creating context-length and efficiency concerns for Transformers. Sequence length can be reduced by:

Increasing codebook size.
Applying acoustic/subword BPE to frequent patterns (Shen et al., 2023).
Adopting group quantization and multi-resolution RVQ (Liu et al., 30 Oct 2025).

Sequence compression (BPE or BPE-like approaches) reduces both inference latency and exposure bias in generation (Dekel et al., 8 Jun 2024).

4.3 Robustness and Consistency

Tokenization of identical audio segments can be inconsistent, especially when context differs or during synthesis–retokenization loops (discrete representation inconsistency, DRI). Neural codecs, especially with deep receptive fields, exacerbate this effect. Mitigations include slice-consistency and perturbation-consistency losses to enforce stability in RVQ models (Liu et al., 28 Sep 2024).

5. Evaluation, Benchmarks, and Empirical Performance

Common evaluation setups span reconstruction, downstream discriminative tasks (ASR, speaker/emotion recognition), and generative tasks (TTS, separation, enhancement). Consistent benchmarks such as DASB (Mousavi et al., 20 Jun 2024) have standardized the comparison of tokenizers:

Semantic tokens outperform acoustic/compression tokens for ASR, emotion, and intent tasks at comparable bitrate.
Acoustic tokens (codecs) retain more paralinguistic and speaker information.
Performance is typically evaluated using word error rates (WER), UTMOS (speech quality), FAD (audio quality), and speaker similarity metrics.
Continuous (non-discretized) SSL embeddings retain a notable performance advantage for most discriminative tasks.

Subjective listening experiments confirm that advanced tokenization (e.g., RVQ-GAN at low rate) can achieve perceptually transparent reconstructions (Shechtman et al., 10 Oct 2024).

Tokenizer	Best on ASR	Best on SID/SV	Fidelity	Compression
Semantic	Yes	No	Low	High
Compression	No	Yes	High	Moderate
Hybrid	Moderate/Best	Moderate	Best	Balanced

6. Applications and Impact

Discrete audio tokens underpin:

High-fidelity speech and music generation (TTS, audio continuation, music modeling), as in AudioLM (Borsos et al., 2022), AudioGen (Kreuk et al., 2022), UniAudio (Yang et al., 2023), and UniTok-Audio (Liu et al., 30 Oct 2025).
Multimodal LLMs that integrate audio and text tokens for reasoning and generation (Verma, 16 Dec 2024, Mehta et al., 28 Mar 2025), facilitated by token-based abstraction.
Efficient, scalable storage and streaming of compressed audio content.
Downstream tasks such as ASR, speaker and emotion recognition, audio event classification, and audio captioning (Tian et al., 21 May 2025).

Recent efforts address unified learning for multi-task audio generation, robust and watermarkable generative systems (Wu et al., 24 Oct 2025, Liu et al., 30 Oct 2025), and fair benchmarking (Mousavi et al., 12 Jun 2025).

7. Limitations and Future Directions

Despite progress, several limitations remain:

The discrete–continuous gap: Semantic tokens, while efficient, still lag behind continuous features in downstream discriminative tasks, and compression codecs may lose semantic detail (Mousavi et al., 20 Jun 2024).
Robust domain adaptation: Tokenizers trained on one domain/generalization regime often degrade on others.
High-quality generation for very long contexts remains challenging due to sequence length/computation scaling (Verma, 16 Dec 2024, Yang et al., 2023).
Inconsistent tokenization (DRI) can degrade the generation stability of neural codec LLMs unless explicitly mitigated (Liu et al., 28 Sep 2024).
Effective integration of discrete tokens into foundation multimodal LLMs, balancing generation, comprehension, and interpretability, is an open area.

Research is advancing toward modular tokenizers (joint acoustic/semantic), improved losses for both fidelity and semantics, enhanced sequence compression, scalable and efficient models for real-time applications, and standard evaluation frameworks (Mousavi et al., 12 Jun 2025, Liu et al., 30 Oct 2025).

References Table (Key Studies by Tokenization Category)

Approach	Methodology	Representative Papers
Acoustic	RVQ neural codecs	(Borsos et al., 2022, Shechtman et al., 10 Oct 2024, Puvvada et al., 2023)
Semantic	SSL embedding + k-means	(Borsos et al., 2022, Mousavi et al., 15 Jun 2024, Tian et al., 21 May 2025, Mousavi et al., 20 Jun 2024)
Hybrid/Hierarchical	Joint modeling, distillation	(Borsos et al., 2022, Liu et al., 30 Oct 2025, Lee et al., 25 Jun 2024)
Sequence Compression	BPE, query tokens	(Shen et al., 2023, Dekel et al., 8 Jun 2024, Yang et al., 14 Apr 2025)
System Integration	LLM, universal generation	(Yang et al., 2023, Liu et al., 30 Oct 2025, Verma, 16 Dec 2024)

Discrete audio token generation continues to be a rapidly evolving field, foundational both for high-performance audio synthesis and for integrating audio into the new generation of foundation multimodal AI systems.