Discrete Audio Tokens

• Discrete audio tokens are compact symbolic representations derived from quantizing latent audio features, enabling efficient storage and classification.
• They power diverse applications like speech synthesis, music generation, and audio captioning by balancing reconstruction fidelity and compression rates.
• Integration with large language models facilitates multimodal processing, though challenges remain in standardizing evaluation and addressing token inconsistency.

Discrete audio tokens are compact, symbolic representations derived from quantization of latent features in audio signals, typically obtained via neural audio codecs or clustering in self-supervised learning (SSL) models. Unlike continuous spectral or waveform features, discrete tokens offer efficiency in storage, modeling, and integration with LLMs, framing audio modeling as a classification problem over a finite vocabulary. These representations enable scalable and multimodal processing in speech, music, and general audio domains, but their practical utility involves complex trade-offs among reconstruction fidelity, semantic preservation, compression rate, and downstream task robustness.

1. Principles of Discrete Audio Tokenization

Discrete audio tokenization converts continuous audio signals $x \in \mathbb{R}^T$ into sequences of tokens by passing $x$ through an encoder $f_e$ to obtain latent embeddings $z_t$, followed by quantization $Q(\cdot)$:

$q_t = Q(z_t)$

The quantizer may use k-means clustering (semantic tokens), RVQ, or neural compression (acoustic tokens). Each token $q_t$ is an index into a codebook of $K$ entries. In RVQ, a recursive process operates over $M$ codebooks for multi-layer granularity:

$$\begin{aligned} &r_t^{(1)} = z_t \ &q_t^{(m)} = \arg\min_k \| r_t^{(m)} - c_k^{(m)} \|^2 \ &\hat{z}_t^{(m)} = c_{q_t^{(m)}}^{(m)} \ &r_t^{(m+1)} = r_t^{(m)} - \hat{z}_t^{(m)} \ &\hat{z}_t = \sum_{m=1}^M \hat{z}_t^{(m)} \end{aligned}$$

Discrete tokens thus encapsulate perceptually relevant or semantically significant aspects of the audio while reducing bit-rate.

2. Taxonomy and Methods

A comprehensive taxonomy, as proposed in "Discrete Audio Tokens: More Than a Survey!" (Mousavi et al., 12 Jun 2025), organizes tokenization approaches by encoder–decoder architecture, quantization algorithm, training paradigm, streamability, and target domain:

• Acoustic tokenizers: Focus on waveform reconstruction; e.g., EnCodec, DAC, RVQGAN (Shechtman et al., 10 Oct 2024), using RVQ with multiple codebooks for bit-rate control and high fidelity.
• Semantic tokenizers: Derived from SSL models (e.g., HuBERT, WavLM, BEATs), quantized via k-means or VQ to preserve phonetic, linguistic, and acoustic event information.
• Hybrid/unitized approaches: Merge semantic and acoustic information, such as AudioLM (Borsos et al., 2022) and ALMTokenizer (Yang et al., 14 Apr 2025), often via staged or parallel token streams.

Training paradigms fall into separate post-training (extract SSL features and quantize) and joint end-to-end optimization (straight-through estimation, soft-to-hard quantization).

Example Encoder-Decoder Path:

Encoder	Quantization	Decoder
SSL (wav2vec2)	k-means clustering	HiFi-GAN vocoder
ConvNet (DAC)	RVQ	ConvNet decoder
BEATs	RepCodec/VQ	Transformer (AAC)

The design space enables flexible trade-offs between reconstruction, semantic density, sequence length, and bit-rate.

3. Applications and Task-Specific Schemes

Discrete audio tokens are applied in various domains by tailoring the tokenization scheme:

• Speech & Music Generation: Hierarchical tokenization (semantic + acoustic) in AudioLM (Borsos et al., 2022), hybridized with multi-stage autoregressive LMs for controllable generation, long-term consistency, and fidelity.
• Speech Separation/Recognition: Joint separation–translation in TokenSplit (Erdogan et al., 2023), using acoustic tokens (SoundStream), semantic tokens (w2v-BERT), and transcript tokens.
• Captioning & Semantic Tasks: Supervised tokenizers trained on audio tagging loss ("Discrete Audio Representations for Automated Audio Captioning" (Tian et al., 21 May 2025)) outperform unsupervised ones in semantic retention. CLAP-ART (Takeuchi et al., 1 Jun 2025) demonstrates that RVQ-applied semantic-rich AR tokens improve AAC performance over waveform-centric tokens.
• Singing Voice Synthesis: TokSing (Wu et al., 12 Jun 2024) blends tokens from multiple SSL layers and incorporates explicit melody signals (LF0) for fine pitch control:

$$\mathcal{L}_m = \frac{1}{N} \sum_{i=1}^N | m_i - \hat{m}_i |$$

• Speech Enhancement: High-resolution RVQ tokens in DAC-SE1 (Lanzendörfer et al., 2 Oct 2025) are used to flatten multi-codebook tokens and apply large autoregressive Transformers for unified, scalable speech enhancement with strong objective and MUSHRA perceptual metrics.

4. Compression, Bit-rate, and Efficiency Trade-offs

Discrete representation schemes enable substantial reductions in storage and transmission needs compared to mel-spectrograms or continuous features. For instance, RVQ-based EnCodec tokens allow up to 20× compression with performance gaps within 1% of legacy features in speaker and speech tasks (Puvvada et al., 2023). Bitrate computation is governed by:

$\text{bitrate} = \log_2(V) \times C \times R$

where $V$ is codebook size, $C$ number of codebooks, $R$ code rate per second.

Byte-Pair Encoding (BPE) (Shen et al., 2023) and token query aggregation (ALMTokenizer (Yang et al., 14 Apr 2025)) further decrease sequence length (by factors of 1.6–2.4), improving inference speed (2.8–5×) and modeling efficiency, while leveraging morphological pattern information for better syntactic accuracy and diversity in outputs.

5. Semantic and Acoustic Preservation, Consistency, and Evaluation

Semantic tokenizers generally outperform compression-based acoustic tokens across most discriminative and generative tasks, except for applications that require fine-grained speaker attributes (where acoustic tokens—e.g., DAC, EnCodec—excel) (Mousavi et al., 20 Jun 2024). However, both approaches still lag behind the best continuous representations, highlighting information loss during quantization.

Discrete Representation Inconsistency (DRI) (Liu et al., 28 Sep 2024) is unique to audio: codec token sequences for perceptually identical segments can diverge, leading to prediction errors or instability in generative models. Mitigation via slice-consistency and perturbation-consistency losses improves downstream metrics, e.g., WER and speaker similarity.

Unified benchmarks (DASB (Mousavi et al., 20 Jun 2024), DATES (Mousavi et al., 12 Jun 2025)) reveal further challenges:

• High bitrate improves signal-level metrics but may degrade downstream utility due to longer sequences.
• Optimal model design varies by task, domain, and evaluation metric (e.g., ASR vs. enhancement).
• Context-aware and multimodal integration is enhanced by query-based or attention-based token selection (ALMTokenizer (Yang et al., 14 Apr 2025), Hybrid architectures (Verma, 16 Dec 2024, KimiTeam et al., 25 Apr 2025)).

6. Integration with LLMs and Multimodality

Discrete audio tokens enable seamless fusion of audio and text modalities in LLMs. AudioLM (Borsos et al., 2022), Whisper-GPT (Verma, 16 Dec 2024), and Kimi-Audio (KimiTeam et al., 25 Apr 2025) illustrate hybrid architectures that concatenate discrete and continuous features for efficient context management and better perplexity/NLL in autoregressive modeling. Ultra-low bitrate approaches (e.g., 0.23 kbps (Mehta et al., 28 Mar 2025)) make scaling practical for joint audio–text reasoning, though high-fidelity generation remains technically challenging at such rates.

Systems like Kimi-Audio (KimiTeam et al., 25 Apr 2025) employ chunk-wise streaming detokenization and low-resolution semantic tokens (e.g., 12.5 Hz), narrowing token-text sequence length gaps and enabling real-time, piecewise generation with mitigated boundary artifacts.

7. Open Challenges and Future Research Directions

Key open issues include joint optimization for reconstruction and semantic preservation ("Discrete Audio Tokens: More Than a Survey!" (Mousavi et al., 12 Jun 2025)), improved quantization techniques that balance fidelity and modelability, and standardization of cross-domain benchmarks. Addressing DRI and further enhancing multimodal robustness are ongoing research imperatives. Future work will likely focus on:

• Scalable, cross-domain tokenizers for speech, music, and general audio.
• Holistic evaluation protocols disentangling representation quality from decoder capabilities.
• Advanced joint training objectives, adaptive domain and sampling rate handling.
• Addressing security, trust, and deepfake risks inherent in high-fidelity discrete token synthesis.

Discrete audio tokens thus represent a pivotal development in audio modeling, enabling modular, scalable, and multimodal systems but require further research to bridge the fidelity gap with continuous features and ensure robust coverage across all target applications and domains.