ALMTokenizer: Semantic Audio Codec

• The paper presents ALMTokenizer, a novel audio codec tokenizer that compresses waveforms into semantically rich token sequences using query-based compression and residual vector quantization.
• It employs a two-stage training regime combining masked autoencoding and autoregressive objectives to optimize reconstruction fidelity and enhance semantic representation.
• Experimental results demonstrate improved PESQ, SNR, and MUSHRA scores compared to standard codecs, benefiting downstream tasks like audio captioning and language modeling.

ALMTokenizer is a low-bitrate, semantically enriched audio codec tokenizer specifically designed for audio language modeling applications. By integrating a novel query-based compression architecture with residual vector quantization featuring semantic priors and a two-stage training regime incorporating masked autoencoding and autoregressive objectives, ALMTokenizer enables efficient conversion of audio waveforms into compact token sequences that retain contextual and semantic integrity. This approach supports high-fidelity reconstruction and delivers improved semantic representations as required by modern audio LLMs (Yang et al., 14 Apr 2025).

1. Architecture and Encoding Pipeline

ALMTokenizer employs a multi-stage pipeline to transform input audio waveforms into discrete tokens suitable for downstream transformer-based LLMs. The process consists of:

• Patchify Encoder: The input waveform $x$ is framed into $T$ overlapping patches using a convolutional encoder, resulting in frame embeddings $e\in\mathbb{R}^{T\times d}$.
• Query-Based Compression: Instead of uniform down-sampling or isolated frame encoding, $K\ll T$ learnable query tokens $Q\in\mathbb{R}^{K\times d}$ are prepended to the embedding sequence. A transformer encoder processes $[Q; e]$ and outputs only the $K$ query token embeddings. Each query token attends over all $T$ frames, aggregating cross-frame context analogous to multiple [CLS] tokens in text models.
• Bitrate Control: The compression factor is directly managed by the choice of $K$, enabling systematic control over token sequence rates and final bitrate.
• Residual Vector Quantization (RVQ) with Semantic Priors: The $K$ query embeddings pass through an $T$0-layer RVQ stack. The first codebook is fixed offline using $T$1-means clustering on self-supervised audio representations (e.g., Wav2Vec2, BEATs), enforcing semantic priors; subsequent codebooks are trainable end-to-end. Only the first-stage codebook weights remain frozen.
• Unpatchify Decoder: Quantized embeddings are reconstructed to waveforms via a lightweight convolutional decoder.

Component	Technique	Key Feature/Setting
Patchify	Conv. encoder	Frames input into $T$2 patches
Compression	Query-based (K queries)	Holistic, cross-frame context
Quantization	L-layer RVQ	1st codebook: fixed semantic prior
Reconstruction	Conv. decoder (unpatchify)	Lightweight, post-quantization

2. Training Objectives and Mathematical Formulations

ALMTokenizer is trained with a two-stage regime optimizing for reconstruction fidelity and semantic-rich representation:

1. Masked Autoencoder (MAE) Loss: With mask rate $T$3, random parts of the input are masked, and the model is trained to reconstruct masked frames:

$T$4

2. Vector Quantization Loss with Semantic Prior: For each layer $T$5 and query embedding $T$6, with commitment weight $T$7 (see Table 6 in (Yang et al., 14 Apr 2025)):

$T$8

The first codebook $T$9 is fixed offline.

3. Autoregressive (AR) Prediction Loss: An AR transformer predicts the quantized embedding of each RVQ layer given all previous layers, optimized via MSE:

$e\in\mathbb{R}^{T\times d}$0

The final stage-II loss is:

$e\in\mathbb{R}^{T\times d}$1

where typical hyperparameters are $e\in\mathbb{R}^{T\times d}$2; the training schedule initiates with $e\in\mathbb{R}^{T\times d}$3, adding $e\in\mathbb{R}^{T\times d}$4 in stage II.

3. Bitrate, Reconstruction Performance, and Baseline Comparison

ALMTokenizer supports direct and flexible bitrate control via $e\in\mathbb{R}^{T\times d}$5 and $e\in\mathbb{R}^{T\times d}$6. With configurations matching standard neural audio codecs at 1.5 kbps (e.g., $e\in\mathbb{R}^{T\times d}$7 Hz frame rate, $e\in\mathbb{R}^{T\times d}$8), objective and subjective quality evaluations on VCTK and LibriTTS reveal the following:

Metric	ALMTokenizer	Encodec (1.5 kbps)	SoundStream/MimiCodec
PESQ	$e\in\mathbb{R}^{T\times d}$9	$K\ll T$0	$K\ll T$1–$K\ll T$2
SNR (dB)	$K\ll T$3	$K\ll T$4	—
MUSHRA (median)	$K\ll T$5	$K\ll T$6	$K\ll T$7–$K\ll T$8

These outcomes (see Table 1, Figure 1, and Appendix Table 10 in (Yang et al., 14 Apr 2025)) indicate improved fidelity over prior methods, both on acoustically objective and human subjective metrics.

4. Downstream Audio Language Modeling Effectiveness

By producing token sequences approximately $K\ll T$9 times shorter than frame-level codecs (e.g., $Q\in\mathbb{R}^{K\times d}$0 Hz vs. $Q\in\mathbb{R}^{K\times d}$1 Hz), ALMTokenizer directly benefits transformer-based audio LMs:

• BLEU/CIDEr Gains in Audio Captioning: Performance surpasses Encodec tokens by $Q\in\mathbb{R}^{K\times d}$2 BLEU / $Q\in\mathbb{R}^{K\times d}$3 CIDEr.
• Modeling Efficiency: Yields lower perplexity and approximately $Q\in\mathbb{R}^{K\times d}$4 fewer training steps for convergence, attributed to increased semantic richness of the tokens.
• Integrative Compatibility: Discrete IDs from RVQ stages are adopted as tokens by downstream standard transformer LMs, supporting composition via AR prediction loss.

Downstream evaluation details are presented in Appendix Table 11 of (Yang et al., 14 Apr 2025). These results demonstrate consistent advantages for tasks such as audio captioning, keyword spotting, and text-conditioned audio synthesis.

5. Deployment Considerations and Practical Integration

ALMTokenizer introduces straightforward mechanisms for bitrate governance and codec modification:

• Bitrate Adjustment: The number of queries $Q\in\mathbb{R}^{K\times d}$5 can be tuned at inference to match bitrate requirements, without re-training the convolutional encoder.
• Codec Plug-in: The MAE and VQ components can augment existing neural codecs (e.g., Encodec, MimiCodec), supporting retrofitting for semantic enhancement; the AR LM head can be optionally integrated for joint modeling.
• Integration Pipeline:

1. Substitute convolutional down-sampling with Patchify plus query-based compression.
2. Employ a frozen (or optionally fine-tuned) semantic prior codebook for the first VQ stage.
3. Train with $Q\in\mathbb{R}^{K\times d}$6.
4. Optionally fine-tune with $Q\in\mathbb{R}^{K\times d}$7 and retrain the LM head.

Applications include ultra-low-bitrate speech streaming (IoT voice sensors, hearing aids), end-to-end audio language agents (voice chatbots), and multimodal systems (joint audio+text LLMs) (Yang et al., 14 Apr 2025).

6. Contextual Significance and Future Directions

ALMTokenizer represents a significant evolution in learned audio tokenization by:

• Leveraging a query-based compression strategy to aggregate and compress holistic, context-aware audio representations, as opposed to frame-local codebooks in earlier codecs.
• Utilizing semantic priors via k-means codebooks fixed from self-supervised audio models, aligning audio tokens with meaningful structure in the pre-trained feature space.
• Empirically demonstrating consistent improvements in both reconstruction quality and downstream audio language modeling tasks relative to Encodec, MimiCodec, SoundStream, and DAC baselines.

A plausible implication is broader adoption of query-based token compression mechanisms and semantically anchored codebooks for both audio and other continuous modality tokenization. Further exploration of the trade-offs between token length, semantic richness, and modeling efficiency is anticipated.