Vector-Quantized Spectral Tokenizer

• The vector-quantized spectral tokenizer is a learned discrete tokenization scheme that converts spectral decompositions into integer codes for effective autoregressive modeling.
• It uses multiscale decompositions, hierarchical codebooks, and coarse-to-fine conditioning to enhance compression, reconstruction quality, and generative stability.
• Its design integrates transformer architectures with scale or band-causal masking, enabling robust performance in image editing, super-resolution, and audio compression tasks.

A vector-quantized spectral tokenizer is a family of learned discrete tokenization schemes that produce sequences of integer-valued codes from spectral decompositions of input signals (images or audio). These discrete tokens, typically derived via vector quantization of spectral-domain or frequency-band features, serve as atomic units for downstream modeling, including autoregressive generative transformers for images or audio. The vector-quantized spectral tokenizer framework delivers domain-adapted compression, facilitates coarse-to-fine conditioning, and supports efficient and scalable modeling through multiscale representations. Notable instantiations include the Spectral Image Tokenizer (SIT) for images (Esteves et al., 2024) and the Residual Vector Quantization (RVQ) audio tokenizer (EnCodec) for waveforms (Puvvada et al., 2023).

1. Spectral Decomposition and Multiscale Representations

For images, vector-quantized spectral tokenizers utilize multi-level 2D discrete wavelet transforms (DWTs), such as the Haar wavelet basis, to decompose an input image $f[x, y]$ into a hierarchy of subbands at varying spatial resolutions: $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$ where $g = [1, 1]^\top/\sqrt{2}$, $h = [1, -1]^\top/\sqrt{2}$, and $\star$ denotes convolution, $\downarrow 2$ is subsampling by two. After $L$ levels, the full collection $$\{f_{\text{H}_k}, f_{\text{V}_k}, f_{\text{D}_k}\}_{k=1}^L$$ and $f_{\text{low}_L}$ retain all information.

For audio, the vector-quantized spectral approach is exemplified by applying RVQ to embeddings derived from raw waveform frames, implicitly shaping quantization behavior to spectral characteristics through hierarchical residual quantization but without explicit analytic transform per band (Puvvada et al., 2023).

2. Vector Quantization Architectures

In SIT, the multi-scale spectral coefficients are patchified and projected into latent vectors per scale. Each scale $s$ (corresponding to a set of DWT-derived feature maps at resolution $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$0) possesses a separate learnable codebook $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$1 of $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$2 codes with embedding dimension $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$3. The vector quantization process encodes each patch embedding $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$4 as: $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$5 The code index $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$6 maps patches to the nearest codebook entry, producing a discrete token sequence ordered from coarsest to finest scale. Entropy regularization encourages sharp and balanced codebook utilization (see Section 3 for loss terms) (Esteves et al., 2024).

For audio, RVQ operates sequentially using $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$7 codebooks, quantizing each frame embedding $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$8 as a sum of vectors from each stage: $$\begin{align*} f_{\text{low}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,g^\top)\right) \downarrow 2 \ f_{\text{H}_k} &= \left(f_{\text{low}_{k-1}} \star (g\,h^\top)\right) \downarrow 2 \ f_{\text{V}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,g^\top)\right) \downarrow 2 \ f_{\text{D}_k} &= \left(f_{\text{low}_{k-1}} \star (h\,h^\top)\right) \downarrow 2 \end{align*}$$9 Each $g = [1, 1]^\top/\sqrt{2}$0 is obtained by iterative residual quantization (see encoding pseudocode in (Puvvada et al., 2023)). Parameters such as $g = [1, 1]^\top/\sqrt{2}$1 (number of codebooks), $g = [1, 1]^\top/\sqrt{2}$2 (size), and vector dimension $g = [1, 1]^\top/\sqrt{2}$3 govern fidelity, compression, and computational cost.

3. Token Vocabulary and Sequence Construction

Vector-quantized spectral tokenizers are characterized by large, structured vocabularies, with the total number of unique possible tokens given by product over scales (image) or quantization stages (audio). The arrangement of tokens follows the spectral or scale order rather than spatial or temporal positions.

• Image domain (SIT): Vocabulary size is $g = [1, 1]^\top/\sqrt{2}$4, where $g = [1, 1]^\top/\sqrt{2}$5 is the number of scales. Each token at scale $g = [1, 1]^\top/\sqrt{2}$6 is unique due to scale-specific codebooks. For instance, the SIT "Base" employs $g = [1, 1]^\top/\sqrt{2}$7 codes of dimension $g = [1, 1]^\top/\sqrt{2}$8 per scale. Each token encodes a multichannel wavelet patch (3 RGB channels at the coarsest, $g = [1, 1]^\top/\sqrt{2}$9 at finer scales, H/V/D × RGB) (Esteves et al., 2024).
• Audio domain (RVQ): Sequence length is number of frames $h = [1, -1]^\top/\sqrt{2}$0 $h = [1, -1]^\top/\sqrt{2}$1 stages. Each token index is tied to its stage, with $h = [1, -1]^\top/\sqrt{2}$2 and $h = [1, -1]^\top/\sqrt{2}$3 typical.

This symbolic representation of spectral or multi-band content enables efficient modeling and multiscale manipulation in downstream transformer architectures.

4. Autoregressive Modeling and Coarse-to-Fine Conditioning

Spectral token sequences are leveraged in autoregressive (AR) generative models, such as AR-SIT for images. The probability of a token at position $h = [1, -1]^\top/\sqrt{2}$4 (scale $h = [1, -1]^\top/\sqrt{2}$5, patch $h = [1, -1]^\top/\sqrt{2}$6) is modeled, conditioning on all coarser scales and prior positions within a scale: $h = [1, -1]^\top/\sqrt{2}$7 This framework supports coarse-to-fine prediction, where each generated token refines a globally coherent, low-resolution reconstruction toward higher detail. Notably, this improves sample coherence compared to scanline or spatial orderings because generation conditions always contain a full (albeit low-detail) image approximation (Esteves et al., 2024). In audio, autoregressive or masked modeling is possible on token sequences produced by RVQ, although most applications to date have focused on using tokenized representations for compression and recognition tasks rather than generation.

Early termination at a coarser scale allows partial decoding for rapid image previews at arbitrary resolutions (e.g., $h = [1, -1]^\top/\sqrt{2}$8). In SIT, this is realized by passing the available token subsequence through a masked transformer decoder, projecting to wavelet patches, and inverting the DWT up to the specified scale.

5. Transformer Architectures, Scale Causality, and Masking

The vector-quantized spectral tokenizer paradigm is typically coupled with transformer encoders and decoders tuned for scale or band-specific statistics.

• SIT: A transformer encoder $h = [1, -1]^\top/\sqrt{2}$9 processes the full token sequence, quantization maps features to token indices, and a transformer decoder $\star$0 reconstructs feature sequences prior to projection back to wavelet coefficient patches. Reconstruction to the spatial domain is accomplished using the inverse DWT. The architecture optionally supports "scale-causal" masking, restricting token attention within/across scales:
  • Both encoder and decoder masked: Supports arbitrary-resolution multiscale reconstructions.
  • Masked decoder only: Partial/fine generation.
  • Masked encoder only: Text-guided super-resolution and editing.

Key transformer components (key/query/value weights, MLPs, LayerNorms) are maintained as scale-specific to capture the distinct distributions at different frequency bands (Esteves et al., 2024).

• RVQ/EnCodec: Utilizes convolutional encoders and decoders for waveform segments; no transformer is required in the tokenization stage, but downstream modeling (e.g., for language modeling on tokens) can integrate transformer architectures.

6. Empirical Characteristics, Evaluation, and Trade-offs

Image Domain (SIT)

• Multiscale reconstruction: SIT scale-causal (SIT-SC) trained at $\star$1 achieves reconstructions at $\star$2 without retraining. Compared to a ViT-VQGAN baseline, SIT-SC matches or outperforms in LPIPS, PSNR, and L1 metrics with much higher throughput (e.g., at $\star$3, SIT-SC LPIPS 0.122 vs. 0.117, PSNR 30.22 vs. 29.97, throughput 215 vs. 159 images/s) (Esteves et al., 2024).
• High-resolution stability: SIT converges at $\star$4 where ViT-VQGAN diverges; SIT-5 achieves LPIPS 0.248, PSNR 23.6, FID 2.67 (baseline: LPIPS 0.3196, PSNR 22.44, FID 6.92).
• Text-guided upsampling and editing: Scale-causal masking permits flexible super-resolution (FID 6.2 for upsampling vs. 13.7 unconditional) and partial regeneration of finer scales to edit image semantics while preserving coarse structure.
• Sampling efficiency: Coarse-to-fine ordering enables early stopping and rapid image previews with proportional memory and speedup gains.

Audio Domain (RVQ Tokenizer)

• Compression performance: At $\star$5 codebooks (6 kbps), EnCodec maintains $\star$620× compression relative to mel-spectrogram features (128 kbps) with only minor task performance loss (Puvvada et al., 2023).
• Task robustness: Speaker verification, diarization, and ASR models trained on EnCodec tokens are within 1% absolute of mel-spectrogram baselines for in-domain data. Notably, EnCodec tokens are more robust to narrowband (≤8 kHz) test sets, reflecting a low-pass “spectral bottleneck” induced by quantization.
• Spectral transfer function: Empirically, $\star$7 for $\star$8 kHz, rolling off by 5–10 dB at 12–15 kHz—a low-pass characteristic that underlies both compression and robustness to bandwidth variability.

Design Trade-offs

• Number of codebooks ($\star$9), codebook size ($\downarrow 2$0), and vector dimension ($\downarrow 2$1) trade off bit-rate, model size, and fidelity.
• For some modalities or tasks, task-critical features concentrate in certain spectral bands; bottlenecking or perceptual weighting in quantization can be adapted for improved utility in such cases (Puvvada et al., 2023).

7. Design Considerations and Future Directions

Recommendations from empirical findings for advancing vector-quantized spectral tokenizers include:

• Band-aware quantization: Incorporate multi-band or perceptually weighted residual quantization, increasing codebook allocation for high-frequency or task-relevant spectral regions (Puvvada et al., 2023).
• Hybrid feature quantization: Apply VQ or RVQ directly to time–frequency representations (e.g., mel-frequency bands or DWT coefficients), drawing on SIT’s spectral patch paradigm, rather than raw waveform spaces.
• Dynamic resource allocation: Vary codebook utilization conditioned on input signal complexity, optimizing coding efficiency.
• Semantic layering: Stack clustering or self-supervised modeling (e.g., HuBERT/WavLM) atop compression-derived tokens for richer, task-adaptive symbolic representations.
• Transformer model scaling and masking strategies: Continue to exploit scale/band-causal attention and scale-specific parameterization in transformer blocks to maximize performance for multiscale data.
• Partial decoding and progressive refinement: Leverage coarse-to-fine token ordering for efficient preview, rapid generation, and fine-level editing by halting or redirecting generation at arbitrary scales.

A plausible implication is that these approaches, by structuring the discrete token domain to reflect spectral composition, enable more efficient, controllable, and scalable modeling for both generative and recognition-centric tasks. Future vector-quantized spectral tokenizers will likely integrate explicit spectral shaping both in the quantization scheme and in the downstream autoregressive or masked modeling frameworks (Esteves et al., 2024, Puvvada et al., 2023).