Compressed-Spectrum Codecs Overview

• Compressed-spectrum codecs are algorithms that compress and manipulate signals by converting them into time–frequency representations using methods like STFT and Mel-spectrograms.
• They employ advanced quantization techniques such as RVQ, FSQ, and single-codebook VQ to efficiently encode spectral data with structured tokenization.
• These codecs power applications in audio coding, speech synthesis, and streaming, delivering superior rate–distortion performance and low-latency processing.

Compressed-spectrum codecs are a class of audio and signal processing algorithms that operate by compressing and manipulating representations of signals in the frequency domain, instead of (or in addition to) directly processing their time-domain waveforms. These methods exploit the spectral sparsity, structured redundancy, and perceptual relevance of time-frequency or subband representations—using transforms such as the Short-Time Fourier Transform (STFT), Mel-scale analysis, or more specialized filter banks—to achieve rate-efficient, high-fidelity, and often highly structured compression. Compressed-spectrum codecs now form a central strand of both neural and classical approaches to audio compression, spectrum sensing, and enabling compact tokenization for downstream generative models.

1. Principles of Time–Frequency Domain Compression

Compressed-spectrum codecs are unified by the initial conversion of the input signal into a time–frequency, subband, or spectral representation. The most prevalent forms include:

• Short-Time Fourier Transform (STFT): Represents the signal as $X(m, k)$, with $m$ denoting time-frame and $k$ frequency-bin. Log-magnitude and unwrapped or derivative phase features are often processed in parallel to capture both envelope and fine temporal structure (Feng et al., 21 Mar 2025).
• Mel-spectrograms: Apply a filterbank $H_{f,k}$ to modulus STFT $|S(n,k)|$, yielding perceptually motivated frequency bands $M(n,f)$ (Zhang et al., 6 Jan 2026, Langman et al., 2024).
• Pseudo-Quadrature Mirror Filter (PQMF)/Subband Analysis: Decomposes the full-band signal into $M$ uniform or nonuniform subbands via designed prototype filters $h_k[n]$, enabling the assignment of discrete codebooks or supervision per band (Zhang et al., 21 Sep 2025).
• Sparse and Overcomplete Transforms: Redundant trigonometric dictionaries enable highly sparse representations of music or complex sounds via atom selection and greedy pursuit (Rebollo-Neira, 2015).

These transforms front-load much of the dimensionality reduction, compactness, and invertibility, laying the groundwork for both data-driven and analytical compression strategies.

2. Quantization Frameworks and Discrete Tokenization

Spectral compression requires quantizing the latent, typically continuous, coefficients arising from the spectral transform stage. The major frameworks are:

• Residual Vector Quantization (RVQ): Sequential stacks of codebooks, each quantizing the residual remaining after previous stages. The $m$-th codebook at each step produces $c_m = Q_m(r_{m-1})$ with recursive residuals (Zhang et al., 21 Sep 2025, Feng et al., 21 Mar 2025, Wu et al., 4 Feb 2025). RVQ is pervasive in neural codecs, being highly expressive but yielding complex, often hierarchical token distributions.
• Finite Scalar Quantization (FSQ): Each spectral coefficient (or group) is quantized independently into $m$0 uniformly spaced levels, yielding parallel, flat codebooks that are easier for downstream models to predict (Langman et al., 2024). FSQ’s per-dimension discretization contrasts with the hierarchical structure of RVQ and offers advantages for parallel speech synthesis.
• Single-codebook VQ: Operating on downsampled 2D spectral latents, a large learnable codebook captures entire low-rate, high-dimensional “spectral frames” (e.g., SimVQ in UniSRCodec) (Zhang et al., 6 Jan 2026).
• Sparse Approximation and Pursuit: In overcomplete dictionaries, codec operation corresponds to selecting $m$1 atoms (basis functions) and quantizing/scaling their active coefficients. Adaptive entropy coding further compresses the index and coefficient bitstreams (Rebollo-Neira, 2015).

Bit-allocation strategies are strongly coupled to the transform domain, with separate control over time–frequency resolution, token rate, and discriminative power for low versus high frequencies.

3. Neural and Hybrid Architectures

Neural compressed-spectrum codecs are structured as encoder–quantizer–decoder pipelines, adapted to the frequency-domain input. Typical architectures include:

• Parallel Amplitude and Phase Branches: Separate ConvNeXt-like modules process log-amplitude and phase-derivative or wrapped-phase streams, later fused at the latent bottleneck (Ai et al., 2024, Feng et al., 21 Mar 2025).
• Multi-branch or Multi-codebook Designs: Networks assign codebooks and loss supervision to distinct subbands or semantic dimensions (e.g., HuBERT-derived tokens), yielding functionally disentangled representations (Zhang et al., 21 Sep 2025).
• Streaming-friendly and Lightweight Models: Causal CNN+RNN hybrids, 2D convolutions, and GRU-based layers achieve online inference with blockwise or framewise normalization in the spectral domain (Wan et al., 24 Oct 2025).
• Vocoder Decoding and Phase Recovery: When phase is omitted or oversimplified (as with Mel-spectrogram inputs), an external trained vocoder (e.g. BigVGAN-v2) reconstructs the waveform from the compressed spectral magnitude (Zhang et al., 6 Jan 2026).

Table: Selected compressed-spectrum codecs and their design attributes

Codec	Spectral Transform	Quantization	Architecture
STFTCodec	STFT (log-mag + phase)	RVQ	Parallel ConvNeXt
UniSRCodec	Mel-spectrogram	Single-codebook VQ	2D CNN + vocoder
APCodec	STFT (ampl. + phase)	Multi-RVQ	ConvNeXt v2 dual-path
SpecTokenizer	Complex STFT	RVQ stack/EMA	CNN+RNN2D streaming
Spectral Codec	Mel-spectrogram	FSQ	Parallel softmax (TTS)

4. Rate–Distortion Performance and Applications

Compressed-spectrum codecs have demonstrated strong empirical performance across a wide range of tasks and domains:

• Audio Coding: STFTCodec achieves superior PESQ (e.g., 4.28 at 12 kbps, 3.38 at 3 kbps), higher STOI, and lower log-spectral distance than waveform-based codecs at the same bitrate (Feng et al., 21 Mar 2025).
• Speech Synthesis and TTS: Spectral codecs provide tokens with less complex distributions, enabling parallel, non-autoregressive TTS models to achieve higher MOS and intelligibility than those trained on waveform-token RVQ codes (Langman et al., 2024).
• Speech, Music, and Universal Audio: High-fidelity recovery across domains is achieved by architectures such as MBCodec and SwitchCodec, with disentangled semantic/acoustic codebooks and combinatorially-expanded embedding spaces permitting near-lossless compression even at $m$22.2 kbps and below (Zhang et al., 21 Sep 2025, Wang et al., 30 May 2025).
• Low-Latency and Streaming: SpecTokenizer yields strong SDR (8.36 dB) and PESQ (3.04) at 4 kbps with less than 20 ms latency and sub-M parameter count in the “mini” variant (Wan et al., 24 Oct 2025).

A plausible implication is that compressed-spectrum codecs can operate at lower bitrates than waveform-based systems for matched or superior perceptual quality, especially when making full use of psychoacoustic frequency masking and time–frequency nonuniformity.

5. Training Objectives, Losses, and Phase Strategies

Optimization of compressed-spectrum codecs typically involves hybrid objective functions to ensure fidelity, perceptual propriety, and codebook utilization:

• Reconstruction Losses: Multi-scale L1/L2 on mel-spectrograms, complex-domain time–frequency errors, and anti-phase-wrapping measures ensure invertibility and stability (Ai et al., 2024, Feng et al., 21 Mar 2025).
• Adversarial and Feature Matching: GAN losses on both time and spectral domains (multi-period, multi-STFT discriminators) are combined with feature matching in intermediate discriminator layers (Feng et al., 21 Mar 2025, Wang et al., 30 May 2025, Wu et al., 4 Feb 2025).
• Codebook and Commitment Losses: Quantization stability and codebook diversity are enforced with commitment penalties, often using the stop-gradient operator (Zhang et al., 6 Jan 2026, Feng et al., 21 Mar 2025).
• Knowledge Distillation: For streaming scenarios, teacher–student dist