HH-Codec: Neural Audio Codec for Speech

• HH-Codec is a discrete neural codec for spoken language that leverages vector quantization with a highly regularized codebook for ultra-high compression.
• It employs a two-layer residual VQ during training and an asymmetric encoder–decoder architecture with attention-enhanced ConvNeXt blocks for robust speech reconstruction.
• The codec achieves a bitrate of approximately 0.3 kbps while preserving semantic integrity and perceptual quality, enabling scalable spoken language modeling.

HH-Codec is a discrete neural codec for spoken language modeling that achieves ultra-high compression and high-fidelity audio reconstruction using a single-quantizer, vector-quantization (VQ) approach and a highly regularized codebook. This design addresses the complexity and computational challenges inherent to large-scale speech-to-speech systems by providing efficient tokenization at scale while maintaining semantic integrity and perceptual quality of speech (Xue et al., 25 Jul 2025).

1. Vector Quantization and Codebook Design

HH-Codec employs a single-quantizer VQ at inference, augmented with a two-layer residual VQ (RVQ) during training. The codebook comprises $K=8192$ entries, each of $D=512$ dimensions, totaling $4\,194\,304$ codebook parameters. During training, VQ₁ and VQ₂ are both present in $\mathbb{R}^D$. Only the output of VQ₁ is used to drive the decoder at inference time, while VQ₂ acts solely as a “virtual class” for regularization. Quantization of encoder outputs $e\in\mathbb{R}^{T \times D}$ is performed per time step via nearest neighbor search among codewords, with gradients passed using the "rotational trick" (Xue et al., 25 Jul 2025).

The VQ losses combine reconstruction and commitment:

$$L_{\mathrm{VQ}} = \|\mathrm{sg}[e] - \hat{e}\|_2^2 + \beta\|\mathrm{sg}[\hat{e}] - e\|_2^2$$

where $\mathrm{sg}[\cdot]$ is the stop-gradient operator and $\beta=1$.

Semantic collapse at high compression is mitigated by including a distillation loss from a frozen HuBERT teacher, penalizing misalignment in encoded representations:

$$L_{\mathrm{distill}} = -\frac{1}{D}\sum_{d=1}^D \log \sigma\bigl(\cos\big(VQ_1(e)_{:,d},H_{:,d}\big)\bigr)$$

where $\sigma$ is the sigmoid and $D=512$0 cosine similarity across the batch.

2. Asymmetric Encoder–Decoder Architecture

The encoder processes waveform input $D=512$1 (for 1s at 24 kHz) via a Conv1D layer (kernel size 7), followed by four convolutional blocks—each block containing a dilated Conv1D and a strided Conv1D for downsampling. The aggregate downsampling factor is $D=512$2, yielding $D=512$3 tokens per second per 24 kHz input. A BiLSTM is included to improve semantic modeling.

The quantized encoder output, $D=512$4, is projected by an MLP for the decoder path. The decoder consists of attention-enhanced ConvNeXt blocks, which perform upsampling and output Mel-spectrograms $D=512$5. Final waveform synthesis to $D=512$6 is performed by a Big VGAN vocoder, which is initially frozen and subsequently fine-tuned (Xue et al., 25 Jul 2025).

Architecture Data flow (tensor shapes):

Stage	Input/Output shapes	Notes
Waveform → Encoder	(24,000) → (T, 512)	$D=512$7 for 1 s of 24 kHz
Encoder → VQ₁	(T, 512) → (T, 512)	Applies VQ and MLP projection
VQ₁ → Decoder → Melrec	(T, 512) → (Tₘₑₗ, 80)	Mel-spectrogram frame upsampling
Melrec → Vocoder → $D=512$8	(Tₘₑₗ, 80) → (24,000)	Big VGAN vocoder for waveform synthesis

3. Dual Supervision and Progressive Training

Training employs three coordinated objective groups:

A. Mel-domain reconstruction:

$D=512$9

B. Adversarial and feature-matching losses use the Big VGAN discriminator. Generator loss $4\,194\,304$0, feature-matching $4\,194\,304$1, and discriminator loss $4\,194\,304$2 are defined as: $4\,194\,304$3

$4\,194\,304$4

$4\,194\,304$5

C. VQ and distillation losses: $4\,194\,304$6 and $4\,194\,304$7 as above.

The full objective (in stage 2) is a weighted combination:

$4\,194\,304$8

A progressive training schedule provides stability:

• Stage 1 (≈20 epochs): optimize $4\,194\,304$9 only (GAN/vocoder frozen) until convergence.
• Stage 2: enable all objectives, unfreeze vocoder for joint fine-tuning.

By staggering adversarial supervision, the model avoids early gradient instabilities and achieves robust convergence (Xue et al., 25 Jul 2025).

4. Compression, Bitrate, and Practical Trade-offs

At a downsampling of $\mathbb{R}^D$0 tokens/s and $\mathbb{R}^D$1 codewords, the encoded bit-rate is: $\mathbb{R}^D$2 where each token uses $\mathbb{R}^D$3 bits ($\mathbb{R}^D$4). This compression of $\mathbb{R}^D$5 kbps represents an order of magnitude reduction compared to contemporary discrete audio codecs (Xue et al., 25 Jul 2025).

Despite the compression, HH-Codec maintains high UTMOS, STOI, and speaker similarity (SIM) scores, outperforming or matching models operating at 10x the bit-rate. The model also streamlines inference by requiring only VQ₁ output, with all runtime decoding driven by a single quantizer.

5. Empirical Evaluation and Ablations

Evaluation is performed on LibriTTS test-other, test-clean, and Seed-TTS-eval protocols with UTMOS, STOI, voiced/unvoiced (V/UV) F1, and speaker similarity (SIM) metrics.

Model	Bitrate (kbps)	UTMOS (test-other/clean/eval)	STOI	V/UV F1	SIM
HH-Codec (1 Q)	0.3	3.21 / 3.61 / 3.33	0.86 / 0.89 / 0.85	0.86 / 0.90 / 0.88	0.71 / 0.73 / 0.73
DAC (9 Q)	9	3.36 / 4.03 / 3.46	0.95 / 0.97 / 0.96	0.97 / 0.97 / 0.99	0.92 / 0.94 / 0.91
SpeechTokenizer	3	3.28 / 3.87 / 3.34	0.87 / 0.91 / 0.88	0.92 / 0.94 / 0.91	0.79 / 0.82 / 0.72

Ablation studies demonstrate:

• Two-layer training-time VQ (SLM-VQ) substantially improves code utilization and final quality versus classic VQ or single-layer SLM-VQ (e.g., UTMOS decreases by up to $\mathbb{R}^D$6, SIM decreases by up to $\mathbb{R}^D$7).
• Dual supervision and progressive training are critical for stability; omitting these drops UTMOS by $\mathbb{R}^D$8 to $\mathbb{R}^D$9, and SIM decreases by $e\in\mathbb{R}^{T \times D}$0 to $e\in\mathbb{R}^{T \times D}$1.
• Removing attention-enhanced ConvNeXt blocks or substituting the vocoder causes further performance drops.

The codebook utilization under SLM-VQ is $e\in\mathbb{R}^{T \times D}$2 for $e\in\mathbb{R}^{T \times D}$3 and $e\in\mathbb{R}^{T \times D}$4 for $e\in\mathbb{R}^{T \times D}$5, compared to classic VQ’s $e\in\mathbb{R}^{T \times D}$6 and $e\in\mathbb{R}^{T \times D}$7 at the same sizes, indicating superior dispersion and memory usage in the learned quantizer space.

6. Implications and Use in Spoken Language Modeling

HH-Codec’s ultra-low bandwidth, high-fidelity discrete coding enables scalable spoken language modeling. Notably, it provides:

• Efficient token space for speech-to-speech and speech-to-text pipelines requiring tractable computation at scale.
• High codebook utilization and improved semantic preservation compared to previous VQ-based codecs.
• Stable convergence and speech quality even at extreme bitrates due to the dual supervision and progressive training schedule.

A plausible implication is increased tractability for generative spoken LLMs, especially in low-resource or streaming settings, where the complexity of prior multi-quantizer or hierarchical token pipelines previously hindered deployment (Xue et al., 25 Jul 2025).

HH-Codec's open-source release facilitates integration with downstream generative models and LLMs that require compact, semantically meaningful audio tokens. The approach also provides a template for future codecs seeking to balance statistical efficiency with downstream compatibility.