---
title: 'HH-Codec: Neural Audio Codec for Speech'
url: https://www.emergentmind.com/topics/hh-codec
type: topic
---

# HH-Codec: Neural Audio Codec for Speech

HH-Codec is a discrete neural codec for spoken language modeling that achieves ultra-high compression and high-fidelity audio reconstruction using a single-quantizer, vector-quantization (VQ) approach and a highly regularized codebook. This design addresses the complexity and computational challenges inherent to large-scale speech-to-speech systems by providing efficient tokenization at scale while maintaining semantic integrity and perceptual quality of speech [2507.18897].

## 1. Vector Quantization and Codebook Design

HH-Codec employs a single-quantizer VQ at inference, augmented with a two-layer residual VQ (RVQ) during training. The codebook comprises $K=8192$ entries, each of $D=512$ dimensions, totaling $4\,194\,304$ codebook parameters. During training, VQ₁ and VQ₂ are both present in $\mathbb{R}^D$. Only the output of VQ₁ is used to drive the decoder at inference time, while VQ₂ acts solely as a “virtual class” for regularization. Quantization of encoder outputs $e\in\mathbb{R}^{T \times D}$ is performed per time step via nearest neighbor search among codewords, with gradients passed using the "rotational trick" [2507.18897].

The VQ losses combine reconstruction and commitment:
$$
L_{\mathrm{VQ}} = \|\mathrm{sg}[e] - \hat{e}\|_2^2 + \beta\|\mathrm{sg}[\hat{e}] - e\|_2^2
$$
where $\mathrm{sg}[\cdot]$ is the stop-gradient operator and $\beta=1$.

Semantic collapse at high compression is mitigated by including a distillation loss from a frozen HuBERT teacher, penalizing misalignment in encoded representations:
$$
L_{\mathrm{distill}} = -\frac{1}{D}\sum_{d=1}^D \log \sigma\bigl(\cos\big(VQ_1(e)_{:,d},H_{:,d}\big)\bigr)
$$
where $\sigma$ is the sigmoid and $\cos(\cdot,\cdot)$ cosine similarity across the batch.

## 2. Asymmetric Encoder–Decoder Architecture

The encoder processes waveform input $z \in \mathbb{R}^{24,000}$ (for 1s at 24 kHz) via a Conv1D layer (kernel size 7), followed by four convolutional blocks—each block containing a dilated Conv1D and a strided Conv1D for downsampling. The aggregate downsampling factor is $8\times8\times4\times4=1024$, yielding $T\approx24$ tokens per second per 24 kHz input. A BiLSTM is included to improve semantic modeling. 

The quantized encoder output, $\hat{e}^{(1)}$, is projected by an MLP for the decoder path. The decoder consists of attention-enhanced ConvNeXt blocks, which perform upsampling and output Mel-spectrograms $\mathrm{melrec}\in\mathbb{R}^{T_\mathrm{mel}\times 80}$. Final waveform synthesis to $\hat{y} \in \mathbb{R}^{24,000}$ is performed by a Big VGAN vocoder, which is initially frozen and subsequently fine-tuned [2507.18897].

**Architecture Data flow (tensor shapes):**

| Stage                        | Input/Output shapes         | Notes                                    |
|------------------------------|----------------------------|-------------------------------------------|
| Waveform → Encoder           | (24,000) → (T, 512)        | $T\approx24$ for 1 s of 24 kHz           |
| Encoder → VQ₁                | (T, 512) → (T, 512)        | Applies VQ and MLP projection            |
| VQ₁ → Decoder → Melrec       | (T, 512) → (Tₘₑₗ, 80)      | Mel-spectrogram frame upsampling         |
| Melrec → Vocoder → $\hat{y}$ | (Tₘₑₗ, 80) → (24,000)      | Big VGAN vocoder for waveform synthesis  |

## 3. Dual Supervision and Progressive Training

Training employs three coordinated objective groups:

**A. Mel-domain reconstruction:**
$$
L_\mathrm{mel} = \|\mathrm{mel}(z) - \mathrm{melrec}\|_1 + \|\mathrm{mel}(z) - \mathrm{mel}(\hat{y})\|_1
$$

**B. Adversarial and feature-matching losses** use the Big VGAN discriminator. Generator loss $L_g$, feature-matching $L_\mathrm{feat}$, and discriminator loss $L_D$ are defined as:
\[
L_g = \tfrac{1}{K}\sum_{k=1}^K \max(1-D_k(\hat{y}),0)
\]
\[
L_\mathrm{feat} = \tfrac{1}{KL}\sum_{k=1}^K\sum_{\ell=1}^L \| D_k^{(\ell)}(z) - D_k^{(\ell)}(\hat{y})\|_1
\]
\[
L_D = \tfrac{1}{K}\sum_{k=1}^K \left[ \max(1 - D_k(z),0) + \max(1 + D_k(\hat{y}),0) \right]
\]

**C. VQ and distillation losses:** $L_{\mathrm{VQ}}$ and $L_\mathrm{distill}$ as above.

The full objective (in stage 2) is a weighted combination:
$$
L = \lambda_\mathrm{rec}(L_\mathrm{mel} + L_g + L_\mathrm{feat}) + \lambda_\mathrm{vq}L_\mathrm{VQ} + \lambda_\mathrm{dist}L_\mathrm{distill} + \lambda_{D}L_D
$$

A progressive training schedule provides stability:
- **Stage 1 (≈20 epochs):** optimize $L_\mathrm{mel} + L_\mathrm{feat}$ only (GAN/vocoder frozen) until convergence.
- **Stage 2:** enable all objectives, unfreeze vocoder for joint fine-tuning.

By staggering adversarial supervision, the model avoids early gradient instabilities and achieves robust convergence [2507.18897].

## 4. Compression, Bitrate, and Practical Trade-offs

At a downsampling of $T=24$ tokens/s and $K=8192$ codewords, the encoded bit-rate is:
\[
\text{bitrate} = \frac{T \times \log_2 K}{10^{3}} \approx 0.312~\text{kbps}
\]
where each token uses $13$ bits ($\log_2(8192)$). This compression of $0.3$ kbps represents an order of magnitude reduction compared to contemporary discrete audio codecs [2507.18897].

Despite the compression, HH-Codec maintains high UTMOS, STOI, and speaker similarity (SIM) scores, outperforming or matching models operating at 10x the bit-rate. The model also streamlines inference by requiring only VQ₁ output, with all runtime decoding driven by a single quantizer.

## 5. Empirical Evaluation and Ablations

Evaluation is performed on LibriTTS test-other, test-clean, and Seed-TTS-eval protocols with UTMOS, STOI, voiced/unvoiced (V/UV) F1, and speaker similarity (SIM) metrics.

| Model                    | Bitrate (kbps) | UTMOS (test-other/clean/eval) | STOI | V/UV F1 | SIM  |
|--------------------------|---------------|-------------------------------|------|---------|------|
| HH-Codec (1 Q)           | 0.3           | 3.21 / 3.61 / 3.33            | 0.86 / 0.89 / 0.85 | 0.86 / 0.90 / 0.88 | 0.71 / 0.73 / 0.73 |
| DAC (9 Q)                | 9             | 3.36 / 4.03 / 3.46            | 0.95 / 0.97 / 0.96 | 0.97 / 0.97 / 0.99 | 0.92 / 0.94 / 0.91 |
| SpeechTokenizer          | 3             | 3.28 / 3.87 / 3.34            | 0.87 / 0.91 / 0.88 | 0.92 / 0.94 / 0.91 | 0.79 / 0.82 / 0.72 |

Ablation studies demonstrate:
- Two-layer training-time VQ (SLM-VQ) substantially improves code utilization and final quality versus classic VQ or single-layer SLM-VQ (e.g., UTMOS decreases by up to $-0.45$, SIM decreases by up to $-0.12$).
- Dual supervision and progressive training are critical for stability; omitting these drops UTMOS by $-1.19$ to $-1.22$, and SIM decreases by $-0.31$ to $-0.32$.
- Removing attention-enhanced ConvNeXt blocks or substituting the vocoder causes further performance drops.

The codebook utilization under SLM-VQ is $\geq98\%$ for $K\leq8192$ and $94\%$ for $K=16,384$, compared to classic VQ’s $56\%$ and $42\%$ at the same sizes, indicating superior dispersion and memory usage in the learned quantizer space.

## 6. Implications and Use in Spoken Language Modeling

HH-Codec’s ultra-low bandwidth, high-fidelity discrete coding enables scalable spoken language modeling. Notably, it provides:
- Efficient token space for speech-to-speech and speech-to-text pipelines requiring tractable computation at scale.
- High codebook utilization and improved semantic preservation compared to previous VQ-based codecs.
- Stable convergence and speech quality even at extreme bitrates due to the dual supervision and progressive training schedule.

A plausible implication is increased tractability for generative spoken language models, especially in low-resource or streaming settings, where the complexity of prior multi-quantizer or hierarchical token pipelines previously hindered deployment [2507.18897]. 

HH-Codec's open-source release facilitates integration with downstream generative models and large language models (LLMs) that require compact, semantically meaningful audio tokens. The approach also provides a template for future codecs seeking to balance statistical efficiency with downstream compatibility.

Source: https://www.emergentmind.com/topics/hh-codec