---
title: 'DeCodec: Disentangled Neural Audio Codec'
url: https://www.emergentmind.com/topics/decodec
type: topic
---

# DeCodec: Disentangled Neural Audio Codec

DeCodec is a neural audio codec formulated as a universal disentangled representation learner rather than a reconstruction-only compressor. It is designed to decompose mixed audio into orthogonal latent subspaces for speech and background sound, and to further decompose the speech subspace into semantic and paralinguistic components. In this formulation, the codec is intended to serve as a controllable front-end for multiple downstream tasks, including speech enhancement, one-shot voice conversion on noisy speech, ASR, and TTS with controllable background sound preservation or suppression, while maintaining advanced signal reconstruction [2509.09201].

## 1. Conceptual basis and research motivation

DeCodec is motivated by a mismatch between the structure of real-world audio and the structure of most neural codecs. Real-world audio often contains mixed speech and background sounds, whereas universal codecs such as EnCodec, DAC, and UniCodec learn entangled representations across audio types. Conversely, some disentangled codecs exist for speech, but they are limited to speech and are not designed to support selective access to speech and background components in noisy mixtures. DeCodec addresses this by treating the codec itself as a representation-learning system whose latent variables can be functionally selected and recombined across tasks [2509.09201].

The core signal model is additive:
\[
\mathbf{y} = \mathbf{s} + \mathbf{n},
\]
where \(\mathbf{y}\) is the observed mixture, \(\mathbf{s}\) is clean speech, and \(\mathbf{n}\) is background sound. DeCodec assumes \(\mathbf{s}\) and \(\mathbf{n}\) are mutually independent, and seeks a latent decomposition
\[
\mathbf{Y} = \mathbf{S} + \mathbf{N},
\]
with \(\mathbf{S} \in \mathcal{V}_S\), \(\mathbf{N} \in \mathcal{V}_N\), and
\[
\mathcal{V}_Y = \mathcal{V}_S \oplus \mathcal{V}_N.
\]
The intended interpretation is that \(\mathcal{V}_S\) carries speech and \(\mathcal{V}_N\) carries background sound, with orthogonality used to discourage overlap between the two subspaces [2509.09201].

Within the speech subspace, DeCodec imposes a second factorization. The first speech RVQ layer is designated as the semantic code \(\mathbf{Z}_c\), while deeper speech RVQ layers are aggregated as a residual paralinguistic code \(\mathbf{Z}_r\). This creates a hierarchical latent organization in which semantic content, residual speech attributes, and background sound are all separately addressable [2509.09201].

A useful contextual contrast is provided by DisCo-Speech, whose DisCodec factorizes speech into content, prosody, and timbre for controllable zero-shot TTS, but remains centered on speech generation rather than mixed-audio universal representation learning [2512.13251].

## 2. Architecture and latent organization

DeCodec follows a DAC-style codec backbone with added structure. The encoder is a 1D-convolutional encoder with several downsampling blocks at rates \([2,4,5,8]\), optional (Bi)LSTMs, and a final convolution to dimension \(D=1024\). Given waveform input \(\mathbf{y}\), it produces a frame-wise embedding \(\mathbf{Y}\). A decoder mirrors the encoder with transposed convolutions and LSTMs and maps combined discrete codes back to waveform \(\hat{\mathbf{y}}\). Two variants are provided: DeCodec-c, which uses a causal LSTM, and DeCodec, which uses a BiLSTM [2509.09201].

The Subspace Orthogonal Projection block applies two trainable linear projection layers:
\[
\mathbf{S} = \mathbf{P}_S \mathbf{Y}, \quad \mathbf{N} = \mathbf{P}_N \mathbf{Y},
\]
with \(\mathbf{P}_S, \mathbf{P}_N \in \mathbb{R}^{D \times D}\). In practice, these are learned linear layers of size \(1024 \to 1024\). The design target is
\[
\mathbf{Y} = \mathbf{P}_S \mathbf{Y} + \mathbf{P}_N \mathbf{Y},
\]
so that speech and background sound occupy distinct, ideally orthogonal, subspaces [2509.09201].

Quantization is performed by parallel residual vector quantizers. The speech RVQ has \(K_s=8\) stages and the noise RVQ has \(K_n=8\) stages; both use codebooks of size 1024 and embedding dimension 1024. The speech quantization is defined as
\[
\mathbf{Z}_s = \sum_{k=1}^{K_s} \mathbf{Q}_s^k,
\]
and the background quantization as
\[
\mathbf{Z}_n = \sum_{k=1}^{K_n} \mathbf{Q}_n^k.
\]
Within speech,
\[
\mathbf{Z}_c = \mathbf{Q}_s^1, \qquad \mathbf{Z}_r = \sum_{k=2}^{K_s} \mathbf{Q}_s^k,
\]
so that
\[
\mathbf{Z}_s = \mathbf{Z}_c + \mathbf{Z}_r.
\]
This organization is the basis for selective latent reuse and recombination across tasks [2509.09201].

| Latent component | Construction | Intended information |
|---|---|---|
| \(\mathbf{Z}_c\) | \(\mathbf{Q}_s^1\) | Semantic content |
| \(\mathbf{Z}_r\) | \(\sum_{k=2}^{K_s} \mathbf{Q}_s^k\) | Paralinguistic and acoustic residual |
| \(\mathbf{Z}_n\) | \(\sum_{k=1}^{K_n} \mathbf{Q}_n^k\) | Background sound |

The resulting bitrate for the reported system is \(4.0 + 4.0\) kbps, corresponding to parallel speech and background quantization. This is higher than some ultra-low-bitrate semantic codecs, but the stated trade-off is broader reconstruction fidelity and general-purpose functionality [2509.09201].

## 3. Disentanglement mechanisms

DeCodec uses three coordinated mechanisms: subspace orthogonality, representation swap training, and semantic guidance. Their roles are complementary rather than interchangeable [2509.09201].

The orthogonality mechanism is encoded by
\[
\mathcal{L}_{\perp} = \left\| \langle \mathbf{S}, \mathbf{N} \rangle - \mathbf{0} \right\|_2.
\]
Minimizing \(\mathcal{L}_{\perp}\) encourages \(\mathbf{S}\) and \(\mathbf{N}\) to be orthogonal. The paper further argues that if the covariance \(\mathbf{Y}\mathbf{Y}^T\) is approximately diagonal, then \(\mathbf{P}_S \mathbf{P}_N^T \approx \mathbf{0}\), so the learned operators behave as orthogonal projectors. Empirically, this is supported by a cosine-similarity distribution between rows of \(\mathbf{P}_S\) and \(\mathbf{P}_N\) centered around 0, and by distinct singular value spectra for the two projection matrices [2509.09201].

Orthogonality alone, however, is not sufficient. Representation Swap Training assigns functional meaning to the two subspaces. Given two uncorrelated mixtures,
\[
\mathbf{y}_{11} = \mathbf{s}_1 + \mathbf{n}_1,\quad \mathbf{y}_{22} = \mathbf{s}_2 + \mathbf{n}_2,
\]
the model encodes speech from the first mixture and background from the second, then decodes the swapped combination:
\[
\hat{\mathbf{y}}_{12} = \operatorname{Dec}(\mathbf{Zs}_1 + \mathbf{Zn}_2) \approx \mathbf{s}_1 + \mathbf{n}_2.
\]
The corresponding loss is
\[
\mathcal{L}_{\text{RST}} = \left\| \operatorname{Dec}(\mathbf{Zs}_1 + \mathbf{Zn}_2) - (\mathbf{s}_1 + \mathbf{n}_2) \right\|_1.
\]
The paper’s argument is that if swapping is to remain consistent across arbitrary pairs, then speech codes cannot carry noise and noise codes cannot carry speech. This makes RST a functional disentanglement criterion rather than merely a geometric one [2509.09201].

Semantic Guidance produces the second-stage split inside the speech subspace. DeCodec uses HuBERT-L9 as a semantic teacher. Let \(\mathbf{\mathcal{H}}\) be the HuBERT-L9 embedding of the corresponding clean speech and \(\mathbf{W}\) a learnable projection mapping \(\mathbf{Z}_c\) to HuBERT’s dimensionality. The loss is
\[
\mathcal{L}_{\text{SG}} = \left\| \log \sigma\left( \cos(\mathbf{W}\mathbf{Z}_c, \mathbf{\mathcal{H}}) \right) \right\|_1.
\]
This is intended to make \(\mathbf{Z}_c\) semantic-dominant and push speaker identity, prosody, and related non-linguistic factors into \(\mathbf{Z}_r\) [2509.09201].

A frequent misconception is that orthogonality alone guarantees disentanglement. The ablation results contradict that view: SOP-only and RST-only settings did not yield consistent speech/background separation, while SOP + RST produced workable speech/BGS decoupling, and adding SG markedly improved semantic quality as measured by downstream ASR WER\* [2509.09201].

## 4. Objective, optimization, and ablation structure

DeCodec is trained as a GAN-based codec with a weighted sum of disentanglement, reconstruction, adversarial, and quantization objectives. The total objective includes orthogonality loss \(\mathcal{L}_{\perp}\), representation swap loss \(\mathcal{L}_{\text{RST}}\), semantic guidance loss \(\mathcal{L}_{\text{SG}}\), a reconstruction loss based on the DAC multi-scale Mel loss, adversarial losses using a multi-period discriminator and a multi-band multi-scale STFT discriminator, and standard RVQ codebook and commitment losses of the VQ-VAE form [2509.09201].

The stated loss weights are: \(\mathcal{L}_{\text{RST}}\) at 500.0, \(\mathcal{L}_{\text{SG}}\) at 150.0, \(\mathcal{L}_{\perp}\) at 10.0, reconstruction loss at 10.0, discriminative loss at 1.0, and codebook and commitment losses at 1.0 and 10.0, respectively, for all parallel RVQs. This weighting makes explicit that the training objective prioritizes disentanglement, especially through RST and SG, rather than treating it as a weak regularizer [2509.09201].

Training data combine speech corpora and noise corpora. The reported setup uses Aishell3, LibriTTS train-clean-100/360, VCTK, and WSJ0/WSJ1 for speech; ESC-50 and DNS-Noise for background; and approximately 700 hours of speech randomly mixed with noise at SNR from \(-5\) dB to 40 dB. Evaluation uses LibriSpeech test-clean and noisy versions formed by mixing test-clean with DNS-Noise at SNR \(-5\) to 20 dB, plus the DNS Challenge blind test for speech enhancement [2509.09201].

The ablation study isolates the contributions of SOP, RST, and SG. On the noisy test set, SOP-only achieved SDR-O 8.93 but SDR-B \(-13.15\) and SDR-S \(-1.91\), indicating that overall reconstruction can remain high even when disentanglement fails. RST-only yielded SDR-O 6.70, SDR-B \(-10.67\), and SDR-S 3.03, again without consistent decoupling. SOP + RST produced SDR-O 6.68, SDR-B 0.49, SDR-S 7.90, and WER\* 41.9, which the paper interprets as successful speech/BGS separation. Adding SG reduced SDR but improved WER\*: DeCodec-c reached SDR-O 4.62, SDR-B \(-1.11\), SDR-S 5.70, WER\* 25.8, while the non-causal DeCodec reached SDR-O 5.21, SDR-B \(-0.36\), SDR-S 6.73, WER\* 23.6 [2509.09201].

This trade-off is central to the method. The paper explicitly notes that semantic guidance slightly compromises SDR while substantially improving semantic purity and ASR robustness. This suggests that DeCodec optimizes for structured controllability rather than for maximal waveform reconstruction alone [2509.09201].

## 5. Reconstruction quality and downstream capabilities

DeCodec is evaluated both as a codec and as a latent front-end. On clean speech, DeCodec-c at \(4+4\) kbps reports SDR 6.79, Mel 0.88, and Whisper WER 1.98, while the non-causal DeCodec reports SDR 7.61, Mel 0.89, and WER 1.92. Baselines on the same evaluation include EnCodec at 6 kbps with SDR 6.86, Mel 1.03, WER 2.28; HiFi-Codec at 2 kbps with SDR 4.85, Mel 0.75, WER 2.61; DAC at 4.5 kbps with SDR 0.60, Mel 0.65, WER 2.21; and SpeechTokenizer at 4 kbps with SDR 3.41, Mel 0.76, WER 1.82. On noisy speech, the non-causal DeCodec reports SDR 5.21 and Mel 0.81, compared with EnCodec at SDR 4.88 and Mel 0.84, HiFi-Codec at SDR \(-0.66\) and Mel 0.90, DAC at SDR \(-1.62\) and Mel 0.69, and SpeechTokenizer at SDR \(-0.50\) and Mel 0.90 [2509.09201].

The downstream manipulations are direct consequences of latent recombination. Speech enhancement is performed by dropping background codes and decoding \(\mathbf{Z}_s\) with background codes obtained from blank audio. On the DNS Challenge test set, DeCodec achieved DNSMOS OVL 3.39, SIG 3.64, and BAK 4.13 on synthetic data without reverb, outperforming Inter-SubNet, StoRM, and SELM on all three reported measures. On real recordings, DeCodec achieved OVL 3.13, SIG 3.45, and BAK 3.99, which was the best BAK and competitive in OVL and SIG [2509.09201].

One-shot voice conversion on noisy speech is performed by combining semantic codes from a source utterance, residual speech codes from a target-speaker reference, and optionally blank background codes for simultaneous denoising:
\[
\hat{\mathbf{y}}_{\text{VC+SE}} = \operatorname{Dec}(\mathbf{Z}_c^{\text{src}} + \mathbf{Z}_r^{\text{ref}} + \mathbf{Z}_n^{\text{blank}}).
\]
On the reported noisy test set, DeCodec achieved SIM 0.83 and WER 50.46. SpeechTokenizer without denoising achieved SIM 0.80 and WER 74.18, and StoRM + SpeechTokenizer achieved SIM 0.83 and WER 52.73. The reported interpretation is that DeCodec matches the best baseline in speaker similarity while improving intelligibility without requiring an external speech-enhancement module [2509.09201].

For ASR, DeCodec is used as a front-end feature extractor for a decoder-only Transformer trained on LibriSpeech. The semantic-only speech code and the full speech code both outperform other codec latents under noise. On the noisy evaluation, DeCodec achieved WER\* 26.7 using SRVQ-1 only and 23.6 using SRVQ-1:8, compared with SpeechTokenizer at 59.2 and 55.1, StoRM + SpeechTokenizer at 34.5 and 32.1, and DAC at 78.0 and 82.5. Clean-condition WER\* values were 14.7 and 12.5 for DeCodec, versus 15.5 and 13.5 for SpeechTokenizer. The paper interprets this as evidence that SOP + RST + SG yields semantically rich and noise-robust features [2509.09201].

For TTS, DeCodec latents are used to train VALL-E on LibriTTS and test on noisy reference speech. By retaining or suppressing background codes, the system can synthesize either background-preserving or background-suppressed speech. Reported subjective scores are: SpeechTokenizer alone, MOS 1.48, SMOS 1.82, BRMOS 1.97; StoRM + SpeechTokenizer, MOS 4.05, SMOS 3.68, BRMOS 4.68; DeCodec with background removed, MOS 3.96, SMOS 3.69, BRMOS 4.74; and DeCodec with background preserved, MOS 4.09 and BPMOS 4.19. The result is not merely denoising robustness, but explicit control over whether background sound is retained in synthesis [2509.09201].

## 6. Position in the broader research landscape

DeCodec occupies a distinct position among codec-related disentanglement and decoding methods. ScoreDec augments a 24 kbps AudioDec system with a diffusion-based complex-spectral post-filter that preserves phase and avoids GAN-based decoder fine-tuning, but it is a codec post-filter rather than a latent factorization framework [2401.12160]. “Audio Decoding by Inverse Problem Solving” treats transform-domain audio decoding as posterior sampling with a task-agnostic diffusion prior and analytic conditioning functions, again emphasizing decoding quality rather than latent controllability [2409.07858]. LaDiffCodec separates low-bitrate discrete representation learning from information reconstruction by using latent diffusion for generative de-quantization, but its central decomposition is between discrete and continuous reconstruction spaces, not between speech, semantics, and background sound [2311.08330]. DisCodec in DisCo-Speech explicitly disentangles content, prosody, and timbre for zero-shot controllable TTS, but it is speech-specific and does not model mixed speech-plus-background audio as the primary object of representation learning [2512.13251].

Within that landscape, DeCodec’s stated novelty is that it is the first codec to explicitly achieve speech versus background disentanglement and semantic versus paralinguistic decomposition inside speech in a single unified model, while also matching or surpassing top codecs in reconstruction on the reported tasks [2509.09201]. A plausible implication is that the method reframes codec evaluation away from a single rate-distortion axis and toward a multi-objective axis that includes latent selectivity and recomposability.

The term “DeCodec” also has a broader, sometimes ambiguous usage. In communications, it can refer to deep-learning-based decoders for binary linear codes, such as ADMM-unfolded decoding networks with learned penalty parameters and learned piecewise-linear penalties [2002.07601]. In audio, it can refer more loosely to decoder-centric or decomposed-codec designs. In the specific sense established by the 2025 audio paper, however, DeCodec denotes a codec whose latent space is explicitly structured for task-aware access to speech, background sound, semantics, and residual speech attributes [2509.09201].

The limitations are explicit. Adding semantic guidance slightly reduces overall SDR relative to SOP + RST alone. The reported bitrate of \(4.0 + 4.0\) kbps is higher than some ultra-low-bitrate semantic codecs. Training relies on HuBERT as an external semantic teacher. The noise model is based on DNS-Noise and ESC-50, so very unusual acoustic conditions may remain challenging. The training corpora are typical speech datasets, which leaves cross-lingual and highly expressive generalization as open questions. The paper also notes that disentanglement is not mathematically perfect and that some leakage may remain, especially at very low SNR or under unusual acoustic conditions [2509.09201].

Source: https://www.emergentmind.com/topics/decodec