---
title: 'DisCo-Speech: Zero-Shot Disentangled TTS'
url: https://www.emergentmind.com/topics/disco-speech
type: topic
---

# DisCo-Speech: Zero-Shot Disentangled TTS

DisCo-Speech is a zero-shot, controllable text-to-speech (TTS) architecture that addresses the challenge of disentangling linguistic content, prosody, and speaker timbre at the codec level. The approach introduces a disentangled speech codec (DisCodec), paired with an autoregressive Transformer language model (LM), achieving independent prosody control and voice cloning in a unified framework. This separation of prosody and timbre is accomplished through parallel encoding, quantization, and carefully constructed loss objectives, establishing state-of-the-art flexibility in zero-shot speech synthesis and voice conversion [2512.13251].

## 1. Disentangled Speech Codec Architecture

DisCodec factorizes an input waveform $x$ into three distinct streams: content ($c$), prosody ($p$), and timbre ($t$), utilizing parallel encoders and finite scalar quantization (FSQ).

- **Content Encoder ($E_c$)**: Employs convolutional blocks inspired by DAC, downsampling $x$ into frame-level latent $h_c$, quantized to discrete content tokens $q_c = Q_c(h_c)$. Supervision leverages a phone recognizer (fine-tuned Wav2Vec) via cross-entropy loss, ensuring $q_c$ encodes purely linguistic information.

- **Prosody Encoder ($E_p$)**: Utilizes dilated causal convolutions (Wavenet architecture) to produce latent $h_p$, quantized by two-stage residual FSQ into $q_{p1}$, $q_{p2}$, summed as $q_p = q_{p1} + q_{p2}$. Supervision comprises a frame-level F0 (pitch) regression loss, a correlation loss controlling overlap between $q_{p1}$ and $q_{p2}$ (with target similarity $\alpha=0.2$), and a gradient-reversal layer (GRL) to minimize timbre leakage.

- **Timbre Encoder ($E_t$)**: Based on ECAPA-TDNN applied to Mel-spectrograms, with global aggregation via cross-attention using learnable queries. The resulting global timbre vector $g_t$ is quantized to 48-dimensional token $q_t$ and optimized with a speaker-classification loss to encode only speaker identity.

Tri-factor disentanglement losses use soft orthogonality regularization:
\[
\mathcal{L}_{soft}^{p,c} = 
\Bigl(\frac{1}{BL} \sum_{b,l} \bigl|\cos(\ell_p^{(b,l)}, \ell_c^{(b,l)})\bigr| - \beta_c \Bigr)^2
\]
\[
\mathcal{L}_{soft}^{p,t} = 
\Bigl(\frac{1}{BL} \sum_{b,l} \bigl|\cos(\ell_p^{(b,l)}, q_t^{(b)})\bigr| - \beta_t \Bigr)^2
\]
with $\beta_c=0.01$, $\beta_t=0.0001$, balancing decoupling with information retention.

The first-stage DAC-style decoder recombines $(q_c, q_p, q_t)$ to reconstruct the waveform, supervised by multi-scale Mel-spectrogram and waveform reconstruction objectives.

## 2. Fusion for Language Modeling and Token Reconstruction

In the second stage, DisCo-Speech fuses content and prosody into a single token sequence $z_{cp}$, optimized for joint prediction by the language model. Encoders are frozen, and a new decoder $D_2$ (built with Transformer blocks and a BigVGANv2 generator) takes tokens $(z_{cp}, q_t)$ to reconstruct the waveform. Fusion is performed as:
\[
z_{cp} = Q_{cp}(\mathrm{Dequantize}(q_c) + \mathrm{Dequantize}(q_p))
\]
where $Q_{cp}$ is an FSQ quantizer (codebook size 65,536), and the reconstruction objective is a combination of adversarial, feature-matching, and multi-scale spectrogram losses as in BigVGAN.

This staged design resolves the entanglement-reconstruction trade-off by optimizing $\mathcal{L}_{rec}^{(1)}$ for disentangled representation, then $\mathcal{L}_{rec}^{(2)}$ for fidelity, keeping disentangled features fixed.

## 3. Transformer LM Integration and Inference Pipeline

A standard autoregressive Transformer LM (initialized from Qwen2.5-1.5B) models the conditional distribution over text tokens $t_c$ and fused content–prosody tokens $z_{cp}$. The model is trained using next-token cross-entropy with input sequences of the form $[S, t_c, T, z_{cp}, E]$, where $S$, $T$, $E$ denote start, turn, and end tokens respectively.

During inference, the procedure is:

1. **Style prompt**: Given a reference speech sample $x'$ and its transcript $t_c'$, extract $(z_{cp}', q_t')$.
2. **Context construction**: Build sequence $[S, t_c', T, z_{cp}', T, t_c^{sys}]$ where $t_c^{sys}$ is the new target text.
3. **Prosodic continuation**: The LM autoregressively generates $z_{cp}^{sys}$, extending the prosody pattern of the prompt onto the target text.
4. **Timbre injection**: The decoder $D_2$ synthesizes the waveform from $(z_{cp}^{sys}, q_t^{target})$, injecting the desired timbre.

This design ensures clean separation of prosody continuation (via LM) and timbre injection (via the decoder), supporting zero-shot prosody transfer with arbitrary style prompts.

## 4. Training Objectives, Regularization, and Optimization

Training proceeds in two stages:

- **Stage 1 (Disentanglement)**: Joint optimization of reconstruction, phone, F0, correlation, soft orthogonality, and speaker losses:
\[
\mathcal{L}^{(1)} = \mathcal{L}_{rec}^{(1)} + \lambda_{pho}\mathcal{L}_{pho} + \lambda_{f0}\mathcal{L}_{f0} + \lambda_{cor}\mathcal{L}_{cor} + \lambda_{sc}\mathcal{L}_{soft}^{p,c} + \lambda_{st}\mathcal{L}_{soft}^{p,t} + \lambda_{spk}\mathcal{L}_{spk}.
\]
- **Stage 2 (Fusion and Fidelity)**: Only the fusion decoder $D_2$ is trained (with encoders frozen) on the BigVGAN-style loss.

- **Language Model**: Pre-training and fine-tuning on 120,000 hours of speech data, using Adam optimizer (with learning rates: stages 1/2: $1\times10^{-4}$, LM: $2\times10^{-4}$, batch size 176 on eight A800 GPUs, 500k steps in stage 1).

Hyperparameters (codebook sizes, GRL for prosody, orthogonality regularization) are tuned to balance disentanglement and information retention.

## 5. Experimental Evaluation and Results

Experiments span codec reconstruction, voice conversion, and zero-shot prosody/voice control. Key metrics and benchmarks include:

| Evaluation Task         | Metrics & Results                                                            | Comparative Baselines    |
|------------------------|------------------------------------------------------------------------------|--------------------------|
| Codec Reconstruction   | WER↓=3.44%, STOI↑=0.88, PESQ WB↑=2.31, UTMOS↑=4.10, SSIM↑=0.81               | Top single-stream codecs |
| Voice Conversion       | UTMOS↑=3.98, SSIM↑=61.1%, F0 corr↑=0.58 (1,680 pairs)                        | SeedVC, Vevo             |
| Prosody Control (AB)   | Timbre: 45.3–51.5% vs. Vevo (21.3–40.2%); Prosody: 48.9–50.6% vs. Vevo (20.0–36.7%) | Vevo, IndexTTS 2         |
| Voice Cloning (EN/ZH)  | WER=3.08%, SSIM=58.8% (EN); CER=1.64%, SSIM=68.1% (ZH)                       | Spark-TTS, multistage    |

DisCo-Speech consistently demonstrates high speaker similarity (SSIM), intelligibility (WER/CER), and prosody preservation, while outperforming strong baselines in zero-shot prosody control and disentanglement. Subjective AB preference also favors DisCo-Speech for timbre and prosody transfer in style/emotion tasks.

## 6. Significance, Limitations, and Implications

By explicitly factorizing speech into content, prosody, and timbre at the codec level and fusing only content–prosody for language modeling, DisCo-Speech resolves the entanglement–reconstruction trade-off that has hindered previous continuation-based TTS architectures. This clean separation enables flexible, zero-shot prosody continuation and robust voice cloning, establishing a foundation for controllable TTS systems. A plausible implication is enhanced downstream adaptability in scenarios requiring fine-grained speech parameter control.

No claims or results suggest fundamental limitations within the described evaluation regime; however, as with all TTS systems, scaling to new languages and domains may require adaptation of encoders or quantization strategies.

## 7. Availability and Reproducibility

Audio samples, code, and model weights for DisCo-Speech are available at [https://github.com/disco-speech/DisCo-Speech](https://github.com/disco-speech/DisCo-Speech) [2512.13251]. The reproducibility of results is facilitated by the release of code and pretrained weights alongside full objective definitions and architecture details.

Source: https://www.emergentmind.com/topics/disco-speech