FC-TTS: Style and Timbre Control in Zero-Shot Text-to-Speech with Disentangled Speech Representations

Abstract: Recent advances in zero-shot text-to-speech (TTS) have enabled accurate imitation of reference speech in terms of both speaking style and speaker timbre. However, achieving disentangled control over these aspects from separate references remains a challenging task. Several studies have proposed disentangled speech representations that decompose speech into interpretable attributes (e.g., timbre, prosody, and content), providing a promising foundation for TTS with attribute control from separate references. Yet, how to effectively integrate such representations into TTS systems to achieve independent and precise control remains underexplored. In this paper, we present FC-TTS, a zero-shot TTS framework that enables disentangled control of style and timbre by conditioning on two distinct reference utterances. Unlike existing systems that inherit limitations from those pre-trained disentangled representations, FC-TTS introduces key design strategies, including architectural choices, training framework, and auxiliary training objectives, which improve the reliability of attribute separation and dual-reference control. Experiments show that FC-TTS achieves high-fidelity synthesis and competitive zero-shot naturalness, while uniquely supporting consistent and independent manipulation of style and timbre. Audio samples are available at https://qualcomm-ai-research.github.io/fc-tts

• The paper introduces a two-stage hierarchical TTS architecture that separates style and timbre processing for robust zero-shot synthesis.
• It employs a VQ-VAE style encoder and conditional consistency loss to achieve clean disentanglement and improved prosodic fidelity.
• Evaluation on Libriheavy and RAVDESS confirms superior performance in both objective and subjective metrics compared to baseline codec systems.

FC-TTS: Disentangled Style and Timbre Control for Zero-Shot Text-to-Speech Synthesis

Motivation and Context

Modern zero-shot text-to-speech (TTS) calls for fine-grained control over multiple speech attributes, notably speaker timbre and expressive style. Prior systems typically entangle these attributes in a single reference, limiting their independent manipulability. The FC-TTS framework explores how factorized, disentangled representations can be practically leveraged for robust, independent style and timbre control, addressing limitations of prior codec-based and reference-guided TTS frameworks.

Architectural Innovations

FC-TTS introduces a two-stage hierarchical spectrogram generation pipeline, a VQ-VAE based style encoder, and a conditional consistency loss to achieve reliable disentanglement and controllability.

The first stage produces a blurry log-mel spectrogram anchored by timbre embedding $z_{\text{spk}}$. The second stage refines this intermediate spectrogram with prosody tokens $\mathbf{c_p}$, imprinted via dedicated hierarchical TCF modules employing Q-Former-style cross-attention and finite scalar quantization. This separation ensures each attribute reference affects only its designated phase, minimizing information leakage and maximizing attribute-specific control.

Figure 1: FC-TTS architecture: sequential conditioning enables independent control of timbre and style through distinct reference utterances.

The style encoder (TCF module) hierarchically extracts prosodic representations at both the phoneme and frame level using a Transformer encoder, Q-former compression, and FSQ discretization. An auxiliary ResNet-based reconstruction component counteracts latent collapse, supporting high-fidelity, semantically rich prosodic encoding.

Figure 2: TCF module: hierarchical prosody compression and quantization, with joint reconstruction for stability.

Disentanglement and Conditioning Efficacy

Purely autoencoding, factorized codecs (e.g., FACodec) have shown imperfect disentanglement, with residual attribute leakage across streams. FC-TTS mitigates this by excluding content and detail tokens from conditioning and employing a cross-conditioned consistency loss that trains attribute predictors to reinforce clean timbre/style separation.

The conditional consistency loss (CCL) extends prior regularization schemes by enforcing joint coherence: predictors trained to recover target prosody and timbre from generated spectrograms also receive non-target attribute inputs, sharpening gradient directions and improving disentangled conditioning during noisy synthesis steps.

Figure 3: Gradients for CCL with multiple conditions. Cross-conditioning enhances precision in attribute control.

Robustness to Real-World Variability

FC-TTS explicitly avoids assumptions of style uniformity within utterances. It conditions on style representations dynamically extracted from actual target speech, enabling control over intra-utterance style variations and avoiding shortcut learning. The architecture is validated on the Libriheavy and RAVDESS corpora, both exhibiting significant diversity in timbre and style.

Figure 4: Libriheavy sample with multiple stylistic modes in a single utterance.

Empirical Evaluation and Ablation

Zero-Shot and Disentangled Attribute Control

FC-TTS achieves competitive zero-shot synthesis quality and intelligibility, as measured by UTMOS, WER, and SPK, while uniquely supporting consistent and independent manipulation of style and timbre. On objective and subjective metrics—including ABX perceptual tests and AudioLLM-as-a-Judge protocols—FC-TTS exhibits strong performance in mismatched style-timbre control settings, outperforming baseline and oracle codec-based TTS systems.

In prosodically rich RAVDESS scenarios, FC-TTS maintains high UTMOS (4.03 vs. 3.19 for FACodec-VC), significantly superior SPK and WER, and achieves strong preference rates in ABX evaluations (66.1% vs. 10.7% for FACodec-VC).

A similar trend is observed for style control, where FC-TTS outperforms retrained F5-TTS across all metrics under stricter, disentangled inference with separate references.

Spectral Analysis and Component Contribution

Spectrograms generated by ablation variants confirm the necessity of each architectural innovation:

• Removal of the two-stage pipeline produces unstable acoustic patterns with over-reflected prosody.
• Absence of the VQ-VAE encoder results in flattened F0 contours, undermining accurate prosody following.
• Eliminating CCL sharply degrades intelligibility and prosodic consistency.

  Figure 5: Log-mel spectrograms from ablation variants; boxed regions highlight prosody and timbre control failures.

Practical and Theoretical Implications

FC-TTS establishes an explicit foundation for scalable, expressive TTS with interpretable and independent attribute control at inference. Practically, it enables a spectrum of personalized speech synthesis applications—virtual assistants, audiobooks, interactive media—requiring robust control over speaker identity and expressive mode.

Theoretically, FC-TTS exposes limits of current factorized codec designs and points to the necessity of stronger disentanglement mechanisms or non-codec alternatives. The efficacy of cross-conditioned consistency loss suggests promising avenues for multi-attribute generative modeling in speech and beyond. The model additionally surfaces open questions about attribute boundaries (e.g., timbre vs. style) and reliable automatic metrics for prosodic similarity.

Ethical Considerations

The strong zero-shot and independent control capabilities pose risks for misuse in deepfake generation, impersonation, and emotionally manipulated speech. Mitigation could involve restricting style controllability or conditioning deployment on authorized sources, though further technical and policy research is necessary.

Conclusion

FC-TTS demonstrates robust disentangled control of style and timbre in zero-shot TTS, enabled by a hierarchical, two-stage pipeline, a quantized style encoder, and conditional consistency loss. It achieves competitive synthesis quality and interpretable control not prioritized nor achieved by prior reference-based or codec-based systems. The framework serves as a basis for future research in attribute decomposition, codec-free modeling, and safeguarded expressive TTS technologies.