Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS

Abstract: Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is typically controlled by two conditions: the target text and a prompt speech signal. Standard CFG strengthens these conditions jointly, while recent branch-selective guidance methods attempt to enhance text or speaker conditioning separately, often leading to a trade-off between text correctness and speaker similarity. In this paper, we revisit the CFG under independently masked text and speech-prompt conditions, and decompose the guidance field into text, speaker, and joint residuals. We show that conventional speaker-selective guidance entangles the speaker residual with the joint residual, which may disturb text-related generation. Based on this observation, we propose joint residual reweighting, which independently controls the speaker and joint residuals within the standard CFG framework. Experiments on F5-TTS and CosyVoice2 show that the proposed method improves speaker similarity while maintaining competitive text correctness, demonstrating the usefulness of the joint residual for balancing speaker fidelity and text accuracy in zero-shot TTS.

• The paper introduces a four-branch residual decomposition that isolates text, speaker, and joint contributions for improved classifier-free guidance.
• It proposes joint residual reweighting to independently amplify speaker and joint effects, balancing speaker similarity with text fidelity.
• Empirical evaluations on F5-TTS and Cosy Voice2 demonstrate improved speaker similarity and reduced error rates compared to selective CFG approaches.

Joint Residual Reweighting for Classifier-Free Guidance in Flow-Matching Zero-Shot TTS

Overview and Motivation

The paper "Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS" (2606.25672) investigates the inference-time guidance mechanisms used in flow-matching-based zero-shot text-to-speech (TTS). The central concern is the balancing act between text fidelity and speaker similarity when synthesizing speech for unseen speakers based on minimal prompt data. Standard classifier-free guidance (CFG) amplifies both text and speaker conditioning for improved generation quality, but recent branched guidance strategies demonstrate inherent trade-offs: enhancing speaker similarity often comes at the cost of text accuracy, and vice versa.

The authors identify that such strategies generally rely on pairwise-branch differences (e.g., text-only vs. full conditioning), which do not adequately isolate the interaction effects between text and speaker prompts. To address this, the authors propose a four-branch decomposition aligned with masking strategies for text and speech-prompt conditions. This framework allows explicit separation of text, speaker, and joint residual contributions within the CFG vector field, and facilitates fine-grained control via "joint residual reweighting"—independent adjustment of joint and speaker residuals.

Technical Contributions

Four-Branch Residual Decomposition

The paper formalizes the velocity field predictions in flow-matching TTS models into four branches:

• Null Branch: No text or speaker prompt.
• Text-Only Branch: Text present, speaker prompt absent.
• Speaker-Only Branch: Speaker prompt present, text absent.
• Full-Condition Branch: Both text and speaker prompt present.

The standard CFG amplifies the difference between the full and null branches. However, the decomposition shows this difference contains three distinct components: text residual, speaker residual, and a joint residual (the latter arises only in the presence of both text and speaker prompt). The authors define the joint residual mathematically as:

$$T_{\text{joint}} = V_{\text{full}} - V_{\text{text}} - V_{\text{spk}} + V_{\text{null}}$$

This formalism exposes the limitations of prior branch-centric CFG rules, which implicitly conflate speaker and joint effects, potentially disrupting text generation.

Joint Residual Reweighting

Building upon the decomposition, the paper proposes joint residual reweighting. The guided velocity during sampling in flow-matching TTS can be controlled as:

$$V = U_{CFG} + \gamma_{\text{spk}} \cdot A_{\text{spk}} + \gamma_{\text{joint}} \cdot T_{\text{joint}}$$

Where $U_{CFG}$ is the base CFG, and the $\gamma$ coefficients enable independent amplification. This allows empirical tuning to optimize speaker similarity without sacrificing text accuracy. Implementation involves additional per-step branch computation (four branches instead of two), but remains computationally tractable via batched evaluation.

Relation to Prior Guidance Strategies

The paper situates joint residual reweighting within a unified taxonomy of guidance rules, demonstrating that conventional CFG and selective CFG strategies can be viewed as special cases under the four-branch residual-weight framework. Speaker-selective CFG and methods from image generation (e.g., VoiceLDM, DualSpeech) are contrasted, with the authors highlighting that joint residual editing introduces new control degrees of freedom absent from these prior approaches.

Empirical Evaluation

The method is evaluated on F5-TTS and Cosy Voice2 backbones, using standard datasets (LibriSpeech-test, SEED-EN, SEED-ZH). Key metrics include Speaker Similarity (SIM), Word Error Rate (WER), and Character Error Rate (CER). Major quantitative findings include:

• On Cosy Voice2, SIM improves from 0.6561 to 0.6690 (LibriSpeech-test), from 0.6586 to 0.6706 (SEED-EN), and from 0.7531 to 0.7631 (SEED-ZH), with reductions in error rates.
• On F5-TTS, SIM increases from 0.6745 to 0.6819 (LibriSpeech-test) and from 0.6811 to 0.6875 (SEED-EN), while CER marginally decreases.
• Compared to Selective CFG, joint residual reweighting achieves comparable or superior speaker similarity, with improved text accuracy.

Ablation studies further clarify component contributions. Amplifying joint residuals yields better speaker similarity, while indiscriminant addition of text and speaker residuals risks degrading respective objectives.

Discussion and Implications

Practical Impact

The joint residual reweighting framework offers nuanced control over TTS generation, especially for zero-shot applications where prompt scarcity constrains speaker adaptation. Empirical evidence demonstrates utility in pushing speaker similarity without text degradation, which is desirable for personalized TTS services, voice cloning, and multilingual synthesis.

The method incurs higher inference cost by requiring four branches, but this is mitigated by parallelism. Implementation nuances depend on model-specific conditioning, emphasizing the importance of careful branch definition for transferability across architectures.

Theoretical Relevance

Joint residual decomposition enriches the mathematical understanding of guidance dynamics in conditional generative modeling, suggesting new routes for multi-factor disentanglement in flow-based or diffusion-based TTS architectures. The explicit separation of joint effects addresses subtleties in multimodal conditioning largely unexamined in prior CFG literature.

Future Directions

There are actionable paths to reduce computational overhead, such as adaptive scheduling, partial application, or distillation strategies. The conceptual framework may also generalize to other generative tasks (e.g., image, video synthesis), where multiple conditions interact nontrivially.

Conclusion

This paper establishes a four-branch residual decomposition for classifier-free guidance in flow-matching zero-shot TTS and proposes a joint residual reweighting strategy to independently tune speaker and joint residual strengths. Experimental results validate that controlling the joint residual yields consistent improvements in speaker similarity with competitive or superior text accuracy. The research provides a refined perspective and practical mechanism for balancing conditional objectives in flow-matching generative models, with implications for zero-shot TTS and broader multi-condition generative modeling (2606.25672).