VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

Abstract: Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid personalization. We propose VoiceTTA, a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models. VoiceTTA introduces two style rewards based on coefficient-of-variation differences of F0 and energy, combined with speaker similarity and intelligibility (WER from a pretrained Whisper model), and optimizes learnable prefixes via group relative preference optimization (GRPO) in a flow matching-based model at inference time. Extensive experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines. Audio samples are available at https://voicetta.pages.dev/

• The paper introduces a minimal-parameter, RL-based test-time adaptation method that enhances zero-shot TTS by optimizing learnable prefix tokens.
• It leverages composite rewards—including F0-CV, Energy-CV, speaker similarity, and intelligibility—to balance stylistic adaptation with speech clarity.
• Experimental results demonstrate superior style similarity and near-parity in naturalness, achieving rapid personalization with minimal overhead.

VoiceTTA: Reinforcement Learning-Based Test-Time Adaptation for Zero-Shot TTS

Motivation and Problem Setting

Zero-shot text-to-speech (TTS) has made significant progress in synthesizing natural, high-fidelity speech from brief or single reference prompts. However, state-of-the-art zero-shot TTS models are challenged by domain shifts, particularly when confronted with speaking styles substantially different from their training distribution—such as regional dialects, dramatic prosodic variations, accented or slurred speech, and niche communicative contexts. Existing adaptation approaches (speaker embeddings, fine-tuning) either depend on high-quality embedding models or require substantial target-speaker data, rendering them inefficient for rapid, on-the-fly personalization and arguably limited in capturing uncommon or highly variable styles.

VoiceTTA proposes a minimal-parameter, reinforcement learning-based test-time adaptation (TTA) paradigm to bridge this gap. The method performs adaptation at inference on a few seconds of test speech, using reward-guided optimization of token prefixes within a flow matching-based TTS model. Unlike conventional fine-tuning (repeat of pre-training tasks) or embedding-based adaptation, VoiceTTA leverages auxiliary tasks (prosody/style rewards, speaker similarity, intelligibility) to guide adaptation explicitly towards the target speech's stylistic attributes.

The distinction between the fine-tuning and TTA paradigms is visualized in (Figure 1).

Figure 1: (a) Fine-tuning repeats pre-training on target data; (b) TTA adapts using a different, style-oriented task on test data.

Methodology

VoiceTTA decomposes TTA into a reward-driven RL loop constructed atop a flow matching-based TTS backbone (F5-TTS) with learnable prefix tokens. The adaptation objective is formulated as minimizing a speech distance metric, approximated through composite rewards: (i) F0 coefficient of variation (F0-CV), (ii) energy coefficient of variation (Energy-CV), (iii) speaker similarity (S-SIM), and (iv) text intelligibility (WER via Whisper model). At each TTA iteration, $k$ speech candidates are generated with controlled variation (temperature sampling), evaluated via these rewards, and the group relative preference optimization (GRPO) algorithm updates the prefix parameters.

The workflow is concretely encapsulated in (Figure 2).

Figure 2: VoiceTTA overview—GRPO optimizes learnable prefixes under combined style and intelligibility rewards in a flow matching TTS model; adapted prefixes guide subsequent speech synthesis.

Actionable Components

• Learnable Prefixes: Prefix tokens prepended to the acoustic model backbone, parameter-efficient for per-prompt adaptation.
• Flow Matching Generation: Candidates are generated by decoding latent variables with varied temperature $T$; adjustment of $T$ governs output diversity versus stability.
• Composite Reward: Style reward = $r_{\text{F0-CV}}$, $r_{\text{Energy-CV}}$, $r_{\text{S-SIM}}$; Intelligibility reward = $r_{\text{Intel}}$ (inverted WER).
• GRPO Optimization: Policy update minimizes a clipped surrogate loss, normalizing cumulative rewards per candidate, with non-KL-divergence-constrained adaptation due to prefix-only optimization.

Ablation and analysis on the number of prefixes and temperature sampling demonstrated a trade-off between adaptability, speaker similarity, and output intelligibility—see (Figure 3).

Figure 3: (a) S-SIM variation with prefix number; (b) High sampling temperature $T$ degrades intelligibility.

Experimental Evaluation

Setup

VoiceTTA is benchmarked on an in-house dataset (accented, children’s, slurred, crosstalk samples) and a Chinese dialect corpus (KeSpeech) against multiple SOTA zero-shot TTS systems (CosyVoice, MaskGCT, Vevo, F5-TTS baseline). Each test sample uses four candidates for adaptation, and only four prefixes are introduced to the first DiT layer—accounting for 16 KB per speaker in storage overhead.

Objective Results

VoiceTTA achieves a WER of 3.12 (versus baseline F5-TTS 3.19 and MaskGCT 3.26; CosyVoice and Vevo underperform notably), and attains the highest S-SIM (0.64). Crucially, enhanced stylistic imitation does not incur intelligibility loss.

Subjective Results

In human MOS evaluations, VoiceTTA surpasses competitors in style similarity (S-MOS 3.27), and achieves near-parity in naturalness. Its balance of style transfer and clarity is supported both numerically and perceptually.

Ablation and Analysis

Ablation on reward design established the necessity of combining all four rewards: style-only objectives increase S-SIM but sharply degrade WER; pure intelligibility optimization harms style. The prefix count and candidate temperature sampling must be tuned to balance domain adaptation and generalization. Over-adaptation (excessive prefixes, high $T$) can harm output quality.

Practical Implications and Future Directions

VoiceTTA presents a parameter-efficient, online-adaptable TTS solution—requiring only seconds of target speech and storing lightweight personalized states. This directly addresses personalization challenges in interactive voice agents, assistive technologies, and speech generation for underrepresented language scenarios. From a theoretical perspective, the work demonstrates the efficacy of reward-compositional, RL-driven test-time adaptation in sequence generation models beyond language modeling, and motivates future development of even more robust, self-improving generative systems capable of handling rapid, distribution-shifted adaptation in speech and multimodal scenarios.

The group relative preference optimization framework, as adapted for continuous output space (flow matching TTS vs. token generation in LLMs), provides a compelling design pattern for future RL-based adaptation protocols. Integrating more fine-grained perceptual rewards, exploring prefix-based adaptation in diffusion TTS backbones, or extending the approach to joint speech-to-speech and cross-lingual imitation tasks remain promising research directions.

Conclusion

VoiceTTA sets a new standard for lightweight, rapid, and high-fidelity zero-shot TTS adaptation under realistic distribution shifts. Its reward-driven, prefix-based, and RL-optimized TTA approach offers both theoretical novelty and practical scalability. This framework establishes a versatile foundation for continual test-time personalization in speaker- and style-diverse deployment environments.