ReStyle-TTS: Fine-Grained Style Control

• ReStyle-TTS is a text-to-speech framework that provides structured, fine-grained, and disentangled style control using both discrete and continuous representations.
• It leverages two-stage transformer architectures and diffusion-based methods to independently control prosody, emotion, and timbre while preserving intelligibility and speaker fidelity.
• The system uses techniques like decoupled guidance and LoRA fusion for multi-attribute manipulation, enabling practical, reference-relative adjustments in stylistic features.

ReStyle-TTS denotes a class of text-to-speech (TTS) systems that provide structured, fine-grained, and disentangled style control over generated speech. The term encompasses two major system lines in recent literature: (1) ReStyle-TTS based on masked-autoencoded discrete style-rich representations with two-stage transformer architectures for multi-attribute controllability (Wang et al., 3 Jun 2025), and (2) ReStyle-TTS for relative and continuous style control in zero-shot speech synthesis, enabling reference-relative and multi-attribute manipulation via decoupled guidance and orthogonal parameter fusion (Li et al., 7 Jan 2026). Both variants emphasize practical user control over prosody, emotion, and timbre, while preserving intelligibility and speaker identity.

1. Motivation and Challenges

Standard zero-shot TTS systems, such as VALL-E and Voicebox, can adapt to arbitrarily sampled speaker timbres from a short reference utterance. However, these approaches inherit not only timbre but also the prosody, emotion, and other speaking style features from the reference audio. Consequently, style specification in “in-the-wild” or low-data regimes is not practical, as acquiring reference samples that match the target style for a specific speaker is often intractable. While traditional controllable TTS systems employ fixed, absolute style prompts—either text-based or audio-based—they are typically limited to a finite set of style attributes, and only admit coarse-grained or discrete control (Li et al., 7 Jan 2026). Addressing these limitations requires (i) reducing the implicit entanglement between reference guidance and output style, and (ii) introducing mechanisms for multi-dimensional, continuous, and reference-relative style specification.

2. System Architectures

There are two principal architectural paradigms for ReStyle-TTS:

A. Two-Stage LM-Driven Discrete Style Control

The first paradigm utilizes a two-stage, language-model-based generation pipeline (Wang et al., 3 Jun 2025):

• Stage 1: An autoregressive transformer (“Style LM”) generates a sequence of discrete, style-rich tokens $s$ from the phoneme sequence and any style-control signals (discrete attribute bins or a continuous speaker embedding).
• Stage 2: A second autoregressive transformer (“Acoustic LM”) generates codec tokens $a$ from the same phoneme sequence and the style tokens $s$. The tokens $a$ are then decoded via a neural codec (such as EnCodec) into waveform samples.

Data flow:

$$\text{Text} \rightarrow \text{Phoneme sequence } \tau \rightarrow \text{Style LM (with control)} \rightarrow s \rightarrow \text{Acoustic LM} \rightarrow a \rightarrow \text{Codec Decoder} \rightarrow \text{Waveform}$$

Control signals (attribute labels, speaker embedding) are incorporated as “prefix tokens” to the phoneme sequence for the style LM. Stage 2 takes no explicit control input, as style is encoded in $s$.

B. Diffusion-Based Reference-Relative Continuous Style Control

A complementary paradigm enables relative, continuous, and disentangled style control (Li et al., 7 Jan 2026). It integrates:

• Diffusion TTS backbone fine-tuned for style control.
• Decoupled Classifier-Free Guidance (DCFG): Text and reference guidance are modulated independently to balance text fidelity and reference-style inheritance.
• Style-specific LoRA modules with Orthogonal LoRA Fusion (OLoRA): Low-rank adapters are fine-tuned for individual style attributes; at inference, multiple adapters are fused orthogonally and each scaled continuously ($\alpha_i$) for user-specified control strength.
• Timbre Consistency Optimization (TCO): A reward-based scheme ensures robust timbre maintenance as reference influence is reduced.

3. Discrete and Continuous Style Representations

Discrete Masked-Autoencoded Style Units

The two-stage LM system constructs a representation space via a masked autoencoder (MAE) trained to reconstruct masked mel-filterbank frames from both audio and temporally aligned phoneme embeddings. The style encoder’s output is then quantized using a three-stage residual vector quantizer (RVQ). Per-phoneme, quantized “style tokens” serve as intermediaries for downstream LMs, permitting robust control and style transfer in discrete bins (e.g., pitch, energy, arousal, dominance, valence, SNR, C50) (Wang et al., 3 Jun 2025).

The total MAE loss is: $$\mathcal{L}_{\mathrm{MAE}} = \lambda_r \mathcal{L}_r + \lambda_c \mathcal{L}_c + \lambda_p \mathcal{L}_p + \lambda_e \mathcal{L}_e$$ with $\lambda_r=10$ and $\lambda_c=\lambda_p=\lambda_e=1$, integrating reconstruction, contrastive, pitch, and energy classification objectives.

Relative and Continuous Style Axes

The diffusion-based ReStyle-TTS line admits free movement along continuous, reference-relative style scales. Users specify increments in pitch, energy, or emotion as standardized deviations from the reference, such as "+1σ pitch, –0.5σ energy, neutral→happy" (see Section 7 usage example in (Li et al., 7 Jan 2026)). LoRA adapters fine-tuned on style-annotated datasets are fused orthogonally at inference, with individual intensities scaled continuously for disentangled, multi-attribute control.

4. Control Mechanisms and Guidance

Prefix Control (Two-Stage LM): Discrete attribute labels and/or continuous speaker embeddings are embedded and prepended to the phoneme sequence, allowing for synthesis strategies such as "emotion control on a given speaker" or "novel speaker synthesis with specified timbral characteristics." Fine attribute binning and classifier-free guidance (CFG) are leveraged for sharper adherence to user specifications (Wang et al., 3 Jun 2025).

Decoupled Guidance (Diffusion): The DCFG mechanism explicitly separates the effects of text and reference conditions within the diffusion denoising step: $$\hat{\epsilon}_\theta(x_t | c_t, s_r) = \epsilon_\theta(x_t | c_t, s_r) + w_\text{text} \left[ \epsilon_\theta(x_t | c_t, s_r) - \epsilon_\theta(x_t | s_r) \right] + w_\text{ref} \left[ \epsilon_\theta(x_t | c_t, s_r) - \epsilon_\theta(x_t | c_t) \right]$$ Here, $w_{\text{text}}$ accentuates or attenuates text prompt adherence, and $w_{\text{ref}}$ scales reference-style inheritance independently.

Method	Guidance Mechanism	Control Granularity
Two-Stage LM	Prefix tokens + CFG	Fine, discrete bins
Diffusion + OLoRA	DCFG ($w_\text{text}$, $w_\text{ref}$) + LoRA scale $\alpha_i$	Continuous, relative

5. Timbre Preservation and Training Methodology

Weakening the reference influence (lower $w_\text{ref}$) can degrade speaker timbre. To address this, the TCO scheme employs reward-weighted flow-matching loss, leveraging a pretrained speaker-verification model to measure similarity:

• Calculate speaker similarity $r_t$ for a synthesized sample.
• Maintain a baseline $b_t = \mu b_{t-1} + (1-\mu) r_t$.
• Compute advantage $A_t = r_t - b_t$, and weight the regression loss: $w_t = 1 + \lambda \tanh(\beta A_t)$

$$\mathcal{L}_\text{total}(\theta) = \mathbb{E}[ w_t \| f_\theta(x) - y \|_2^2 ]$$

This incentivizes high-fidelity timbre transfer even as style conditioning is modified (Li et al., 7 Jan 2026).

Comprehensive multi-dataset pretraining (e.g., GigaSpeech-xl, LibriTTS for LM-based models (Wang et al., 3 Jun 2025); LibriTTS and emotion-labeled corpora for LoRA-based models (Li et al., 7 Jan 2026)) is employed to strengthen style disentanglement and control generalization.

6. Evaluation and Empirical Results

Objective Metrics:

• Text intelligibility: WER, as measured by Whisper-large-v3.
• Speaker similarity: Cosine similarity via pretrained models (wavLM-sv).
• Attribute transfer/control: Fraction of attribute bins matched; for continuous control, measured by regression slope and intercept (relative style shift).
• Naturalness: UTMOS or MOS-Q.
• Style accuracy: Classification via Emotion2Vec or expert rater Likert scores.

Experimental Findings:

• Two-stage models with large-scale Stage 1 training achieve superior control accuracy for fine-grained attributes and comparable or better naturalness compared to one-stage or zero-shot competitors (e.g., YourTTS, XTTS-V2) (Wang et al., 3 Jun 2025).
• Diffusion-based ReStyle-TTS produces smooth, monotonic control curves over $\alpha \in [-4, +4]$ for pitch, energy, and emotion, with stability in both WER and speaker similarity under strong manipulation (Li et al., 7 Jan 2026).
• Multi-attribute and reference-relative control reliably induces independent perceptual changes along each attribute axis without degrading intelligibility or timbre. Contradictory-style and mismatched-reference evaluation shows ReStyle-TTS outperforming alternative frameworks in both emotion accuracy ($>$85%) and prosody transfer accuracy ($>$92%).

Ablation studies confirm that DCFG is essential for continuous style control; removing TCO degrades speaker similarity by more than 0.08 (measured by Spk-sv) (Li et al., 7 Jan 2026).

7. Practical Usage and Significance

ReStyle-TTS frameworks allow users to synthesize speech with arbitrary speaker references and explicit, multi-dimensional style control. Example usage in the diffusion paradigm involves:

• Weakening reference guidance ($w_\text{ref}$) and increasing text guidance ($w_\text{text}$) to decouple inherited style.
• Specifying style strengths for LoRA adapters (e.g., $\alpha_\text{pitch} = +1.0$, $\alpha_\text{energy} = -0.5$, $\alpha_\text{happy} = +2.0$).
• System orthogonally fuses the style controls and generates speech that reflects the specified relative and continuous style shifts, preserving intelligibility and timbral identity (Li et al., 7 Jan 2026).

This approach enables a flexible, user-driven interface, facilitating research and practical applications that require nuanced stylistic variation within TTS while maintaining speaker fidelity and content accuracy. ReStyle-TTS provides a methodology for achieving fine-grained style manipulation in both discrete and continuous spaces, with demonstrated robustness across mismatched references and complex, multi-attribute settings (Wang et al., 3 Jun 2025, Li et al., 7 Jan 2026).