TTS-CtrlNet: Controllable TTS via ControlNet

• The paper introduces a ControlNet-based TTS model that attaches a trainable emotion control branch to a frozen backbone, enabling time-varying emotion modulation.
• It employs zero-initialized residual injection and selective block attachment to preserve key properties like zero-shot voice cloning and naturalness.
• Empirical results demonstrate improved emotional similarity metrics and a controllable balance between expressive intensity and intelligibility.

Searching arXiv for TTS-CtrlNet and closely related controllable TTS / audio ControlNet work to ground the article. TTS-CtrlNet denotes a ControlNet-based approach to controllable flow-matching text-to-speech that targets fine-grained, time-varying emotion control while preserving the capabilities of a large pretrained zero-shot TTS backbone. In the formulation introduced in “TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet” (Jeong et al., 6 Jul 2025), the original TTS model is frozen and a trainable copy of selected blocks is attached as an auxiliary control branch, with zero-initialized coupling so that the base model’s behavior is preserved at initialization. The method is designed to add intuitive, scalable, and time-varying emotion control to an existing flow-matching TTS system while inheriting the ability of the original model, including zero-shot voice cloning and naturalness (Jeong et al., 6 Jul 2025). In a broader technical sense, the term also identifies a design pattern for controllable speech generation: structured control signals are processed by a side network and injected into a pretrained speech generator, rather than being entangled solely through end-to-end fine-tuning. Related work on description-based controllable TTS, adapter-style audio control, controllable accented TTS, dynamic speaking-rate control, and hierarchical prosody control provides adjacent formulations of the same underlying problem—how to expose interpretable control over speaker, style, prosody, accent, timing, or emotion without degrading synthesis quality (Yamamoto et al., 2024, Zhu et al., 4 Feb 2026, Xinyuan et al., 10 Aug 2025, Torgashov et al., 13 Mar 2026, Raitio et al., 2021).

1. Historical positioning and problem formulation

TTS-CtrlNet emerges from a specific limitation in controllable TTS: many systems support only utterance-level control, whereas applications such as expressive speech generation, emotional voice conversion, and speech-to-speech translation require emotion to change over time within the utterance (Jeong et al., 6 Jul 2025). The central difficulty is that emotion labels are usually utterance-level rather than frame-level, so direct supervision for temporal emotion trajectories is scarce. At the same time, zero-shot TTS must preserve speaker identity and intelligibility, and strong emotional modulation can damage text fidelity, speaker similarity, and naturalness (Jeong et al., 6 Jul 2025).

The ControlNet formulation addresses this by adding new control capability to a strong pretrained TTS model without overwriting the original model. The specific motivation is to freeze the original pretrained TTS backbone, train only an auxiliary control branch, and use zero-initialized coupling so that the added branch starts with no effect and gradually learns useful conditioning (Jeong et al., 6 Jul 2025). This differs from full fine-tuning approaches that adapt the whole model and may compromise pretrained performance. The paper explicitly contrasts this design with large-scale full-finetuning emotion-control systems and argues that ControlNet-style conditioning is much cheaper and can work with a much smaller emotional dataset (Jeong et al., 6 Jul 2025).

This framing places TTS-CtrlNet within a broader family of controllable speech systems. “Description-based Controllable Text-to-Speech with Cross-Lingual Voice Control” (Yamamoto et al., 2024) uses a separate description control model that predicts explicit timbre and style embeddings into a shared conditioning space, rather than injecting residuals into internal layers. “Audio ControlNet for Fine-Grained Audio Generation and Editing” (Zhu et al., 4 Feb 2026) freezes a pretrained text-to-audio backbone and trains ControlNet-style or adapter-style control branches for loudness, pitch, and event-roll control. “Scalable Controllable Accented TTS” (Xinyuan et al., 10 Aug 2025) uses explicit accent-token conditioning rather than a ControlNet side branch. These systems suggest that TTS-CtrlNet is one instance of a larger architectural trend: control is increasingly externalized into structured interfaces.

2. Core architecture and ControlNet mechanism

The base backbone in TTS-CtrlNet is F5-TTS, described as a flow-matching-based zero-shot TTS model using a DiT-style transformer backbone and learning an infilling task on mel spectrograms (Jeong et al., 6 Jul 2025). The input notation uses $\mathbf{x} \in \mathbb{R}^{F \times T}$ for the mel spectrogram and $y$ for the transcript. The pretrained TTS backbone receives a noisy or interpolated speech representation, a masked speech representation, and the transcript, and predicts the flow field used to transform noise into target speech (Jeong et al., 6 Jul 2025).

The system consists of two branches: the original frozen TTS branch and an emotion conditioning branch that is a trainable copy of selected parts of the original DiT network (Jeong et al., 6 Jul 2025). The emotion branch processes the same TTS input plus an additional time-varying emotion representation $\mathbf{e}$ and injects features into corresponding blocks of the original backbone. The architecture uses zero-convolution connections initialized to output a zero tensor, thereby preserving the behavior of the original model while allowing the model to gradually incorporate additional conditioning (Jeong et al., 6 Jul 2025).

The block-level fusion is written as

$$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$

and at inference as

$$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$

where $\theta$ are the frozen original TTS parameters, $\phi$ are the trainable ControlNet parameters, and $\lambda$ is the control scale (Jeong et al., 6 Jul 2025). This makes the control effect explicitly tunable at inference.

A notable design choice is selective block attachment. Rather than connecting ControlNet to all DiT blocks, the authors perform block analysis by skipping each original block during inference and measuring the effect on WER and speaker similarity. Blocks whose removal strongly hurts WER and speaker identity are excluded from ControlNet connection (Jeong et al., 6 Jul 2025). This suggests that control injection into all layers is not desirable, and that layer selectivity is a central design variable in ControlNet-style TTS.

3. Flow-matching formulation and emotion representation

The paper summarizes conditional flow matching through the flow map

$$\boldsymbol{\psi}_t(\mathbf{x}) = \sigma_t(\mathbf{x}_1) x + \mu_t(\mathbf{x}_1),$$

with boundary conditions $\mu_0(\mathbf{x}_1)=0$, $y$0, $y$1, and $y$2 (Jeong et al., 6 Jul 2025). The induced vector field is

$y$3

The conditional flow-matching loss is given as

$y$4

and the OT-CFM form as

$y$5

(Jeong et al., 6 Jul 2025).

For TTS, the model constructs a noisy sample

$y$6

with $y$7 and target mel spectrogram $y$8, and a masked speech sample

$y$9

where $\mathbf{e}$0 is a random binary temporal mask (Jeong et al., 6 Jul 2025). The ControlNet-conditioned training loss becomes

$\mathbf{e}$1

(Jeong et al., 6 Jul 2025).

The emotion control signal is not a discrete emotion label. It is a continuous time-varying arousal-valence trajectory extracted using a wav2vec-based speech emotion recognition model from Wagner et al., with dominance discarded (Jeong et al., 6 Jul 2025). Emotion features are represented as

$\mathbf{e}$2

and then projected by a $\mathbf{e}$3 convolution to match the TTS feature dimension (Jeong et al., 6 Jul 2025). The alignment procedure feeds audio into the wav2vec encoder, obtains token-level features proportional to audio duration, applies an emotion regression head, and interpolates these features using an emotion window size $\mathbf{e}$4 to produce a sequence aligned with the TTS temporal axis (Jeong et al., 6 Jul 2025).

This continuous representation is one of the defining characteristics of TTS-CtrlNet. The control mechanism is not a categorical emotion tag but a smoothed temporal affect trajectory. That property distinguishes it from token-conditioned accent control (Xinyuan et al., 10 Aug 2025), language-description embedding control (Yamamoto et al., 2024), and symbolic pronunciation control schemes such as hybrid phoneme-text or character-pinyin interfaces (Cui et al., 16 Dec 2025, Deng et al., 8 Feb 2025).

4. Inference-time control strategies

TTS-CtrlNet introduces three practical inference controls: selective block attachment, emotion-specific flow-step control, and flexible control scale (Jeong et al., 6 Jul 2025). The most distinctive is the emotion-specific flow step. Rather than applying the control branch throughout the full flow interval $\mathbf{e}$5, the method uses only early timesteps,

$\mathbf{e}$6

with experiments centered on

$\mathbf{e}$7

(Jeong et al., 6 Jul 2025). For later flow steps, the model sets

$\mathbf{e}$8

so ControlNet contributes only during early ODE integration (Jeong et al., 6 Jul 2025). The rationale is that very late flow steps are near the final speech sample and perturbations have little effect, whereas there exists an early interval in which emotion is most sensitive and controllable.

The inference pipeline requires reference audio $\mathbf{e}$9, reference mel spectrogram $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$0, reference transcript $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$1, target text $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$2, initial noise $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$3, and emotion sequence $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$4 (Jeong et al., 6 Jul 2025). The combined text input is

$$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$5

The original branch takes $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$6, while the ControlNet branch takes $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$7 (Jeong et al., 6 Jul 2025). In the demonstrated setup, time-varying emotion is therefore specified implicitly through a reference audio prompt whose emotion trajectory is extracted and transferred.

The control scale $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$8 is a second key interface. It adjusts the strength of ControlNet injection at inference and therefore the trade-off between emotional intensity and intelligibility (Jeong et al., 6 Jul 2025). This suggests that TTS-CtrlNet functions not only as a conditioning mechanism but also as a calibrated intervention mechanism. A plausible implication is that control scale in TTS-CtrlNet serves a similar role to control strength parameters in other controllable generation systems, such as the duration-distribution control strength $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$9 in VoXtream2’s speaking-rate control (Torgashov et al., 13 Mar 2026).

5. Empirical performance and ablation results

The main objective metrics in TTS-CtrlNet are WER, SIM-o, AutoPCP, Emo-SIM, and Aro-Val SIM; subjective metrics are SMOS, NMOS, and EMOS (Jeong et al., 6 Jul 2025). On JVNV S2ST, TTS-CtrlNet reports SIM-o 0.464, WER 5.4, AutoPCP 2.36, Emo-SIM 0.751, and Aro-Val SIM 0.742, outperforming the compared systems on the emotional metrics (Jeong et al., 6 Jul 2025). On EMO-Change, it reports SIM-o 0.589, WER 0.6, AutoPCP 3.68, Emo-SIM 0.724, and Aro-Val SIM 0.864, again yielding the best emotion metrics among the compared systems while maintaining competitive intelligibility (Jeong et al., 6 Jul 2025).

The ablations are central to the method’s interpretation. Full ControlNet attachment yields SIM-o 0.630, WER 8.9, Emo-SIM 0.450, and Aro-Val SIM 0.746, whereas selective block attachment yields SIM-o 0.684, WER 0, Emo-SIM 0.565, and Aro-Val SIM 0.872 (Jeong et al., 6 Jul 2025). This shows that injection location matters more than simply increasing the amount of control. Training on the full flow interval $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$0 yields WER 0, Emo-SIM 0.389, and Aro-Val SIM 0.674, whereas using $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$1 yields WER 1.9, Emo-SIM 0.565, and Aro-Val SIM 0.876 (Jeong et al., 6 Jul 2025). The emotion-window ablation compares window size 1 with window size 30, improving from WER 4.7, Emo-SIM 0.500, Aro-Val SIM 0.836 to WER 1.9, Emo-SIM 0.565, Aro-Val SIM 0.876 (Jeong et al., 6 Jul 2025).

The control-scale study further illustrates the trade-off. At $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$2, the model reports WER 2.9, Emo-SIM 0.692, Aro-Val SIM 0.845, AutoPCP 3.62; at $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$3, WER 0.63, Emo-SIM 0.724, Aro-Val SIM 0.864, AutoPCP 3.68; and at $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$4, WER 9.58, Emo-SIM 0.742, Aro-Val SIM 0.892, AutoPCP 3.67 (Jeong et al., 6 Jul 2025). The reported pattern is that larger $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$5 improves emotional similarity but raises WER. This makes flexible control scale a practical mechanism for balancing intelligibility and emotion.

The subjective MOS results also support the claim that the ControlNet branch adds emotion alignment while largely preserving the base model’s quality. On JVNV S2ST, F5-TTS yields SMOS 2.0, NMOS 2.3, EMOS 2.6, whereas TTS-CtrlNet yields SMOS 2.7, NMOS 3.2, EMOS 3.4. On EMO-Change, F5-TTS yields SMOS 3.7, NMOS 3.8, EMOS 2.0, while TTS-CtrlNet yields SMOS 3.6, NMOS 3.8, EMOS 3.3 (Jeong et al., 6 Jul 2025).

6. Relation to adjacent controllable TTS paradigms

TTS-CtrlNet is one architectural answer to controllable TTS, but not the only one. A useful comparison is with description-conditioned control. In (Yamamoto et al., 2024), a description control model maps natural-language descriptions into a shared SSL-derived timbre/style embedding space used by the TTS acoustic model and NANSY++ synthesizer. The description encoder is a pretrained BERT base uncased model, and the controller predicts timbre, duration-style, linguistic-style, and pitch-style embeddings through separate MDNs (Yamamoto et al., 2024). The control path is thus explicit and interpretable: description $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$6 predicted timbre/style embeddings $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$7 conditioned duration/linguistic/pitch generation $$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$8 NANSY++ synthesis. Relative to TTS-CtrlNet, this is a latent-variable controller rather than a residual feature-injection controller.

Another nearby design is adapter-based audio control. In (Zhu et al., 4 Feb 2026), T2A-ControlNet replicates backbone layers and injects residuals, whereas T2A-Adapter uses a lightweight condition encoder with cross-attention plus zero-convolution residual addition into a frozen FluxAudio backbone. The T2A-Adapter formulation

$$\mathbf{Z}_\text{new} = \mathcal{F}_{k}(\cdot\, ; \, \theta) + \lambda \cdot \mathbf{m} \odot \mathcal{F}_{k}(\cdot\, ; \, \phi),$$9

is architecturally close to TTS-CtrlNet’s residual coupling (Zhu et al., 4 Feb 2026). The paper’s results show that the lighter adapter design can be more parameter efficient and less sensitive to depth. This suggests that ControlNet-style controllable TTS need not require a full copied branch; lightweight cross-attentive controllers may be an alternative.

Accent control introduces a different control interface. “Scalable Controllable Accented TTS” (Xinyuan et al., 10 Aug 2025) replaces the language token in XTTS-v2 with an accent token, making accent an explicit discrete conditioning variable. That system is controllable, but it is not a ControlNet-style architecture; there is no separate accent-control branch and no intermediate-layer injection (Xinyuan et al., 10 Aug 2025). Likewise, GLM-TTS exposes pronunciation control through a hybrid phoneme-text input scheme and speaker customization through LoRA-based adaptation, while IndexTTS uses hybrid character-pinyin modeling and a Conformer-based speech conditioning encoder (Cui et al., 16 Dec 2025, Deng et al., 8 Feb 2025). These approaches show that controllability can also be implemented through symbolic input design and parameter-efficient adaptation rather than ControlNet residual injection.

Finally, TTS-CtrlNet should be distinguished from classical hierarchical prosody control. The system in (Raitio et al., 2021) uses an utterance-level variance adaptor to predict pitch, pitch range, duration, energy, and spectral tilt, which are then used to condition phone-level predictors. This is a semantically factorized control hierarchy with explicit acoustic dimensions. TTS-CtrlNet instead uses a continuous emotion trajectory extracted from SER and relies on a ControlNet branch to modulate the backbone. A plausible interpretation is that TTS-CtrlNet prioritizes temporal affect transfer, whereas hierarchical prosody modeling prioritizes interpretable low-dimensional prosodic factors.

7. Limitations, controversies, and broader significance

The main stated limitation of TTS-CtrlNet is dependence on the SER model. The current emotion encoder cannot reliably capture laughter, crying, and other non-verbal emotional expressions (Jeong et al., 6 Jul 2025). The system is therefore strong for continuous emotional prosody transfer but weaker for non-verbal expressive events. The paper also notes that increasing control strength can hurt WER, that stronger emotional modulation may reduce phoneme-level precision, that the F5-TTS backbone has artifacts on some high-pitch speech and TTS-CtrlNet inherits some of these, and that cross-lingual Japanese-input cases can produce atypical English pronunciation because the backbone was only trained on English and Chinese (Jeong et al., 6 Jul 2025).

More generally, ControlNet-style TTS raises a recurring question: whether control should be injected as dense residual features into a frozen backbone, or exposed through explicit latent variables or symbolic tokens. The evidence across the cited work is mixed. TTS-CtrlNet demonstrates that residual ControlNet injection can improve time-varying emotion similarity while preserving zero-shot capabilities (Jeong et al., 6 Jul 2025). The description-based cross-lingual system argues for explicit shared embedding spaces (Yamamoto et al., 2024). Audio ControlNet work suggests lightweight adapters may be preferable to large copied branches (Zhu et al., 4 Feb 2026). Accent-controllable and pronunciation-controllable systems show that simple discrete control tokens or mixed symbolic inputs can be highly effective for specific attributes (Xinyuan et al., 10 Aug 2025, Deng et al., 8 Feb 2025, Cui et al., 16 Dec 2025).

This suggests that “TTS-CtrlNet” can be understood in two senses. In the strict sense, it refers to the ControlNet-based flow-matching TTS architecture of (Jeong et al., 6 Jul 2025). In the broader editorial sense, it refers to a research direction in which pretrained TTS backbones are augmented by modular control pathways—residual, adapter, latent, or symbolic—to achieve interpretable and efficient control over time-varying speech attributes. Within that broader landscape, the specific contribution of TTS-CtrlNet is to establish that a frozen flow-matching TTS model can be retrofitted with a trainable ControlNet branch for temporally aligned emotion transfer, with selective block design, early-flow control, and inference-time control scaling as the central practical recipes (Jeong et al., 6 Jul 2025).