WeSCon: Word-Level Expressive TTS

• WeSCon is a two-stage self-training framework that delivers word-level control over both emotion and speaking rate in zero-shot TTS using pseudo-labeled data.
• It builds upon CosyVoice2 by incorporating a teacher stage with multi-round inference, transition smoothing, and a student stage with dynamic emotional attention bias.
• The framework preserves original zero-shot synthesis quality while enhancing expressivity through fine-grained control, all with small public datasets and no intra-sentence annotations.

Searching arXiv for the target paper and a few directly mentioned related works to ground citations. {"^^^^^^^^3queries3 Emotional Expression Control in Zero-Shot Text-to-Speech Synthesis\"","max_results":5},{"^^^^^^^^3query3 zero-shot TTS arXiv","max_results":5},{"^^^^^^^^3query3 arXiv","max_results":5},{"^^^^^^^^3query3 arXiv","max_results":5}]} Here are the search results from arXiv. WeSCon is a two-stage self-training framework for zero-shot text-to-speech synthesis that equips a pretrained model with word-level control of both emotion and speaking rate while preserving the original zero-shot synthesis capability and avoiding any requirement for datasets with intra-sentence emotion or speed annotations. It is built on top of CosyVoice^^^^3ti:\3^^^^ and addresses a specific limitation of prevailing emotional TTS systems: most provide utterance-level control, in which a single emotion label or reference speech governs the entire utterance, rather than permitting intra-sentence plans such as assigning different emotions and tempos to different words or short phrases (&&&^^^^3queries3^^^^&&&).

^^^^3query3^^^^. Problem formulation and scope

The central problem addressed by WeSCon is fine-grained expressive control in emotional TTS under severe data constraints. Public emotional corpora such as ESD, IEMOCAP, and CREMA-D provide mainly utterance-level labels rather than word-level or frame-level emotion trajectories, whereas corpora with annotated intra-sentence emotion transitions are described as rare and largely private. Manual word- or frame-level emotion labeling is further characterized as expensive and subjective. In parallel, modeling multi-emotion transitions is intrinsically difficult because the model must determine which prompt to attend to for which word while maintaining smooth acoustic continuity and consistent speaker identity across segment boundaries.

Within this setting, WeSCon targets two dynamic dimensions of prosody and expressivity: emotion and speaking rate. The paper explicitly distinguishes this objective from conventional zero-shot TTS, which can clone speaker identity, global emotion, and average speaking rate, but does not allow a user to specify a sequence such as one word being angry and slow, the next surprised and fast, and a subsequent return to neutral. The stated goal is to enable word-level control of emotion and speaking rate in a pretrained zero-shot TTS model without any training data containing explicit intra-sentence emotion or speed transitions and without sacrificing the original zero-shot TTS capabilities.

This formulation also clarifies what WeSCon is not. It is not a framework that learns from naturally annotated intra-sentence emotional speech. It is instead a method for synthesizing and then distilling such behavior from model-generated pseudo-labels. A common misconception would be that word-level emotional control necessarily depends on dense human annotations; the reported framework is designed precisely to avoid that dependency.

^^^^3ti:\3^^^^. Architectural basis and interface to CosyVoice^^^^3ti:\3^^^^

WeSCon is built on CosyVoice^^^^3ti:\3^^^^, described as a zero-shot TTS model based on semantic speech tokens, a neural codec LLM, a flow-matching reconstructor, and a vocoder. CosyVoice^^^^3ti:\3^^^^ uses a supervised speech tokenizer to convert waveform into discrete speech tokens, and a Transformer LLM takes mixed text and speech tokens in the autoregressive sequence

PRESERVED_PLACEHOLDER_^^^^3queries3^^^^

where PRESERVED_PLACEHOLDER_^^^^3query3^^^^ denotes text tokens, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^ denotes speech tokens, $\circled{S}$ is the start-of-text symbol, and $\circled{B}$ is the boundary or start-of-speech symbol. In this decomposition, the LLM focuses on semantic content, coarse prosody, and emotion, whereas the flow-matching module converts speech tokens plus speaker embedding into mel-spectrograms and primarily controls speaker identity.

WeSCon interfaces with this backbone differently in its two stages. In the teacher stage, CosyVoice^^^^3ti:\3^^^^ is kept frozen and used in its original form. A non-causal content aligner is attached on top of the language-model outputs to map speech tokens to text positions and detect content boundaries. In the student stage, the input format is extended with explicit emotion tokens $E^i$ that mark emotional segments, and a dynamic emotional attention bias module is inserted into each Transformer layer. The LLM and flow-matching modules remain initially pretrained as in CosyVoice^^^^3ti:\3^^^^; the additions introduced by WeSCon are the content aligner in the teacher stage and the emotion aligner plus dynamic emotional attention bias in the student stage (&&&^^^^3queries3^^^^&&&).

This design is significant because it localizes the new control mechanisms around prompt conditioning, alignment, and attention modulation rather than replacing the backbone. The reported consequence is that the original zero-shot synthesis behavior can be preserved while adding word-level expressivity.

3. Stage-^^^^3query3^^^^ teacher: multi-round inference, transition smoothing, and dynamic speed control

The first stage constructs a teacher through multi-round inference. The input text is partitioned according to a user-specified word-level emotion and speed plan. For each segment, the system selects an emotional speech prompt of the desired emotion from a small corpus and synthesizes the segment with that prompt. Because naive concatenate-and-generate strategies introduce abrupt transitions, WeSCon adds a content aligner and a transition-smoothing mechanism.

The content aligner is specified as 5 non-causal Transformer layers plus two $5 \times 5$ convolution layers with stride ^^^^3query3^^^^ and batch normalization. It is trained on approximately ^^^^3ti:\3queries3queries3^^^^ hours of ASR-type data from LibriSpeech-^^^^3query3queries3queries3^^^^-clean and AISHELL-^^^^3query3^^^^, with no emotion labels. Its targets are token-level text index labels, indicating which text token each speech token corresponds to, and binary boundary labels indicating content boundaries. During multi-round inference, the final text and speech tokens from round $i-1$ are appended as a tail to the prompt sequence for round $i$. This tail-to-head linkage allows the LLM to observe previous context and continue naturally, consistent with continuation-style generation in neural codec LLMs. The aligner supplies the mapping needed to choose a tail segment aligned with the overlapping text region. The paper emphasizes that smoothing is not performed by an explicit formula over emotion embeddings; rather, it is done implicitly at the sequence level by overlapping content and letting the LLM generate a consistent continuation.

Speaking-rate control is introduced through a dynamic speed control mechanism operating on prompt speech tokens. If $\alpha$ is the resampling factor, then PRESERVED_PLACEHOLDER_^^^^3query3queries3^^^^ leaves speed unchanged, PRESERVED_PLACEHOLDER_^^^^3query3query3^^^^ interpolates to length PRESERVED_PLACEHOLDER_^^^^3query3ti:\3^^^^ and approximates slower speech, and PRESERVED_PLACEHOLDER_^^^^3query33^^^^ downsamples to length PRESERVED_PLACEHOLDER_^^^^3query3^^^^4 and approximates faster speech. Appendix B is reported to show that stable control is achievable when token length is between 5^^^^3queries3^^^^% and ^^^^3ti:\3queries3queries3^^^^% of the original. At the word level, each text segment in the control plan is assigned a speed factor, and the corresponding prompt tokens are resampled accordingly before multi-round generation.

The Stage-^^^^3query3^^^^ inference pipeline is explicitly organized as user specification, prompt selection, dynamic speed control, segment-wise generation with tail-to-head linkage, and concatenation plus reconstruction. For the first segment, the model generates from PRESERVED_PLACEHOLDER_^^^^3query3^^^^5. For later segments, it appends the aligned tail of the previous output and generates the next segment conditioned on accumulated text context. The resulting speech tokens are concatenated, passed through flow-matching with a target speaker embedding, and then vocoded into waveform (&&&^^^^3queries3^^^^&&&).

This stage is described as sufficient to produce synthetic utterances with word-level emotion and speed variation despite the absence of training data with such transitions. It is also the source of the pseudo-labeled data used later in self-training.

4. Stage-^^^^3ti:\3^^^^ student: self-training and dynamic emotional attention bias

The second stage reduces the teacher’s inference complexity by distilling its behavior into a student model that performs word-level expressive synthesis in a single forward pass. The process begins with emotion-transition text generation: GPT-4o is prompted to produce single sentences segmented into ^^^^3ti:\3^^^^–4 sub-spans, each annotated with an "emotion" drawn from discrete categories such as happy, sad, angry, surprise, and neutral, and a "speed" given as a continuous value between ^^^^3queries3^^^^.5 and ^^^^3ti:\3^^^^.^^^^3queries3 The Stage-^^^^3query3^^^^ teacher then synthesizes pseudo-labeled speech from these sentences using emotional prompts from the ESD training set.

The pseudo-labeled data are filtered before training. For each synthesized utterance, the system computes ASR CER or WER using SenseVoice for ASR, speaker similarity using Resemblyzer embeddings, and emotion similarity using a Whisper-based SER model. These quantities are normalized and aggregated into a composite score, and the top 5^^^^3queries3^^^^% of sample pairs are retained as self-training data. The original CosyVoice^^^^3ti:\3^^^^ is then treated as the student and fine-tuned end-to-end with a small learning rate.

Two architectural additions define this student stage. The first is explicit emotion indicator tokens inserted into the text stream to mark segment-level emotion. The second is the dynamic emotional attention bias mechanism. The student input remains compatible with the original CosyVoice^^^^3ti:\3^^^^ pattern, but it is augmented with emotion indicators preceding each emotional prompt text. A lightweight causal Transformer, termed the emotion aligner, predicts a token-level emotion label for each target speech token. These predicted labels drive the dynamic emotional attention bias.

At each Transformer layer, the current text-speech representation is concatenated with predicted emotion features and projected; one path forms a residual representation, and the other passes through an MLP and softmax to produce weights PRESERVED_PLACEHOLDER_^^^^3query3^^^^6. There are seven predefined attention bias templates PRESERVED_PLACEHOLDER_^^^^3query3^^^^7, representing typical attention patterns including standard causal language-model attention, strict emotion-aligned attention, full text history with emotion-aligned speech, and additional cross-prompt and self-attention configurations. The dynamic bias is then

PRESERVED_PLACEHOLDER_^^^^3query3^^^^8

Scaled dot-product attention is modified by reweighting the attention map with PRESERVED_PLACEHOLDER_^^^^3query3^^^^9, thereby emphasizing regions corresponding to emotion-consistent prompts and down-weighting emotion-inconsistent prompt areas. The student is optimized with a speech-token prediction loss and an emotion-prediction loss, with the LLM and emotion aligner fine-tuned while the flow-matching and vocoder modules remain unchanged (&&&^^^^3queries3^^^^&&&).

The conceptual shift from Stage ^^^^3query3^^^^ to Stage ^^^^3ti:\3^^^^ is important. The teacher achieves control through explicit multi-round generation and stitched continuations; the student internalizes that control through pseudo-supervision, explicit emotion tokens, and layer-wise attention modulation.

5. Data regime, evaluation protocol, and empirical findings

A defining property of WeSCon is its reliance on small-scale and public datasets. For Stage ^^^^3query3^^^^, aligner training uses ^^^^3ti:\3queries3queries3^^^^ hours total from LibriSpeech-^^^^3query3queries3queries3^^^^-clean and AISHELL-^^^^3query3^^^^, both non-emotional and without intra-sentence emotion transitions. For Stage ^^^^3ti:\3^^^^, emotional prompts come from ESD, characterized as ^^^^3ti:\3queries3^^^^ speakers and 5 emotions with utterance-level labels only. GPT-4o supplies emotion- and speed-segmented text, and the teacher synthesizes pseudo-labeled emotion-transition audio. No dataset with natural intra-sentence transitions or word-level emotion annotations is used.

The evaluation setup uses four zero-shot TTS baselines with multi-round concatenative inference: Index-TTS, F5-TTS, Spark-TTS, and the CosyVoice^^^^3ti:\3^^^^ backbone. Objective metrics comprise intelligibility, measured by WER with Whisper-Large for English and CER with Paraformer for Chinese; speaker similarity, measured as S-SIM via WavLM-Large cosine similarity; transition smoothness, measured by DNSV, the variance of DNSMOS-PRO over time, where lower is smoother; prosody alignment, measured by AutoPCP; and emotion similarity, measured by Emo^^^^3ti:\3^^^^v using emotion^^^^3ti:\3^^^^vec-Large embeddings and by arousal similarity using a wav^^^^3ti:\3^^^^vec ^^^^3ti:\3^^^^.^^^^3queries3 model. Subjective evaluation uses ^^^^3query3^^^^5 expert listeners and reports EMOS for emotion match, SPMOS for speed match, SMOS for speaker similarity, and NMOS for naturalness of emotion transitions, all on a ^^^^3query3^^^^–5 scale with 95% confidence intervals.

The reported main results show consistent gains over CosyVoice^^^^3ti:\3^^^^ and over the first-stage teacher. For emotion similarity, English Emo^^^^3ti:\3^^^^v improves from ^^^^3queries3^^^^.866 for CosyVoice^^^^3ti:\3^^^^ to ^^^^3queries3^^^^.879 for WeSCon(^^^^3query3^^^^st) and ^^^^3queries3^^^^.88^^^^3ti:\3^^^^ for WeSCon(^^^^3ti:\3^^^^nd), while Chinese Emo^^^^3ti:\3^^^^v improves from ^^^^3queries3^^^^.843 to ^^^^3queries3^^^^.866 and then ^^^^3queries3^^^^.87^^^^3ti:\3 For arousal similarity, English rises from ^^^^3queries3^^^^.446 to ^^^^3queries3^^^^.463 and ^^^^3queries3^^^^.468, and Chinese from ^^^^3queries3^^^^.537 to ^^^^3queries3^^^^.55^^^^3query3^^^^ and ^^^^3queries3^^^^.556. For transition smoothness, English DNSV decreases from 7.894 to 4.577 and then 4.36^^^^3query3^^^^, and Chinese from 7.6^^^^3query3ti:\3^^^^ to 4.98^^^^3queries3^^^^ and then 4.^^^^3ti:\3query3queries3^^^^. Chinese AutoPCP rises from ^^^^3ti:\3^^^^.5^^^^3query3 for CosyVoice^^^^3ti:\3^^^^ to ^^^^3ti:\3^^^^.65^^^^3queries3^^^^ for WeSCon(^^^^3query3^^^^st) and ^^^^3ti:\3^^^^.663 for WeSCon(^^^^3ti:\3^^^^nd). WER and CER are described as remaining comparable to CosyVoice^^^^3ti:\3^^^^ and acceptable for all models. In subjective MOS, the second-stage model reports EMOS of PRESERVED_PLACEHOLDER_^^^^3ti:\3queries3^^^^, SPMOS of PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^, SMOS of PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^, and NMOS of PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^, with NMOS substantially above CosyVoice^^^^3ti:\3^^^^’s PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4 and the other baselines below 3.^^^^3queries3^^^^ (&&&^^^^3queries3^^^^&&&).

The ablation studies further isolate the role of each component. On the Chinese test set, removing transition smoothing raises DNSV from 4.98^^^^3queries3^^^^ to 7.568 and lowers Emo^^^^3ti:\3^^^^v from ^^^^3queries3^^^^.866 to ^^^^3queries3^^^^.85^^^^3query3^^^^ and Aro. from ^^^^3queries3^^^^.55^^^^3query3^^^^ to ^^^^3queries3^^^^.53^^^^3query3 while also reducing speaker similarity and AutoPCP. Removing speed control lowers AutoPCP from ^^^^3ti:\3^^^^.65^^^^3queries3^^^^ to ^^^^3ti:\3^^^^.499 and degrades emotion metrics. Removing attention bias in the student increases DNSV from 4.^^^^3ti:\3query3queries3^^^^ to 5.534 and lowers Emo^^^^3ti:\3^^^^v from ^^^^3queries3^^^^.87^^^^3ti:\3^^^^ to ^^^^3queries3^^^^.837 and Aro. from ^^^^3queries3^^^^.556 to ^^^^3queries3^^^^.5^^^^3query3 Removing emotion flags or using the naive input format is reported to cause marked degradation, with CER reaching 4.^^^^3query3max_results3query3^^^^ rather than ^^^^3ti:\3^^^^.^^^^3query3ti:\3ti:\3^^^^ and emotion similarity dropping from ^^^^3queries3^^^^.87^^^^3ti:\3^^^^ to ^^^^3queries3^^^^.8^^^^3query3 Varying self-training data size improves performance up to approximately 5^^^^3queries3queries3^^^^ hours of synthetic data, after which slight degradation is attributed to overfitting to the limited emotion and speaker diversity of ESD. Out-of-domain CASIA experiments are reported to show similar patterns.

6. Zero-shot preservation, limitations, and place in the literature

A central claim of WeSCon is that added controllability does not materially damage zero-shot TTS performance. On SEED test-zh, CosyVoice^^^^3ti:\3^^^^ reports CER = ^^^^3query3^^^^.45 and S-SIM = ^^^^3queries3^^^^.748. WeSCon(^^^^3query3^^^^st), with the backbone frozen, is identical. WeSCon(^^^^3ti:\3^^^^nd) reports CER = ^^^^3query3^^^^.47 and S-SIM = ^^^^3queries3^^^^.744, described as essentially unchanged. The out-of-domain CASIA experiments are further reported to confirm generalization to unseen speakers and to some emotions not present in the ESD training set.

The limitations are stated explicitly. First, WeSCon does not include explicit semantic modeling of gradual emotion transitions: it achieves smooth acoustic transitions, but does not model intermediate emotional states or continuous emotional trajectories such as a gradual progression from neutral to anger. Second, emotion diversity is limited because emotions are treated as discrete categories rather than blended or compositional states. Third, the framework depends on predefined emotion plans supplied by a user or GPT-4o; it does not autonomously choose emotion dynamics from context or adapt them interactively. The suggested future directions are to incorporate semantic emotion evolution, extend to more complex emotion spaces such as continuous valence-arousal and blended styles, and explore multilingual, context-aware, and interactive control.

Within the literature, WeSCon is positioned against three classes of prior work. Utterance-level control systems such as Seed-TTS, CosyVoice, VALL-E, and NaturalSpeech3 provide global emotion or style control from prompts but not word-level control. Text-based fine-grained emotion prediction methods can operate at phone or word level, yet they lack direct access to acoustic cues such as prosody and intensity. Reference-speech-based intra-utterance control methods such as ELaTE and EmoCtrl-TTS are described as requiring large, often private datasets with time-aligned emotion transitions and as focusing on specific patterns such as laughter–crying. Against this background, WeSCon is presented as, to the authors’ knowledge, the first self-training framework that provides a pretrained zero-shot TTS model with word-level control over both emotion and speaking rate using only small public datasets with utterance-level emotion labels and no datasets containing annotated emotion or speed transitions. This suggests a broader methodological implication: fine-grained expressive control in TTS need not be limited by the absence of dense manual annotations if a sufficiently capable zero-shot backbone can be leveraged to synthesize and distill the missing supervision (&&&^^^^3queries3^^^^&&&).