TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion

Abstract: Traditional emotional voice conversion (EVC) conditions generation on explicit target emotions like labels or references, defining the target affective state but omitting the direction or nature of the transition. We introduce instruction-guided relative emotional voice conversion, a task where natural-language instructions specify source-conditioned affective transformations (e.g., "make the speech slightly calmer" or "sound noticeably more confident") instead of fixed targets. To support this task, we construct TRACE-Instruct, a dataset of relative emotion instructions covering categorical transitions, intensity modifications, and open-ended affective changes. We propose TRACE-EVC, a zero-shot framework built around Emo-Compass, a module that models each conversion as a source-anchored rectified flow. Rather than conditioning on an explicit target, it predicts the direction and degree of the affective change. Experiments demonstrate that TRACE-EVC accurately follows relative emotion instructions while preserving speaker identity, linguistic content, and speech quality, and remains competitive with conventional EVC systems on standard categorical emotion conversion.

• The paper introduces instruction-guided relative affect control for zero-shot emotional voice conversion using free-form natural language instructions.
• The methodology leverages the TRACE-Instruct dataset and an Emo-Compass module to predict emotion embedding shifts in a continuous VAD space.
• Empirical results demonstrate high emotion embedding cosine similarity and instruction compliance, surpassing baseline methods on ESD and MEAD datasets.

TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion

Introduction and Motivation

Emotional Voice Conversion (EVC) has traditionally relied on categorical labels or reference utterances to define the target affect, limiting expressiveness and practical utility in real-world scenarios where nuanced or intuitive control of emotion is required. The work in "TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion" (2607.03666) reframes the EVC paradigm by introducing instruction-guided relative EVC, where natural language instructions specify the desired affective transformation relative to a source utterance, rather than absolute affect targets.

This reformulation necessitates a framework that can interpret relative, free-form instructions (such as "make the speech slightly calmer") and map them appropriately in the affective space, which may involve complex trajectories in prosodic and emotional features. Two core contributions underlie this work: the TRACE-Instruct dataset for supervised learning of relative affective instructions, and the TRACE-EVC model leveraging a novel Emo-Compass component for emotion embedding prediction.

Figure 1: Illustration of the instruction-guided relative EVC task. The instruction specifies how the source emotion should change rather than a fixed target.

Framework Overview: TRACE-Instruct and TRACE-EVC

The method comprises two primary modules, arranged in a cascaded pipeline:

• TRACE-Instruct Dataset: Automatically generates a large corpus of free-form, relative emotion instructions labeling transitions between source and target utterances. Using paired emotional speech from ESD and MEAD corpora, emotion shifts are quantified in the VAD (Valence, Arousal, Dominance) space; a LLM (Qwen3) is prompted to provide natural-language descriptions of these affective transitions, without revealing target emotion categories or intensity values.
• TRACE-EVC Model: The centerpiece is the Emo-Compass, which reframes emotion embedding prediction as source-anchored rectified flow. Given the source emotion embedding and the instruction encoding, Emo-Compass predicts a displacement vector in the affective space corresponding to the instructed change. The synthesis module (based on DurFlex-EVC) then generates speech using the predicted target emotion embedding, the source content, and speaker identity. This architecture supports zero-shot conversion without requiring target labels, reference utterances, or explicit emotion categories at inference.

Figure 2: Overview of the proposed framework: a TRACE-Instruct generation pipeline produces relative instructions for training, while TRACE-EVC performs the conversion using predicted emotion embeddings guided by instructions alone at inference.

Emo-Compass: Source-Anchored Flow for Relative Emotion Embedding Prediction

The Emo-Compass module formalizes emotion conversion as a learned flow in affective embedding space, anchored at the source emission. At training, pairs $(\mathbf{z}_{src}, \mathbf{z}_{tgt})$ with a relative instruction are used to supervise the model to predict the displacement vector $\mathbf{v}^* = \mathbf{z}_{tgt} - \mathbf{z}_{src}$, jointly using the instruction and the source embedding as context.

The emotion embedding concatenates information from emotion2vec (categorical emotion), VAD coordinates (continuous affective state), and prosodic features (mean $F_0$ and energy), supporting both coarse and fine-grained affect control. Training also employs auxiliary losses (directional and categorical emotion, VAD/prosody reconstruction, intensity consistency) to ensure that the model does not degenerate to trivial dataset priors and can generalize to fine-grained, continuous affective changes.

At inference, the Emo-Compass integrates the predicted flow from the source embedding using a sequence of steps (Euler integration), producing a target emotion embedding consistent with the instructed change. This embedding is then used for synthesis without further need for target references.

TRACE-EVC Performance and Empirical Analysis

TRACE-EVC demonstrates robust performance across both categorical emotion conversion (inter-emotion) and intensity modulation (intra-emotion), evaluated on ESD and MEAD datasets. The model achieves high emotion embedding cosine similarity (EECS), emotion classification accuracy, and speaker similarity—outperforming reference-based and label-based baselines on key metrics.

• In inter-emotion (ESD), TRACE-EVC achieves EECS: 0.82 and classification accuracy: 93.00% with competitive WER and speech quality.
• In intra-emotion (MEAD), TRACE-EVC closely matches ground-truth naturalness (NISQA), maintains above 0.7 speaker similarity (SECS) on unseen speakers, and provides graded, instruction-controlled intensity shifts in arousal.

A critical ablation demonstrates that removing the explicit flow-based prediction (replacing Emo-Compass with naive embedding concatenation) degrades both emotion conversion and instruction following, highlighting the importance of the source-anchored flow architecture.

Instruction compliance is measured subjectively, with mean instruction-following (IF) scores >4 (on a 1–5 Likert scale) for both conversion types, signifying that human raters perceive the converted speech as following the intended relative affective transformation.

Figure 3: Mean arousal shift under relative intensity instructions on MEAD. Free-form instructions are grouped by direction and strength, confirming that arousal shifts increase monotonically as instructed.

TRACE-Instruct Dataset and Instruction Diversity

TRACE-Instruct constitutes a large-scale resource for relative affect control, with >500,000 instructions generated. Analysis shows high lexical diversity (HD-D of 0.776), low redundancy (Self-BLEU of 0.028), and longer, more descriptive instructions compared to single-utterance style datasets. This resource enables the learning of nuanced, non-categorical affective transformations that go beyond existing EVC control methods.

Implications and Future Directions

The introduction of instruction-guided relative affective control opens several key avenues for future research:

• Generalization Across Domains: The demonstrated zero-shot generalization to unseen speakers and instructions paves the way for broadly deployable, flexible affective TTS/VC systems in human-computer interaction, assistive communication, and creative voice applications.
• Instruction Compositionality: The source-anchored flow formulation suggests extensibility toward compositional or multi-step affective transformations guided by sequences of instructions.
• Model Interpretability: By explicitly modeling VAD and prosodic shifts, TRACE-EVC frameworks provide interpretable controls for affect, bridging lab-based affective models and real-world subjective perceptions.
• Unified Speech-Text Affect Control: Integrating the TRACE-EVC paradigm into unified TTS/VC systems could yield fine-grained, instruction-driven affective generation from either speech or text, synergizing recent advances in prompt-based speech generation [Promptevc, (2607.03666)].

Conclusion

TRACE-EVC advances the field of expressive speech synthesis by recasting EVC as an instruction-following task, with a robust model architecture that directly predicts relative affective changes in a continuous space. Strong empirical results, both objective and subjective, corroborate the efficacy and flexibility of this approach to zero-shot, instruction-driven voice conversion. This methodology challenges conventional target-based paradigms, aligningly natural user intent with model control, and establishes a new foundation for research in affective speech technologies.