Maestro-EVC: Controllable Emotional Voice Conversion

• Maestro-EVC is a controllable emotional voice conversion framework that separates linguistic content, speaker identity, and emotion for independent control.
• It introduces temporal emotion modeling with explicit prosody transfer to capture fine-grained dynamics in pitch, energy, and duration.
• The system employs dedicated encoders and adversarial losses to ensure effective disentanglement and robust synthesis compared to previous methods.

Maestro-EVC is a controllable emotional voice conversion framework that aims to modify the emotional style of speech while preserving linguistic content, while also enabling independent control of content, speaker identity, and emotion through separate references. The framework is designed around the claim that practical emotional voice conversion requires not only disentanglement of these factors, but also modeling of fine-grained temporal emotional dynamics, including local variation in pitch, energy, rhythm, and speaking rate. It is introduced in “Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosody” (Yoon et al., 9 Aug 2025).

1. Problem formulation and design goals

Emotional voice conversion (EVC) aims to convert a source utterance into a target emotional style while preserving the linguistic content. In the formulation used by Maestro-EVC, a practically useful system must support independent control of four properties: content, speaker identity, emotional style, and fine-grained temporal emotional dynamics (Yoon et al., 9 Aug 2025).

The central difficulty is that these factors are entangled in speech. If a model does not disentangle content, speaker identity, and emotion, it cannot reliably mix content, speaker, and emotion from different references. If it models emotion only at the utterance level, it may reproduce the correct coarse emotion category while failing to capture temporally localized prosodic behavior. The paper therefore argues that utterance-level emotion embeddings are insufficient for expressive synthesis, because they miss local pitch accents, rhythm changes, intensity variation, speaking-rate dynamics, and temporal emotional contours (Yoon et al., 9 Aug 2025).

The paper criticizes prior EVC systems on two main grounds. First, disentanglement is often incomplete: predefined emotion labels are coarse, fixed speaker IDs do not generalize to unseen speakers, and reconstruction-based latent decompositions extracted from a single utterance may fail to fully separate content, speaker, and emotion. Second, prosody is often modeled implicitly rather than directly transferred from an emotion reference, which is especially weak when the content and emotion references are prosody-mismatched. In such cases, naive prosody transfer can yield unnatural speech unless the system is explicitly robust to mismatch (Yoon et al., 9 Aug 2025).

Maestro-EVC is presented as a response to these limitations. Its high-level principle is to use separate references for content, speaker, and emotion so that each attribute can be represented by a dedicated pathway: $x_c,\quad x_e,\quad x_s$ where $x_c$ is the content reference, $x_e$ is the emotion reference, and $x_s$ is the speaker reference. During training, the same utterance is reused for all three roles,

$x_c = x_e = x_s$

which enables reconstruction training while still encouraging attribute-specific encoding (Yoon et al., 9 Aug 2025).

2. Overall architecture

Maestro-EVC consists of four main functional components: a Content Encoder, Temporal Content-aware Emotion Modeling (TCEM), Explicit Emotional Prosody Transfer (EEPT), and an Emotion-Invariant Speaker Encoder (EISE). These representations are then used with a HiFi-GAN vocoder for waveform synthesis (Yoon et al., 9 Aug 2025).

The Content Encoder is intended to extract linguistic information only. It uses a pre-trained HuBERT model. Given the content reference $x_c$, HuBERT produces frame-level continuous representations $z$, which are discretized using K-means clustering to obtain discrete units $\hat{z}$. These units are then mapped through an embedding table to produce the discrete content representation $c$. The paper characterizes this representation as a compact symbolic representation intended to preserve linguistic content while discarding speaker and emotion cues as much as possible (Yoon et al., 9 Aug 2025).

The TCEM module is responsible for emotion representation. Its objective is to extract frame-level emotion features and align them with the target content so that emotion is modeled as temporally structured and content-aware rather than as a single utterance-level vector. This module adopts a pre-trained temporal emotion encoder from speech emotion diarization work and outputs a sequence of emotion embeddings $e$ (Yoon et al., 9 Aug 2025).

The EEPT module provides explicit modeling of prosody. It focuses on three prosodic features extracted from the emotion reference: $x_c$0 for $x_c$1, $x_c$2 for energy, and $x_c$3 for duration. These features are smoothed with a Savitzky-Golay filter to obtain $x_c$4, $x_c$5, and $x_c$6, with the stated purpose of emphasizing broad contour rather than micro-level noise (Yoon et al., 9 Aug 2025).

The EISE module extracts speaker identity from $x_c$7 while suppressing emotional information. It uses a frozen pre-trained ECAPA-TDNN speaker encoder, followed by trainable layers that are regularized adversarially to make the speaker representation emotion-invariant (Yoon et al., 9 Aug 2025).

This architectural partitioning is intended to make the content, speaker, and emotion pathways independently controllable. A plausible implication is that the system’s controllability depends not only on modular decomposition, but on the success of the auxiliary losses used to prevent leakage across these pathways.

3. Temporal emotion modeling and alignment

The core contribution of TCEM is the introduction of a temporal emotion representation aligned to content. The paper states that emotion should be modeled at the frame level because short acoustic windows encode both affective and phonetic information, and because emotional expression is realized through temporal prosodic structure rather than only through utterance-level style labels (Yoon et al., 9 Aug 2025).

The temporal emotion encoder produces a sequence of emotion embeddings $x_c$8. To align emotion with content, Maestro-EVC applies cross-attention where the query is the content sequence $x_c$9 and the key and value are both the emotion sequence $x_e$0. This produces an aligned emotion representation $x_e$1. The intended interpretation is that content determines where emotional information should be applied, while the emotion embedding sequence supplies the emotional contour to be transferred (Yoon et al., 9 Aug 2025).

The paper further argues that even a temporal emotion encoder may retain phonetic information, because frame-level acoustic features inevitably contain content as well as emotion. To address this, TCEM includes a projection block, a gradient reversal layer (GRL), and a content classifier trained with the adversarial content classification loss

$x_e$2

This mechanism is intended to remove residual linguistic leakage from the emotion representation, yielding a disentangled representation denoted

$x_e$3

The paper reports that removing this GRL-based content disentanglement degrades intelligibility and emotion quality in the ablation study (Yoon et al., 9 Aug 2025).

The TCEM design directly reflects the paper’s claim that temporal emotional dynamics are not reducible to coarse emotion categories. The qualitative evidence further states that two different “Surprise” references can produce different converted contours, which the authors interpret as evidence that the model transfers fine-grained prosodic variation rather than only categorical emotion identity (Yoon et al., 9 Aug 2025).

4. Explicit prosody transfer and mismatch robustness

The Explicit Emotional Prosody Transfer module is the second major component of Maestro-EVC. It explicitly transfers $x_e$4, energy, and duration from the emotion reference rather than relying only on latent prosody modeling (Yoon et al., 9 Aug 2025).

A central problem identified by the paper is that reconstruction-style training often exposes the model only to matched prosody, because the content and emotion references are identical during training. This leaves the model poorly prepared for inference settings in which the content and emotion references differ in rhythm, speaking rate, or local timing. Maestro-EVC addresses this with prosody augmentation: $x_e$5 The augmentation randomly applies one of two transformations with equal probability. The first is random shifting, which shifts the entire prosody sequence along the time axis by a random amount. The second is piecewise time warping, which splits the sequence into 2–5 segments, stretches or compresses each segment randomly, and concatenates them back to the original length. According to the paper, this simulates mismatches in speaking rate, rhythm, and timing alignment (Yoon et al., 9 Aug 2025).

The $x_e$6/energy predictor explicitly conditions on projected emotion-reference $x_e$7 and energy, the discrete content representation $x_e$8, and the voiced/unvoiced mask $x_e$9 from the content reference: $x_s$0 The voiced/unvoiced mask is included because $x_s$1 is not meaningful in unvoiced regions, and because pitch and energy transfer should be guided toward voiced segments (Yoon et al., 9 Aug 2025).

Duration is modeled using the discrete content units and their repetition counts. The paper defines

$x_s$2

where $x_s$3 is the unique unit sequence and $x_s$4 is the number of consecutive occurrences of each unit, treated as duration $x_s$5. The duration predictor uses the unique content units from $x_s$6, the smoothed duration from the emotion reference $x_s$7, and the disentangled emotion representation $x_s$8 to predict $x_s$9 (Yoon et al., 9 Aug 2025).

The corresponding prosody supervision is

$x_c = x_e = x_s$0

where $x_c = x_e = x_s$1 and $x_c = x_e = x_s$2 are $x_c = x_e = x_s$3 losses and $x_c = x_e = x_s$4 is an $x_c = x_e = x_s$5 loss. The paper attributes the system’s improved prosody similarity particularly to this explicit supervision together with prosody augmentation (Yoon et al., 9 Aug 2025).

The ablation study gives a nuanced result for prosody augmentation. Removing augmentation worsens WER, CER, EECS, and SCA, but slightly increases $x_c = x_e = x_s$6-PCC and E-PCC. The paper interprets this as evidence that without augmentation the model may overfit to rigid prosody matching, improving correlation scores while harming naturalness and robustness under prosody mismatch (Yoon et al., 9 Aug 2025).

5. Speaker disentanglement, objectives, and synthesis

The Emotion-Invariant Speaker Encoder is intended to preserve speaker identity while removing emotion leakage. Maestro-EVC uses a frozen pre-trained ECAPA-TDNN speaker encoder and appends trainable layers regularized by a GRL and an emotion classifier. This produces an adversarial emotion-classification loss

$x_c = x_e = x_s$7

which discourages emotional information from being present in the speaker embedding (Yoon et al., 9 Aug 2025).

The paper argues that adversarial suppression alone is insufficient, because it does not guarantee that the same speaker will map consistently across different emotions. Maestro-EVC therefore adds a triplet loss: $x_c = x_e = x_s$8 where $x_c = x_e = x_s$9 is cosine similarity, $x_c$0 is the anchor, $x_c$1 is a positive sample from the same speaker but different emotion, $x_c$2 is a negative sample from a different speaker, and the margin is $x_c$3. The total speaker loss is

$x_c$4

The ablation study reports that removing $x_c$5 decreases speaker classification accuracy and slightly decreases emotion similarity, which the authors interpret as evidence of residual speaker-emotion entanglement without this loss (Yoon et al., 9 Aug 2025).

Training is performed in a reconstruction setting with HiFi-GAN-style adversarial objectives. The generator and discriminator losses are

$x_c$6

$x_c$7

and the total generator objective is

$x_c$8

Here, $x_c$9 is used for speaker identity disentanglement, $z$0 for removing content leakage from emotion representations, and $z$1 for explicit prosody transfer (Yoon et al., 9 Aug 2025).

The implementation described in the paper uses a HuBERT base 12-layer model pretrained on LibriSpeech 960h, ECAPA-TDNN pretrained on VoxCeleb, and training and evaluation speech from the English partition of the Emotional Speech Dataset (ESD). The ESD subset contains 350 parallel utterances, 16 kHz audio, 10 English speakers, and 5 emotions: neutral, happy, angry, sad, and surprise. Additional reported implementation details include a content vocabulary size of 500, content embedding dimension 256, 80-bin mel-spectrograms, window size 1024, hop size 256, WORLD $z$2 extraction, two stacked Transformer blocks for the $z$3/energy and duration predictors, AdamW optimization, and a learning rate of $z$4 (Yoon et al., 9 Aug 2025).

6. Evaluation, results, and scope

The evaluation compares Maestro-EVC against StyleVC and ZEST. The reported objective metrics are WER, CER, EECS, SCA, $z$5-PCC, and E-PCC, while subjective evaluation uses MOS on naturalness, emotional similarity, speaker similarity, and prosody similarity. Prosody similarity is explicitly defined in terms of pitch contour, intensity, and speaking-rate variation (Yoon et al., 9 Aug 2025).

Reported objective and subjective results

Setting	Metric	Maestro-EVC
Seen scenario	WER	11.78
Seen scenario	CER	6.54
Seen scenario	EECS	0.819
Seen scenario	SCA	93.69
Seen scenario	F0-PCC	0.551
Seen scenario	E-PCC	0.316
Unseen emotions (UE)	CER	9.64
Unseen emotions (UE)	EECS	0.768
Unseen emotions (UE)	SCA	88.67
Unseen emotions (UE)	F0-PCC	0.370
Unseen speakers (US)	CER	6.10
Unseen speakers (US)	EECS	0.841
Unseen speakers (US)	F0-PCC	0.577
MOS	Naturalness	$z$6
MOS	Emo. Sim.	$z$7
MOS	Spk. Sim.	$z$8
MOS	Pro. Sim.	$z$9

In the seen-scenario comparison, the paper reports that Maestro-EVC outperforms both baselines on all reported metrics. Specifically, WER is 11.78 versus 16.79 for StyleVC and 17.18 for ZEST; CER is 6.54 versus 9.46 and 9.85; EECS is 0.819 versus 0.537 and 0.779; SCA is 93.69 versus 90.10 and 93.54; $\hat{z}$0-PCC is 0.551 versus 0.380 and 0.432; and E-PCC is 0.316 versus 0.297 and 0.293 (Yoon et al., 9 Aug 2025).

The paper further reports best performance in unseen-emotion and unseen-speaker settings. For unseen emotions, Maestro-EVC achieves CER 9.64, EECS 0.768, SCA 88.67, and $\hat{z}$1-PCC 0.370. For unseen speakers, it achieves CER 6.10, EECS 0.841, and $\hat{z}$2-PCC 0.577. The authors interpret these results as evidence that the model generalizes well to unseen identities and unseen emotions (Yoon et al., 9 Aug 2025).

The subjective results likewise favor Maestro-EVC: naturalness $\hat{z}$3, emotional similarity $\hat{z}$4, speaker similarity $\hat{z}$5, and prosody similarity $\hat{z}$6. The particularly high prosody-similarity score is presented as support for the claim that explicit prosody modeling improves perceived expressiveness (Yoon et al., 9 Aug 2025).

The ablation study is central to the paper’s argument. Removing the content GRL raises WER to 20.98, raises CER to 12.38, and lowers SCA to 86.80. Removing the temporal emotion representation lowers SCA sharply to 77.25 and lowers E-PCC to 0.283. Removing prosody augmentation worsens WER, CER, EECS, and SCA. Removing $\hat{z}$7 decreases SCA and slightly decreases EECS. These results are used to support the necessity of each major module: content disentanglement in emotion space, frame-level temporal emotion modeling, mismatch-robust prosody transfer, and emotion-invariant speaker encoding (Yoon et al., 9 Aug 2025).

A common source of confusion is nomenclature: “Maestro-EVC” refers specifically to the controllable emotional voice conversion framework introduced in (Yoon et al., 9 Aug 2025), not to other unrelated systems that use the name “Maestro” in robotics, LLM serving, training systems, multi-agent evaluation, reinforcement learning, or scientific computing. Within the scope of (Yoon et al., 9 Aug 2025), the defining features are separate reference-guided control of content, speaker, and emotion, temporal emotion embeddings, and explicit $\hat{z}$8/energy/duration transfer.

The paper does not provide an extended limitations section, but it does make several constraints clear. The experiments use the English partition of ESD, the training setup relies on parallel utterances, and the method depends on pretrained HuBERT, ECAPA-TDNN, and a temporal emotion encoder. Multilingual generalization is therefore not demonstrated. The paper also states that although the system is robust to prosody mismatch, reference quality still matters. This suggests that Maestro-EVC should be understood as a reference-guided controllable synthesis framework whose performance depends on the informativeness and compatibility of its content, emotion, and speaker exemplars (Yoon et al., 9 Aug 2025).