Emotion-Aware Prefix for Zero-Shot Voice Conversion

• The paper introduces Emotion-Aware Prefix as an explicit control mechanism that integrates pretrained emotion embeddings with content-invariant style cues.
• It employs a Temporal-Shuffle Transformer and Perceiver bottleneck to create a fixed-length, fused prefix, which is injected via deep-prefix prompting into the autoregressive transformer.
• Empirical results show significant improvements in emotion conversion accuracy, intelligibility, and speaker identity preservation, while also demonstrating system portability across backbones.

Emotion-aware prefix denotes a class of explicit emotion-conditioning mechanisms that intervene in the internal computation of a sequence model, rather than leaving affect to be inferred only from generic prompts or latent style cues. In its most specific and formalized usage, the term refers to a method for zero-shot voice conversion that augments the VEVO two-stage backbone with a fixed-length prefix derived from content-invariant style information and a pretrained emotion embedding, then injects that prefix into the autoregressive transformer through Deep-Prefix Prompting (Yang et al., 10 Mar 2026). Closely related work in speech, multimodal analysis, and dialogue modeling shows that explicit affective control can also be implemented through masking, gating, memory reweighting, additive conditioning, or emotion-aware discretization, but those mechanisms are not always prefixes in the strict sequence-model sense (Liu et al., 2024, Sharma et al., 2024, Peng et al., 2022, Jafari et al., 14 Apr 2026, Zhou et al., 22 Mar 2026).

1. Problem formulation and conceptual scope

The core problem addressed by Emotion-Aware Prefix is explicit emotion control in zero-shot voice conversion. Recent zero-shot voice conversion systems can preserve linguistic content while adapting speech toward a target speaker, but the paper identifies emotion conversion as suboptimal or inconsistent because existing systems rely mainly on implicit emotional cues in the prompt and therefore exhibit limited expressive capacity (Yang et al., 10 Mar 2026). The central claim is hierarchical: emotion is not merely a low-level acoustic coloration, but also affects high-level temporal and prosodic structure. If the model receives emotion only indirectly through a reference utterance, it may preserve some style or timbral coloring while failing to reshape the generated token sequence toward a distinct target emotion.

Within that framing, Emotion-Aware Prefix is introduced as an explicit control signal for VEVO’s two-stage architecture. VEVO decomposes voice conversion into sequence modulation and acoustic realization. The proposed method does not replace that decomposition; instead, it augments the sequence stage with a dedicated emotion-conditioned prefix while retaining the acoustic stage as the mechanism that renders the final waveform. The paper’s broader analytical point is that controllable emotion conversion depends on where emotion is injected in the generation hierarchy, not only on whether emotional reference speech is available (Yang et al., 10 Mar 2026).

A recurring terminological issue concerns the meaning of “prefix.” In this literature, prefix refers to conditioning that enters transformer computation as an explicit prompting structure, especially through layerwise key/value states. This differs from looser uses of “emotion-aware conditioning” in adjacent work, where emotion may instead appear as a masking prior, a gated feature stream, a memory update, or an additive bias. That distinction is important because some emotion-aware systems are prefix-like in function while not implementing prefix tuning in the formal sequence-model sense (Jafari et al., 14 Apr 2026).

2. Prefix construction from style and emotion

The Emotion-Aware Prefix encoder operates on a reference mel-spectrogram $M_{\text{ref}} \in \mathbb{R}^{T \times D}$ and has three components: a Temporal-Shuffle Transformer, a Perceiver layer, and an Emotion Fusion Layer (Yang et al., 10 Mar 2026). The first step is a temporal shuffle,

$M_{\text{shuf}} = \pi(M_{\text{ref}}),$

which randomly permutes the temporal order of mel frames. The stated purpose is to reduce content leakage while retaining style and emotion cues. By disrupting phonetic and lexical order, the encoder is pushed toward content-invariant acoustic statistics rather than textual content.

The shuffled reference is then processed by a lightweight transformer. In the reported implementation, this is a 6-layer Temporal-Shuffle Transformer. Because the resulting sequence is variable in length, a Perceiver-style bottleneck compresses it into a fixed-length representation. The Perceiver layer uses $k=32$ learnable latent tokens that cross-attend to the transformer output and produce a style embedding

$s \in \mathbb{R}^{k \times D}.$

This bottleneck makes the conditioning length independent of utterance duration, which is necessary for stable prefix injection into the autoregressive sequence model.

Explicit emotion information is added through a pretrained emotion encoder, Emotion2Vec+ Large, applied to the original unshuffled reference mel-spectrogram. The final prefix is formed by fusing the content-invariant style representation with the emotion embedding: $$\mathbf{E} = \text{Linear}(\text{Concat}[s ; \mathcal{E}(M_{\text{ref}})]).$$ The resulting prefix is therefore neither a pure style code nor a pure emotion embedding. It is a fused conditioning sequence that combines a shuffled-reference style representation with an explicit pretrained emotion representation (Yang et al., 10 Mar 2026).

This construction is central to the paper’s interpretation of explicit emotion control. The shuffled path suppresses lexical content, the Perceiver standardizes the control interface, and the emotion encoder provides a dedicated affect signal. A plausible implication is that the method treats emotion as a structured control factor that should be separated from surface content before entering the generative hierarchy.

3. Deep-prefix prompting in the two-stage voice conversion backbone

Emotion-Aware Prefix is integrated into VEVO’s two-stage backbone. Stage 1, sequence modulation, is an autoregressive transformer that predicts discrete style-rich audio tokens from source content tokens under prefix conditioning: $$P(A \mid C, \mathbf{E}) = \prod_{t=1}^{T} P(a_t \mid a_{<t}, C, \mathbf{E}),$$ where $A=[a_1,\ldots,a_T]$ is the predicted audio-token sequence, $C=[c_1,\ldots,c_n]$ is the content-token sequence extracted from the source speech, and $\mathbf{E}$ is the Emotion-Aware Prefix (Yang et al., 10 Mar 2026). This stage determines the high-level prosodic and style trajectory of the converted utterance.

Stage 2, acoustic realization, uses a Flow-Matching transformer to reconstruct a mel-spectrogram $\widehat{M}$ from the predicted audio tokens, conditioned also on reference audio tokens $M_{\text{shuf}} = \pi(M_{\text{ref}}),$0 and the reference mel-spectrogram $M_{\text{shuf}} = \pi(M_{\text{ref}}),$1. The final waveform is produced by a neural vocoder. The paper does not restate the full Flow-Matching objective, but its functional role is explicit: Stage 2 provides stable low-level spectral realization while the emotionally conditioned Stage 1 shapes the style-rich token sequence (Yang et al., 10 Mar 2026).

A defining technical feature is that the prefix is not merely prepended to the model input. Instead, the paper applies Deep-Prefix Prompting, analogous to P-Tuning v2, by injecting the prefix into every transformer layer as layer-specific key/value prefixes: $M_{\text{shuf}} = \pi(M_{\text{ref}}),$2 These projected prefixes are prepended to the ordinary key/value states of each layer, so self-attention operates over an augmented memory rather than over the hidden states alone (Yang et al., 10 Mar 2026). The method therefore implements prefix conditioning in the strict technical sense: the prefix modifies attention computation throughout the depth of the autoregressive transformer.

The paper’s ablation structure gives this hierarchy a precise interpretation. “Control Sequence” means that Stage 1 receives target emotion while Stage 2 receives source emotion. “Control Acoustic” reverses that assignment. “Joint Control” aligns both stages with the target emotion. The resulting analysis shows that explicit emotion control at the sequence level is the dominant contributor, while acoustic realization amplifies that signal when both stages are emotionally aligned (Yang et al., 10 Mar 2026).

4. Training configuration and evaluation protocol

The experimental benchmark is the Emotional Speech Dataset (ESD), containing 10 English speakers, with 5 male and 5 female speakers, and 5 emotions: Neutral, Happy, Sad, Angry, and Surprised (Yang et al., 10 Mar 2026). Each speaker has 350 utterances per emotion, and the corpus is parallel across emotions. The split uses the first 300 utterances for training; the remaining 50 are partitioned into 20 reference prompts and 30 test utterances.

The adaptation strategy is parameter-efficient. All VEVO backbone parameters remain frozen. The Emotion-Aware Prefix Encoder is fully trainable, and LoRA is applied to the autoregressive transformer with rank $M_{\text{shuf}} = \pi(M_{\text{ref}}),$3. Optimization uses AdamW with learning rate $M_{\text{shuf}} = \pi(M_{\text{ref}}),$4 for 46k fine-tuning steps (Yang et al., 10 Mar 2026). The paper presents the method as a lightweight augmentation rather than a full retraining of the voice conversion system.

Evaluation covers speaker identity, quality, intelligibility, and emotion. Speaker identity is measured by Speaker Centroid Similarity using ECAPA-TDNN embeddings and cross-emotion speaker centroids, and by Equal Error Rate in a speaker verification setting. Quality is measured by UTMOSv2 and DNSMOS P.835, reported as SIG and OVRL. Intelligibility is measured by WER using Whisper-v3-Large transcription against ground-truth text. Emotion is measured in two ways using Emotion2Vec+ Large: Emotion Conversion Accuracy, defined by top-1 emotion-label agreement with the target emotion, and Emotion Similarity, defined as cosine similarity between emotion embeddings of the converted and target utterances (Yang et al., 10 Mar 2026).

Subjective evaluation is also included. Ten human listeners each evaluated 120 utterances. The reported subjective measures are MOS on a 1-to-5 scale, ABX preference for emotional similarity, and a 5-point speaker-similarity scale relative to the reference utterance (Yang et al., 10 Mar 2026).

5. Empirical results, ablations, and documented limitations

The headline result is a large improvement in emotion conversion performance. VEVO yields an Emotion Conversion Accuracy of 42.40%, whereas Emotion-Aware Prefix reaches 85.50%; the paper describes this as doubling the baseline ECA while maintaining linguistic integrity and speech quality without compromising speaker identity (Yang et al., 10 Mar 2026). Emotion Similarity rises from 0.696 to 0.850. Speaker identity metrics also improve modestly, with EER moving from 5.40% to 4.50% and Spk-Cent SIM from 0.476 to 0.500. WER improves from 10.08% to 6.28%. UTMOSv2 is slightly lower than VEVO, at 2.960 versus 3.060, while DNSMOS SIG and OVRL are slightly better, at 3.431 and 3.152 versus 3.409 and 3.093 (Yang et al., 10 Mar 2026).

The deep-prefix ablation isolates the effect of layerwise injection. A variant without Deep-Prefix Prompting, in which the prefix is only prepended to the input sequence, already achieves ECA 83.50% and Emo SIM 0.842. Full deep-prefix prompting improves these to 85.50% and 0.850, respectively (Yang et al., 10 Mar 2026). The result indicates that most of the gain comes from the explicit prefix itself, while layerwise key/value prompting adds a smaller but measurable improvement.

The stagewise control ablation is central to the paper’s interpretation of hierarchy:

Control condition	VEVO ECA	Proposed ECA
Sequence	12.50%	47.00%
Acoustic	32.70%	34.50%
Joint	42.40%	85.50%

These numbers show that, for the proposed method, controlling sequence modulation alone is more effective than controlling acoustic realization alone, but joint control is far stronger than either single-stage condition (Yang et al., 10 Mar 2026). The paper interprets this as evidence that Stage 1 creates emotionally appropriate temporal structure and Stage 2 realizes it more faithfully when both are aligned to the target emotion. For baseline VEVO, by contrast, acoustic-only control is much stronger than sequence-only control, indicating that the unmodified system expresses emotion primarily through low-level acoustic conditioning rather than explicit sequence planning.

A further comparison ports Emotion-Aware Prefix to GenVC. On that backbone, ECA improves from 32.48% to 58.35% and Emo SIM from 0.681 to 0.760, but EER worsens from 20.87% to 44.51% (Yang et al., 10 Mar 2026). The paper attributes this to architectural coupling: VEVO’s separate acoustic realization stage stabilizes speaker identity, whereas GenVC allows strong emotion control at the sequence level to distort identity. The broader implication is that prefix-based emotion control is portable across backbones, but its interaction with speaker preservation depends on how acoustics are decoupled from sequence planning.

The reported limitations are also specific. The experiments are confined to ESD, English, and five categorical emotions. The method does not provide fine-grained intensity control. There is a slight naturalness drop in UTMOSv2 relative to VEVO. Some implementation details, including exact tokenization and the Flow-Matching loss, are inherited from VEVO and not restated in the paper (Yang et al., 10 Mar 2026).

6. Related affective conditioning mechanisms and boundary cases

Emotion-Aware Prefix is part of a broader movement toward explicit affective control, but adjacent systems often use different control interfaces. In speech-driven facial animation, SEDTalker introduces frame-level speech emotion diarization and conditions a hybrid Transformer–Mamba animation model with time-varying emotion–intensity embeddings (Jafari et al., 14 Apr 2026). The conditioning takes the form

$M_{\text{shuf}} = \pi(M_{\text{ref}}),$5

with time-varying interpolation at transitions and additive injection into projected audio features. The paper is explicit that this is not prefix conditioning: there are no prepended tokens, no learned key/value prompts, and no prefix-tuning-style modification of transformer attention states. Instead, it is additive per-time-step conditioning aligned with the audio feature timeline (Jafari et al., 14 Apr 2026). This distinction clarifies a common misconception: not every explicit emotion conditioner constitutes an emotion-aware prefix in the strict transformer sense.

In self-supervised speech representation learning, “Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge” introduces frame-level emotional masking strategy rather than prefix prompting (Liu et al., 2024). Frame-level emotion intensities are extracted by StrengthNet, aligned to acoustic frames by a linear layer, and used to mask the top $M_{\text{shuf}} = \pi(M_{\text{ref}}),$6 most emotionally intense frames. On IEMOCAP, MockingJay improves from 50.28% to 57.42% ACC at 25% masking, and NPC improves from 59.08% to 62.14% with the joint-input formulation (Liu et al., 2024). This suggests that explicit emotion priors can be injected before encoding by altering the corruption process and reconstruction target, not only by adding prompts to the encoder.

In multimodal meme emotion detection, ALFRED uses an auxiliary visual emotion encoder, gated multimodal fusion, and gated cross attention to make emotion-enriched visual cues steer cross-modal interaction (Sharma et al., 2024). On MOOD, ALFRED reaches macro-F1 0.8243 and exceeds the best baseline by 4.94% F1. The ablation removing emotion features drops F1 from 82.43 to 77.19 (Sharma et al., 2024). The paper does not implement prefix tuning, but its gated emotion-aware feature routing is structurally similar to prompt-like steering. This suggests that emotion-aware prefixes in multimodal transformers may be most effective when they behave as routing or gating priors rather than as passive label tokens.

In discrete speech representation learning, emotion-aware quantization shows that residual vector quantization disproportionately degrades affective information, especially in the first 3–4 layers of aggressive compression (Zhou et al., 22 Mar 2026). Emotion-specific and emotion-biased codebooks improve both hard and soft emotion preservation, and Emo-Q provides a routed quantization mechanism that improves macro-F1 at lower bitrates. A plausible implication is that prefix-based emotion control in token speech models depends not only on the prefix mechanism itself but also on whether the tokenizer preserves affective structure in the discrete interface.

In persuasive dialogue, CFO-Net treats emotion as feedback that updates the utility of historical strategies over time (Peng et al., 2022). Its feedback memory module reweights strategy representations according to predicted positive, neutral, or negative emotion, improving macro-F1 from 61.74 to 65.41 on PersuasionForGood. Although the model is not a prefix-tuning system, its strategy-memory update is directly relevant to dynamic emotion-conditioned prompting. This suggests that, in dialogue generation, an emotion-aware prefix need not remain fixed across a conversation; it may instead be updated as a persistent control state reflecting how prior strategic choices were received.

Taken together, these systems define the practical boundaries of the term. Emotion-Aware Prefix, in the narrow sense, refers to explicit prefix-style conditioning of a sequence model by emotion-coded representations, as in the VEVO-based voice conversion method (Yang et al., 10 Mar 2026). In the broader research landscape, however, the term also indexes a design principle: emotion should be represented explicitly and injected at the stage where it can steer sequence planning, fusion, discretization, or memory updating most effectively (Liu et al., 2024, Sharma et al., 2024, Zhou et al., 22 Mar 2026, Peng et al., 2022, Jafari et al., 14 Apr 2026).