Vevo2: Unified Controllable Voice Generation

• Vevo2 is a unified framework for controllable speech and singing generation that leverages a shared prosody-centric representation for flexible control over text, melody, style, and timbre.
• It employs dual audio tokenizers and a two-stage architecture—an autoregressive content-style model and a flow-matching acoustic model—to enable synthesis, conversion, and editing across vocal domains.
• Unified prosody learning with multi-objective post-training enhances intelligibility and expressive nuance while balancing key parameters in diverse voice generation tasks.

Vevo2 is a unified framework for controllable speech and singing voice generation that treats speech and singing as related prosodic domains rather than as completely separate tasks. Its central design premise is to bridge the two through a shared prosody-centric representation, while still providing independent control over text or lyrics, prosody or melody, style, and timbre. Architecturally, Vevo2 combines two audio tokenizers, an auto-regressive content-style modeling stage, a flow-matching acoustic modeling stage, and a Vocos-based vocoder, and is reported to support synthesis, conversion, and editing tasks across both speech and singing (Zhang et al., 22 Aug 2025).

1. Conceptual scope and design premise

Vevo2 addresses controllable human voice generation in settings that include speech synthesis, singing voice synthesis, voice conversion, singing voice conversion, speech editing, singing lyric editing, speech style conversion such as accent or emotion conversion, singing style conversion, and cross-domain melody-driven tasks such as humming-to-singing and instrument-to-singing. The paper frames these tasks as difficult because singing is much more constrained than speech in prosody, annotated singing data is scarce, prior speech tokenizers and models are speech-centric, and disentangling text or lyrics, prosody or melody, style, and timbre is hard (Zhang et al., 22 Aug 2025).

The system is designed to overcome the scarcity of annotated singing data by avoiding dependence on expert music notation. Instead, it uses a music-notation-free prosody tokenizer trained on chromagrams extracted directly from audio. The paper’s broader claim is that unified modeling of speech and singing is beneficial in both directions: speech data can help singing quality and intelligibility, while singing data can improve expressive speech and prosody following. This places Vevo2 in continuity with the earlier Vevo system, which already separated style and timbre in a two-stage zero-shot voice imitation pipeline, but Vevo2 extends that line toward unified speech and singing generation via unified prosody learning (Zhang et al., 22 Aug 2025, Zhang et al., 11 Feb 2025).

The term “controllable” is operational rather than rhetorical. In Vevo2, control includes text or lyrics control, prosody control, melody control, style control, timbre control, editing control, conversion control, duration control, and pitch region control. A plausible implication is that the system is intended not as a narrow singing synthesizer or a narrow zero-shot TTS model, but as a general controllable vocal generation framework spanning both domains (Zhang et al., 22 Aug 2025).

2. Tokenizers and learned representations

Vevo2 begins with two discrete representations that are shared across speech and singing. The first is a music-notation-free prosody tokenizer. Its input is a 50 Hz chromagram extracted from raw audio, and its output is discrete prosody tokens at 6.25 Hz, equivalent to 56.25 bps. The paper motivates chromagram as octave-free, notation-free, and robust across speech, singing, and instrumental music, so that the same token space can encode speech prosody, singing melody, humming, and instrumental prompts (Zhang et al., 22 Aug 2025).

The prosody tokenizer is a single-codebook VQ-VAE with downsampling, encoder, vector quantization, decoder, and upsampling. Given input chromagram $\mathbf{x}$ and codebook

$$\mathbf{E} = [\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_K],$$

the paper defines

$$\begin{aligned} \mathbf{z}_e(\mathbf{x}) &= \text{Encoder}(\text{Downsample}(\mathbf{x})), \ \mathbf{z}_q(\mathbf{x}) &= \mathbf{e}_k,\quad \text{where } k = \arg\min_j \|\mathbf{z}_e(\mathbf{x}) - \mathbf{e}_j\|_2, \ \hat{\mathbf{x}} &= \text{Upsample}(\text{Decoder}(\mathbf{z}_q(\mathbf{x}))). \end{aligned}$$

The training loss is

$$\mathcal{L} = \lambda \|\mathbf{x} - \hat{\mathbf{x}}\|_2^2 + \beta \|\mathbf{z}_e(\mathbf{x}) - \mathbf{z}_q(\mathbf{x})\|_2^2.$$

Its reported technical details are a 24 kHz sampling rate; chromagram extraction via Librosa with 24 chroma bins, FFT points 1920, hop length 480, and window size 1920; downsampling ratio 8; codebook size 512; 38M parameters; AdamW; peak learning rate $1\mathrm{e}{-4}$; 10K warmup; and 300K total updates (Zhang et al., 22 Aug 2025).

The second representation is a low-frame-rate content-style tokenizer. It reconstructs both 50 Hz chromagram and 50 Hz Whisper encoder hidden features from pre-trained whisper-medium, thereby encouraging a token space that combines linguistic content, melody or prosody, and expressive style. Its output rate is 12.5 Hz, or 175 bps, with a codebook size of 16,384 and 44M parameters. The low frame rate is explicitly motivated as a way to shorten the sequence seen by the auto-regressive transformer, reducing learning burden and inference cost relative to 25 or 50 Hz semantic token systems (Zhang et al., 22 Aug 2025).

The paper treats timbre disentanglement in this tokenizer as an empirical property rather than as the result of a separate explicit disentanglement loss. Concretely, content-style tokens serve as the non-timbre conditioning representation, while timbre is injected separately in the flow-matching acoustic model from a timbre reference. This suggests that Vevo2 relies on representational division of labor across stages rather than on an explicit adversarial or information-theoretic disentanglement objective (Zhang et al., 22 Aug 2025).

3. Two-stage generation architecture

Vevo2 is organized as a two-stage system plus the two front-end tokenizers. Stage 1 is an auto-regressive content-style modeling stage. Stage 2 is a flow-matching acoustic modeling stage. A Vocos-based vocoder then synthesizes waveform audio from the predicted mel spectrogram (Zhang et al., 22 Aug 2025).

In Stage 1, an autoregressive transformer predicts target content-style tokens from combinations of text, optional prosody tokens, and optional style reference content-style tokens. This stage is responsible for text, prosody or melody, and style control. The model is initialized from Qwen2.5-0.5B and expanded with the tokenizer vocabularies; the appendix reports 509M total AR parameters, AdamW, peak learning rate $5\mathrm{e}{-4}$, 32K warmup, and 500K updates (Zhang et al., 22 Aug 2025).

Stage 2 converts generated content-style tokens into a mel spectrogram conditioned on a timbre reference. The appendix reports that this acoustic model adopts Vevo’s flow-matching transformer, with 16 transformer layers, 16 attention heads, hidden dimension 1024, 363M parameters, AdamW, peak learning rate $7.5\mathrm{e}{-5}$, 32K warmup, and 700K updates. An additional training trick is REPA, which aligns the hidden output of layer 5 with W2v-BERT 2.0 features during training. The paper states that a flow-matching transformer is used, but does not provide the core flow-matching differential equation or velocity loss formula in the provided text; that omission is itself part of the technical record (Zhang et al., 22 Aug 2025).

The final waveform is produced by a Vocos-based vocoder initialized from Vevo’s speech vocoder and fine-tuned on unified speech plus singing data. Its reported size is 255M parameters, with 572K fine-tuning updates. This decomposition preserves a clear separation of concerns: the AR stage handles text, prosody, and style; the FM stage handles timbre and acoustic realization; and the vocoder performs the final waveform reconstruction (Zhang et al., 22 Aug 2025).

A later comparative paper summarizes Vevo2 from the outside as a token-based autoregressive model with disentangled timbre and melody control, and further describes it as a multi-stage architecture in which an autoregressive LLM handles melody and content generation while a dedicated CFM focuses on timbre reconstruction (Hao et al., 25 Mar 2026). That description is consistent with the two-stage organization reported in Vevo2’s own paper.

4. Unified prosody learning and post-training alignment

The distinguishing training idea in Vevo2 is unified prosody learning during AR pre-training. The model does not reserve implicit prosody learning only for speech and explicit prosody learning only for singing. Instead, for every training sample, speech or singing, the paper states that it randomly chooses explicit prosody learning or implicit prosody learning with equal probability. This is intended to reduce domain-specific training patterns and encourage a unified latent organization across speech and singing (Zhang et al., 22 Aug 2025).

In implicit prosody learning, the model predicts content-style tokens from text alone. The training sequence is

$$[\mathbf{I}_{ipl}, \mathbf{T}, \langle|\text{start-of-cs}|\rangle, \mathbf{Q}_{cs}(u), \langle|\text{end-of-cs}|\rangle].$$

The instruction is: “User will provide you with a text. Please vocalize it with natural expression.” In explicit prosody learning, the model receives text and prosody tokens and predicts content-style tokens from both:

$$[\mathbf{I}_{epl}, \mathbf{T}, \langle|\text{start-of-p}|\rangle, \mathbf{Q}_{p}(u), \langle|\text{end-of-p}|\rangle, \langle|\text{start-of-cs}|\rangle, \mathbf{Q}_{cs}(u), \langle|\text{end-of-cs}|\rangle].$$

The instruction is: “User will provide you with a text. Please first generate a good prosodic instruction, then vocalize the text based on it.” (Zhang et al., 22 Aug 2025)

After pre-training, the paper reports that the AR model is versatile but not stable enough in text following, prosody following, and robustness to out-of-distribution prosody prompts such as instruments. Vevo2 therefore adds post-training with a multi-objective alignment task optimized via GRPO. One reward model is trained on 250K intelligibility preference speech pairs using a Bradley–Terry loss. A second reward measures prosody similarity by rendering singing MIDI as instrumental audio, extracting prosody tokens, generating predicted content-style tokens, decoding them back to reconstructed chromagram, and computing cosine similarity with the ground-truth chromagram from the original singing voice (Zhang et al., 22 Aug 2025).

For each prompt, the policy generates $K$ completions. Let $$\mathbf{E} = [\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_K],$$0 be intelligibility rewards and $$\mathbf{E} = [\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_K],$$1 be prosody similarity rewards. The advantage is defined as

$$\mathbf{E} = [\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_K],$$2

The paper’s interpretation is that each completion is judged relative to the others in its sample group, and that the final optimization pressure balances intelligibility and prosody similarity rather than over-optimizing only one of them (Zhang et al., 22 Aug 2025).

The data regime is correspondingly large. Pretraining uses 101K hours of speech from Emilia and 7K hours of singing prepared from in-the-wild Internet songs via SingNet’s source-separation pipeline. Post-training prompts include 20K speech samples from INTP, 20K singing samples from M4Singer, and MIDI-rendered instrument sounds from M4Singer MIDI using 16 instruments spanning piano, strings, woodwinds, brass, folk, and voice-like MIDI instruments (Zhang et al., 22 Aug 2025).

5. Controllability and task realizations

Vevo2’s control interface combines four inputs: text or lyrics, a prosodic source, a style reference, and a timbre reference. Different task families are defined by different subsets of these signals. Text-to-speech uses target text and a reference waveform for style and timbre. Singing voice synthesis uses lyrics, MIDI, and a reference singer waveform, with MIDI first rendered to piano or instrument audio so that prosody tokens can be extracted. Voice conversion can be style-preserved by using only the FM stage, or style-converted by using AR plus FM so that source audio supplies content and coarse prosody while a reference supplies style and timbre. Editing takes edited text and raw audio, preserving original prosody or melody and regenerating content-style tokens for the new text. Humming-to-singing and instrument-to-singing use humming or instrument audio as the prosodic source and a singer reference waveform as the timbre target (Zhang et al., 22 Aug 2025).

The paper is explicit that style control is reference-based rather than language-based. Speech style examples include accent, emotion, and whisper-to-normal style. Singing style examples include vibrato, falsetto, glissando, and broader vocal technique or style. In style-converted singing voice conversion, prosody tokens are expected to carry coarse-grained melody contour, while reference content-style tokens contribute fine-grained style details such as vibrato (Zhang et al., 22 Aug 2025).

Two further controls are noteworthy. First, duration control exploits the fixed 2:1 ratio between content-style and prosody token lengths; the paper reports duration consistency figures for speech editing, singing lyric editing, and singing voice conversion. Second, pitch region control shifts source pitch before prosody extraction, which improves speaker similarity in voice conversion and singing voice conversion at the cost of a slight WER increase because shifted inputs are somewhat out of the training distribution (Zhang et al., 22 Aug 2025).

A later evaluation paper provides an operational view of Vevo2 in singing editing. There, Vevo2 is used in two relevant modes: in “Sing Edit,” the same clip provides both timbre and melody; in “Melody Control,” separate clips can be used for melody reference and timbre reference. That paper treats Vevo2 as the most comparable baseline because it supports melody-controllable generation without requiring manual alignment (Hao et al., 25 Mar 2026).

6. Empirical findings, comparative position, and limitations

The Vevo2 paper’s first major empirical claim is that unified speech-plus-singing modeling yields mutual benefits. On expressive speech, a Vevo2-base model trained on speech only versus speech plus singing changes WER from 15.52 to 14.32, SIM from 0.677 to 0.681, N-CMOS from -0.84 to -0.49, and SS-CMOS from -0.06 to -0.03. On singing voice, speech-only Vevo2-base versus joint training changes WER from 19.39 to 15.78, N-CMOS from -1.15 to -0.66, and SS-CMOS from -0.65 to -0.17. The final post-trained Vevo2 reports WER 11.48 and SIM 0.689 on expressive speech, and WER 7.66 and SIM 0.725 on singing voice (Zhang et al., 22 Aug 2025).

The system is also evaluated task by task. In zero-shot singing voice synthesis, Vevo2 reports English WER 24.77 and SIM 0.643, Chinese WER 9.83 and SIM 0.669, with Melody-MOS greater than 2 in both languages. In expressive speech editing, it reports WER 16.83 and SIM 0.799, outperforming SSR-Speech and F5-TTS on WER. In singing lyric editing, it reports WER 17.98 and SIM 0.848. In singing voice conversion, Vevo2-FM is used to probe timbre conversion directly, while the full Vevo2 adds AR text and style modeling; the paper interprets this comparison as evidence that the tokenizer-acoustic pipeline has strong timbre disentanglement and that the AR stage trades some fine melody fidelity for substantially better intelligibility and style conversion (Zhang et al., 22 Aug 2025).

On voice conversion, Vevo2 reports English WER 3.53 and SIM 0.692, and Chinese WER 3.01 and SIM 0.755. On accent and emotion conversion, it is compared against Vevo and changes accent conversion WER from 30.37 to 7.73 and emotion conversion WER from 13.31 to 7.89 while keeping style similarity comparable. The post-training ablation is especially central: in humming-to-singing, WER changes from 32.86 for the base model to 17.49 with intelligibility-only post-training and 17.27 with the full objective, while FPC changes from 0.769 to 0.774 to 0.784; in instrument-to-singing, WER changes from 40.38 to 20.03 to 17.94 and FPC from 0.716 to 0.731 to 0.745. The paper also notes that melody accuracy drops from 65.0% to 50.0% with intelligibility-only post-training, whereas the joint objective raises text accuracy to 90.0% and also improves melody accuracy (Zhang et al., 22 Aug 2025).

Later work positions Vevo2 as a strong but imperfect reference system. YingMusic-Singer describes Vevo2 as the most comparable baseline supporting melody control without manual alignment and reports that Vevo2 retains higher speaker similarity, which that paper attributes to its multi-stage architecture. The same comparison argues, however, that Vevo2 suffers from reduced intelligibility, poor melody adherence, incomplete melody disentanglement, hallucinations, unfaithful lyric rendering, melodic misalignment, and higher variance in subjective ratings, especially on translation and code-mixing tasks (Hao et al., 25 Mar 2026). These claims should be read as comparative assessments from a later paper rather than as statements made by Vevo2’s authors.

The limitations reported in Vevo2 itself are narrower but substantial. The framework does not support natural-language style or prosody control such as “sing with a joyful tone” or “use stronger vibrato”; it relies on audio references. It is mainly intended for monophonic melody and may struggle with complex polyphonic instrumental inputs and richer musical structures. Pitch shifting improves speaker similarity but can slightly hurt WER because of train-inference mismatch. The authors also note that the pretrained AR model is versatile but less stable in text and prosody following before post-training, and they acknowledge voice-cloning misuse risks, stating that they plan to build a strong classifier to distinguish real speech from Vevo2-generated audio (Zhang et al., 22 Aug 2025).

Taken together, these details place Vevo2 as a general controllable vocal generation system rather than a task-specific model. Its distinctive contribution is not merely that it can both speak and sing, but that it organizes speech and singing around unified prosody learning, low-rate content-style tokenization, AR control over text, prosody, and style, and flow-matched timbre-conditioned acoustic rendering. In that sense, Vevo2 extends the Vevo family from controllable zero-shot voice imitation into a broader shared framework for synthesis, conversion, and editing across speech and singing (Zhang et al., 22 Aug 2025, Zhang et al., 11 Feb 2025).