CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

Abstract: Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications. Demo page is available at https://haoweilou.github.io/CookVoice/.

• The paper introduces a 43.51M-parameter non-autoregressive framework that unifies text-to-speech and singing generation by aligning content, prosody, and style at the acoustic-frame level.
• CookVoice substantially improves controllability, reaching 91.65% style similarity and 0.7102 F0 correlation for controlled TTS, while achieving 95.00% style similarity and 0.8425 F0 correlation for controlled singing.
• The model delivers these controls efficiently at an RTF of 0.04 using four ODE steps, but its perceptual quality remains slightly below larger autoregressive systems such as IndexTTS and F5-TTS.

Motivation and problem statement

Voice generation research has produced strong task-specific systems—autoregressive (AR) zero-shot TTS models such as CosyVoice and IndexTTS, flow-matching systems such as F5-TTS, and score-conditioned singing voice synthesis (SVS) systems such as DiffSinger, StyleSinger, and TCSinger—but these systems remain architecturally fragmented. AR models determine duration and prosody implicitly through token prediction, making frame-level temporal control impractical; non-autoregressive (NAR) systems impose restricted alignments (phoneme-level durations, note-to-phoneme mappings, or utterance-level prosody references) that limit flexibility when multiple control signals must be combined. CookVoice (2608.11590) addresses this fragmentation with a unified NAR framework that decomposes the voice into three factors—content, prosody, and style—and supports both text-to-speech (TTS) and text-to-singing voice (TTSV) within a single 43.51M-parameter model.

Problem formulation

CookVoice formulates human voice generation as conditional latent acoustic generation. A HiFi-GAN-style autoencoder compresses linear spectrograms into latent embeddings $Y \in \mathbb{R}^{N \times T}$, which serve as generation targets. Three conditioning factors are defined:

• Content $X \in \mathbb{R}^{L_1}$: a phoneme sequence derived from text or lyrics via a multilingual G2P module.
• Prosody $\mathcal{P}$: either discrete signals—lexical tone/stress tokens aligned with phonemes, or musical note tokens with a sequence length $L_2$ that may differ from $L_1$—or a continuous frame-level $F_0$ contour $\in \mathbb{R}^T$ extracted from a reference voice.
• Style $S \in \mathbb{R}^D$: a global paralinguistic embedding sourced from either a text description (encoded with frozen MPNet plus a trainable projection) or a reference voice (encoded by a Transformer encoder with attentive pooling over the latent acoustic representation).

Tasks are realized purely by conditioning combinations rather than task-specific heads: TTS, style-controllable TTS (text or voice), prosody-controllable TTS, TTSV and its style-controllable variants, prosody mimicry, voice/singing conversion, editing, sketch-to-voice, and humming-to-voice.

Flexible frame-level alignment

The central architectural contribution is an alignment strategy that expands all control signals to the acoustic frame length $T$, replacing both implicit AR alignment and restricted NAR alignments. For speech, ground-truth phoneme durations are obtained from pre-trained ParaStyleTTS aligners at training time, and a ParaStyleTTS duration predictor is used at inference. For singing, durations are deterministically derived from musical scores: the relative beat proportion of each phoneme is scaled by the total frame count $T$, avoiding any learned note-to-phoneme binding. Each phoneme embedding is repeated according to its duration $X \in \mathbb{R}^{L_1}$0 (with $X \in \mathbb{R}^{L_1}$1), producing $X \in \mathbb{R}^{L_1}$2; discrete prosody tokens are expanded identically. This permits a single note to span multiple phonemes or vice versa, a case that fixed note-phoneme mappings in prior SVS systems cannot represent.

A notable preprocessing decision addresses style–prosody entanglement: $X \in \mathbb{R}^{L_1}$3 is log-scaled, normalized (floor 50 Hz), and converted to a relative pitch contour by subtracting the voice-level mean, with unvoiced frames set to $X \in \mathbb{R}^{L_1}$4. Since absolute pitch correlates with paralinguistic attributes such as gender and age already captured by the style embedding, using absolute $X \in \mathbb{R}^{L_1}$5 as a condition would create interference; the relative formulation ensures the explicit $X \in \mathbb{R}^{L_1}$6 input controls only intonation and melody.

Generative backbone and multi-task training

The expanded condition $X \in \mathbb{R}^{L_1}$7 is fed via cross-attention to a DiT-S backbone trained under optimal-transport flow matching, regressing the vector field between a Gaussian prior and the target latent with an MSE objective. Inference solves the probability-flow ODE with a first-order Euler solver, requiring as few as 4 steps.

Multi-task capability is achieved through condition randomization rather than multi-objective training: within each batch, the style source is sampled uniformly between text and voice modalities, and the prosody source uniformly between discrete and continuous representations, at the sample level. A single model therefore learns all four style–prosody combinations without architectural or objective changes.

Experimental results

Training used approximately 168 hours of bilingual (English/Chinese) speech and singing data from seven public corpora (123k samples, 6,361 speakers), with a singing-to-speech ratio of 1:9, on a single RTX 5090 for 800K steps. Evaluation combines MOS and Melody Comparative MOS (M-CMOS) listening tests with objective metrics: WER/PhoER/ProER via Whisper transcription, style similarity (S-SIM) via a pre-trained style encoder, and F0-RMSE/F0-CORR for prosody fidelity.

Controllability is the headline result. Under matched style–prosody settings, CookVoice substantially outperforms all baselines. For TTS with voice-based style and continuous $X \in \mathbb{R}^{L_1}$8 control, it reaches S-SIM of 91.65% and F0-CORR of 0.7102, versus roughly 75% and 0.25 for the best TTS baseline (Vevo2). Relative gains over the best matched baselines include +41.48% S-SIM and +121.10% F0-CORR for text-style/discrete-prosody TTS, and −28.41% F0-RMSE with +18.25% F0-CORR for voice-style/continuous-prosody TTSV, where CookVoice attains 95.00% S-SIM and 0.8425 F0-CORR. For singing, CookVoice also achieves the highest MC-MOS (0.28), indicating the best perceived melody consistency.

Quality is competitive but not leading. The paper is explicit that CookVoice's TTS MOS (best 3.98, against a 4.05 ground truth) trails large AR systems such as IndexTTS (4.42) and F5-TTS (4.35), and its TTSV MOS (3.40) is marginally below Vevo2 (3.42). Intelligibility is mid-range: TTSV WERs of roughly 10–24% are partly attributed to Whisper being speech-trained, though ground-truth English singing WER is itself 14.22%, supporting this explanation.

Control signal analysis yields two findings worth noting. Voice-based style conditioning dominates text-based conditioning for style transfer (normalized S-SIM of 0.932 vs 0.199 in TTS), but in TTSV, text conditioning yields better English lyric WER, indicating stronger acoustic imitation does not necessarily improve intelligibility. Continuous $X \in \mathbb{R}^{L_1}$9 control dominates discrete control on prosody metrics in both tasks, yet discrete notes yield higher S-SIM and better English WER in TTSV; replacing notes with continuous $\mathcal{P}$0 under text style conditioning drops S-SIM from 85.75% to 72.51%, suggesting note tokens act as structured regularizers that protect style and articulation.

Efficiency. Against Vevo2, the closest unified baseline, CookVoice uses 4.99% of the parameters, 20.03% of the CUDA memory, and 0.27% of the real-time factor (RTF 0.04 vs 14.85). An ODE-step ablation shows style and prosody metrics converge around 4–8 steps, while intelligibility metrics peak at 4 steps and degrade with more refinement—over-refinement trades linguistic clarity for acoustic detail—motivating 4–8 steps as the recommended operating range.

Limitations and open questions

The authors concede two limitations directly. First, the model is deliberately small: only the DiT-S configuration (43.51M parameters) trained on 168 hours has been evaluated, so the scaling behavior of the frame-level alignment approach relative to large-scale AR systems—whose perceptual quality advantage persists in these results—remains untested. Second, the framework has been applied only to human voice; extension to music, instrumental, or general audio generation is unexplored. Two further questions are left open by the results themselves: whether the TTS quality gap to large AR systems can be closed within this architecture, and whether the observed tension between continuous prosody control and style preservation under text conditioning can be resolved.

Conclusion

CookVoice demonstrates that decomposing voice generation into content, prosody, and style with explicit frame-level alignment of heterogeneous control signals yields a single compact model covering both speech and singing generation. The empirical trade-off is clearly characterized: markedly superior controllability and efficiency (RTF 0.04 at 43.51M parameters) at the cost of a modest perceptual-quality gap to the largest AR baselines, with the scaling behavior of the approach identified by the authors as the principal open question.