VoxCPM2 Technical Report

Abstract: We present VoxCPM2, a https://info.arxiv.org/help/prep#abstractsfully open-source multilingual and controllable speech generation foundation model that extends the hierarchical diffusion-autoregressive modeling paradigm of VoxCPM. VoxCPM2 advances the framework in three key dimensions: (i) capability, by unifying 30 languages, 9 Chinese dialects, natural-language voice design, style-controllable voice cloning, and high-fidelity continuation cloning within a single backbone; (ii) quality, through an asymmetric AudioVAE that encodes at 16 kHz and reconstructs at 48 kHz, enabling implicit super-resolution with high encoding efficiency; and (iii) scale, by jointly scaling the model to 2B parameters and the training data to over 2 million hours of multilingual speech. To support these diverse capabilities within one model, we introduce a unified sequence organization that expresses all generation modes through different arrangements of the same input building blocks, allowing joint training under a single set of parameters and objective. VoxCPM2 achieves state-of-the-art or competitive performance on public zero-shot and instruction-following TTS benchmarks. On our internal 30-language evaluation set, it attains an average WER of 1.68%. These results demonstrate that hierarchical continuous-latent modeling, without relying on any external discrete speech tokenizer, offers a viable and powerful foundation for large-scale multilingual and controllable speech generation. The model weights, fine-tuning code, and inference tools are publicly released under the Apache 2.0 license to foster community research and development.

• The paper introduces a unified continuous-latent TTS model that leverages a hierarchical diffusion-autoregressive architecture for multilingual and controllable speech generation.
• The paper employs an innovative asymmetric AudioVAE V2 codec to encode speech efficiently, achieving state-of-the-art WER, CER, and speaker similarity metrics across 30 languages.
• The paper demonstrates a unified sequence design that supports diverse tasks like natural-language voice design and high-fidelity continuation cloning within a single, parameter-shared model.

VoxCPM2: A Unified Hierarchical Continuous-Latent Model for Multilingual and Controllable Speech Generation

Introduction

VoxCPM2 represents an advancement in large-scale text-to-speech (TTS) synthesis and controllable speech generation, continuing the development trajectory established by VoxCPM. Operating entirely in a continuous latent space, VoxCPM2 provides a fully open-source, 2B-parameter, hierarchical diffusion-autoregressive architecture, unifying multilingual TTS, natural-language voice design, style-controllable voice cloning, and high-fidelity continuation cloning within a single, parameter-shared model. The architectural innovations—an asymmetric AudioVAE V2 codec and comprehensive backbone refinements—enable state-of-the-art performance across 30 languages and 9 Chinese dialects, supporting 48 kHz high-fidelity streaming synthesis without reliance on external discrete tokenizers.

Hierarchical Continuous-Latent Architecture

The core technical premise of VoxCPM2 is hierarchical continuous-latent modeling. The speech signal is encoded into a low-rate continuous latent representation (25 Hz) via AudioVAE V2; every four frames are grouped into a patch (corresponding to 160ms) yielding a 6.25 Hz sequence for autoregressive language modeling. The backbone comprises:

• Local Encoder (LocEnc): Aggregates patch-level acoustic history.
• Text-Semantic LLM (TSLM): Autoregressively models high-level semantic and prosodic structure, implemented on a scaled MiniCPM-4 backbone.
• Differentiable Semi-Discrete Bottleneck (FSQ): Compresses TSLM states into a robust, quantized semantic skeleton using channel-wise finite scalar quantization.
• Residual Acoustic LLM (RALM): Recovers fine-grained acoustic details conditioned on both the semantic skeleton and local acoustic context. Notably, positional encoding is omitted (NoPE) in RALM, improving long-utterance stability.
• Local Diffusion Transformer (LocDiT): Performs per-patch diffusion and denoising in the continuous latent domain, using multi-token conditioning to maintain high semantic and acoustic fidelity.

  Figure 1: Overall architecture of VoxCPM2, illustrating the hierarchical backbone and flow of information across TSLM, FSQ, RALM, and LocDiT modules.

The architectural refinements include replacing additive fusion (VoxCPM) with learnable concatenation-projection, expanding FSQ bottleneck width to accommodate broader linguistic diversity, and separating input for speaker reference via a delimited segment in the sequence.

AudioVAE V2: Asymmetric Latent Codec

AudioVAE V2 is critical for balancing compact latent representation and high output fidelity. The encoder operates at 16 kHz, ensuring compatibility with most large-scale datasets and efficient latent length; the decoder reconstructs audio up to 48 kHz, yielding superior super-resolution and waveform quality. This allows the backbone to operate over a dense, information-preserving latent, greatly mitigating fidelity losses typically induced by quantization or discrete tokenization.

Unified Sequence Organization for Multimodal Generation

All user-facing generation modes are expressed as permutations of input building blocks (text, reference audio, target audio) with a binary modality indicator. These correspond to:

• Basic multilingual TTS.
• Natural-language-driven voice design (free-form descriptions concatenated to text).
• Reference-based and controllable voice cloning (target speaker identity supplied via audio, with optional style description).
• High-fidelity continuation cloning (paired text+audiobased prefix).

Task unification is achieved without mode-specific modules; the sequence structure itself allows the TSLM to disambiguate conditioning signals under a single objective function and set of parameters. This leads to a robust, plug-and-play framework for extending to new generation configurations.

Training Pipeline, Data, and Curriculum

Training utilizes >2M hours of curated multilingual data (30 languages, 9 Chinese dialects), following a staged curriculum:

1. Multilingual TTS pretraining: Focuses on global pronunciation and prosody via basic TTS and continuation cloning.
2. Joint controllable TTS pretraining: Introduces expressive utterances, diverse style/voice annotations, and explicit reference audio, balancing ratio to preserve baseline TTS performance.
3. High-quality SFT (supervised fine-tuning): Uses a selected set of highly annotated data, especially for controllable attributes and longer contexts.

Voice and style annotations are obtained via large foundation audio understanding models (e.g., Step-Audio R1, Gemini 2.5 Pro), with expert validation for gender, accent, and emotion. Notably, style-content confounding is mitigated by synthesized cross-content training, where style is decoupled from content using model-based resampling.

Empirical Results

VoxCPM2 is extensively evaluated on open benchmarks (Seed-TTS-Eval, CV3-Eval, MiniMax-MLS-Test, InstructTTSEval) and an internal 30-language set. Key results include:

• Zero-shot voice cloning: Competitive or SOTA Word/Character Error Rate (WER/CER) and speaker similarity (SIM) metrics across languages, including challenging cases. Combined reference+continuation conditioning delivers the strongest SIM. For example, on Seed-TTS-Eval, VoxCPM2 achieves 1.84 WER / 75.3 SIM (English), 0.97 CER / 79.5 SIM (Chinese), outperforming most discrete-token models even at smaller parameter counts.
• Multilingual coverage: Maintains WER/CER <3% for 28/30 languages on the internal test set; top similarity across 22/24 MiniMax-MLS-Test languages, with observed zero-shot transfer to unseen languages (e.g., Czech, Romanian).
• Controllable and instruction-following generation: Achieves outstanding instruction adherence and naturalness on InstructTTSEval (84.2/83.2/71.4% on APS/DSD/RP, English), matching or surpassing large-scale, hybrid/ensemble systems.
• AudioVAE V2 fidelity: Closes the gap to high-bandwidth codecs; achieves strong Mel-distance and PESQ/STOI scores in both speech and singing domains, supporting overall downstream system performance.

Inference and Deployment

VoxCPM2 achieves an RTF of 0.13–0.3 (PyTorch, Nano-vLLM implementations) on RTX 4090 for 48 kHz streaming generation, with VRAM usage <8 GB, facilitating cost-effective, real-time applications. Causal and patch-local model structure enables chunk-based streaming and production serving, including batched vLLM-Omni compatibility.

Implications and Future Directions

VoxCPM2 demonstrates the scalability and practical advantages of hierarchical continuous-latent modeling, notably dispelling reliance on external discrete tokenizers for competitive performance. Unified sequence design provides architectural parsimony and extensibility across modes—crucial for future multimodal or cross-domain audio generation systems.

Areas for future exploration include:

• Redressing data imbalances, esp. in low-resource and long-tail languages, to further narrow the cross-lingual performance gap.
• Enhancing instruction-following for abstract/complex natural language attributes, where style cues are subtle.
• Expanding the framework to unified speech, singing, and broader audio modalities.
• Integrating robust content provenance, watermarking, and synthetic voice detection for responsible deployment.

The open release (Apache 2.0) of model weights, code, and adaptation tools positions VoxCPM2 as a practical and composable foundation for research and production.

Conclusion

VoxCPM2 achieves competitive or superior results over strong open and closed-source baselines for multilingual TTS, zero-shot voice cloning, and natural-language controllability, within a single, continuous-latent backbone. The model evidences the maturing potential of non-discrete, end-to-end TTS architectures, setting a precedent for extensible, unified, high-fidelity speech foundation models, and provides a solid platform for further advances in scalable, interpretable, and responsible neural audio generation.