CoMelSinger: Zero-Shot Singing Synthesis

• CoMelSinger is a zero-shot singing voice synthesis system that disentangles melody control from timbre using structured pitch tokens and a contrastive learning strategy.
• It utilizes a two-stage MaskGCT-based pipeline where lyric, pitch, and duration tokens are employed to generate score-faithful singing outputs from a short acoustic prompt.
• The framework demonstrates high performance in melody fidelity and timbre preservation, achieving competitive metrics like F0-RMSE, SECS, and SingMOS in both seen and unseen singer evaluations.

CoMelSinger is a zero-shot singing voice synthesis system that uses discrete tokens to synthesize singing from structured musical inputs while attempting to keep melody control disentangled from timbre prompting. It is designed for the setting in which a short reference prompt from an unseen singer supplies speaker identity and timbre, while the generated output is expected to follow externally specified lyrics, pitch, and duration with high fidelity. The framework is built on the non-autoregressive MaskGCT architecture, replaces conventional text inputs with lyric and pitch tokens, introduces a coarse-to-fine contrastive learning strategy to suppress prosody leakage, and adds an encoder-only Singing Voice Transcription module for frame-level supervision (Zhao et al., 24 Sep 2025).

1. Problem setting and design objective

CoMelSinger addresses zero-shot Singing Voice Synthesis, where the model must synthesize singing in the voice of an unseen singer from a short reference prompt without retraining or speaker-specific fine-tuning. The paper formulates the task around a dual requirement: the system should preserve target timbre from the prompt and simultaneously obey the given lyrics, note pitch, and duration. This combination is technically difficult because prompt-based discrete codec models that work well in speech synthesis do not transfer straightforwardly to singing, where externally specified melody must dominate prosodic realization rather than merely guide it (Zhao et al., 24 Sep 2025).

The central obstacle is prosody leakage. In prompt-conditioned codec synthesis, the acoustic prompt is intended to provide speaker identity, timbre, and style cues, but it can also carry pitch contour, rhythm, and related melodic information. In singing, such leakage is harmful because pitch and duration should come from the score. CoMelSinger is therefore organized around disentangling melody control from prompt conditioning rather than only improving audio generation in a generic sense. This places it within the broader line of discrete token-based synthesis while giving it a distinct objective: precise, score-faithful singing under zero-shot transfer (Zhao et al., 24 Sep 2025).

The system is explicitly positioned against a naive extension of prompt-based text-to-speech codec models. The paper states that simply replacing text with lyrics and pitch inputs in a zero-shot TTS-style codec model is not enough for singing. That diagnosis motivates the framework’s structured conditioning and auxiliary supervision.

2. Core architecture

CoMelSinger uses a two-stage pipeline derived from MaskGCT. The first stage is Text-to-Semantic, and the second is Semantic-to-Acoustic. The model preserves the in-context generation paradigm of discrete codec systems, but restructures conditioning so that melody is supplied through explicit musical inputs rather than left to emerge from prompt prosody (Zhao et al., 24 Sep 2025).

Component	Inputs	Output
Text-to-Semantic (T2S)	lyric tokens, semantic prompt	semantic token sequence
Semantic-to-Acoustic (S2A)	semantic tokens, acoustic prompt, regulated pitch sequence	acoustic tokens
Acoustic decoder	acoustic tokens	waveform

In the T2S stage, the input consists of lyric tokens and a semantic prompt extracted from the reference waveform. In the S2A stage, the input consists of semantic tokens, an acoustic prompt extracted from the same reference waveform, and a regulated pitch sequence. The acoustic decoder then reconstructs the output waveform from the generated acoustic tokens. Lyrics are converted by G2P into lyric tokens, while pitch and duration are converted into a frame-aligned regulated pitch sequence so that pitch supervision shares the time axis of the acoustic tokens (Zhao et al., 24 Sep 2025).

A notable architectural modification is the addition of an embedding layer for the pitch sequence. Pitch and semantic embeddings are combined and used as a melody-aligned conditioning stream. The acoustic prompt remains present because zero-shot timbre transfer depends on it, but the architecture is designed so that prompt information functions as a timbre anchor while pitch tokens function as a melody anchor. The paper characterizes this as structured and disentangled melody control.

3. Structured melody control

The “structured” aspect of melody control refers to the way pitch and duration are represented and aligned. Rather than conditioning only on note-level symbols in an abstract way, CoMelSinger expands pitch tokens according to note durations into a frame-aligned regulated pitch sequence. The total duration is defined as $D=\sum_i m_i^d$, and each pitch token is repeated across its corresponding frame span to produce the regulated sequence used downstream (Zhao et al., 24 Sep 2025).

This design makes S2A conditioning compatible with the temporal granularity of acoustic codec tokens. A note-level score is thus transformed into frame-level supervision without requiring heuristic duration prediction at inference time. The paper states that output length is set directly from the score duration rather than predicted heuristically. A plausible implication is that temporal control becomes substantially more deterministic than in prompt-dominant generation regimes.

The system’s use of lyric tokens, pitch tokens, and duration information differentiates it from conventional text-conditioned discrete codec generation. In the CoMelSinger formulation, lyric content and melody are both primary conditioning variables. This avoids treating melody as an auxiliary feature and instead makes it a first-class control signal throughout synthesis (Zhao et al., 24 Sep 2025).

4. Disentanglement via contrastive learning and SVT guidance

The principal mechanism for disentanglement is a coarse-to-fine contrastive learning strategy. At the sequence level, or “coarse” level, the model constructs sample pairs with the same semantic tokens and the same regulated pitch but different acoustic prompts from the same singer. The objective is to make outputs for the same melody close and outputs for different melodies distant, while treating prompt variation as a nuisance factor. This is intended to preserve global melodic shape independently of prompt-specific prosody (Zhao et al., 24 Sep 2025).

At the frame level, or “fine” level, the model perturbs pitch tokens by shifting 50% of them by a random integer in $[-6,6]$ while keeping durations unchanged. Two versions of the sample are passed through S2A, and a frame-level similarity structure is learned. The strongest similarity target is assigned to frames with the same pitch and the same semantic content; a moderate target $\alpha$ is assigned to the same pitch with different semantics; different pitches receive no similarity target; silence and padding are excluded. This encourages local acoustic states to align with local pitch behavior rather than inherit prompt-driven melodic traces (Zhao et al., 24 Sep 2025).

The second major guidance mechanism is an encoder-only Singing Voice Transcription module. Given discrete acoustic codec tokens, the SVT predicts a frame-aligned pitch token sequence. Its architecture is described as a lightweight encoder-only Transformer with 4 Transformer layers, hidden size 512, and 8 attention heads. Each frame contains 12 discrete acoustic codes, which are embedded, concatenated, projected to 512 dimensions, layer-normalized, and passed through a linear classification head (Zhao et al., 24 Sep 2025).

SVT is trained with cross-entropy against pitch labels derived from the score and duration-aligned expansion, and it also includes a segment transition loss and a soft duration loss. The segment transition loss is designed so that within a sustained note, nearby frames should be similar, while note boundaries should change more sharply. The soft duration loss encourages the model to allocate the right amount of probability mass to each pitch over its expected duration, reducing jittery or fragmented predictions. After separate training, the SVT is frozen and used as a frame-level supervisor for S2A. The paper states that removing SVT causes the largest degradation among the major ablations, indicating that this auxiliary module is not peripheral but structurally important (Zhao et al., 24 Sep 2025).

5. Training procedure, inference, and empirical performance

The system is trained on the Mandarin singing datasets M4Singer and Opencpop, with OpenSinger singers used as unseen targets for zero-shot testing. Audio is downsampled to 24 kHz. Preprocessing extracts semantic tokens, acoustic tokens, and regulated pitch tokens using pretrained MaskGCT codec/tokenizer components. Training proceeds in two phases: SVT is trained first and then frozen; S2A is then fine-tuned using masked token reconstruction, contrastive learning, and SVT supervision. The paper also reports the use of LoRA for efficient adaptation of the DiffLlama-style S2A backbone (Zhao et al., 24 Sep 2025).

Implementation details are unusually explicit. SVT is trained with AdamW, learning rate $1\mathrm{e}{-5}$, cosine schedule, 5k warmup, and 100 epochs. S2A is fine-tuned with AdamW, learning rate $1\mathrm{e}{-5}$, inverse square-root schedule, and 300k steps. The reported batch size is 32, with dropout 0.1, label smoothing 0.1, and gradient clipping. Loss weights are given for the sequence-level contrastive loss, frame-level contrastive loss, overall contrastive loss, segment transition loss, duration loss, and SVT loss. These details indicate that CoMelSinger is not presented merely as a conceptual modification to MaskGCT but as a fully specified training recipe (Zhao et al., 24 Sep 2025).

At inference, the system takes lyric tokens, score pitch and duration, and a short acoustic prompt for timbre, then generates acoustic tokens through MaskGCT-style parallel iterative decoding and reconstructs the waveform from those tokens. Because output length is derived directly from score duration, inference preserves the score’s temporal specification rather than relying on an emergent duration model (Zhao et al., 24 Sep 2025).

The empirical results reported in the summary emphasize melody fidelity and timbre preservation. In seen-singer evaluation, CoMelSinger is compared with DiffSinger, VISinger2, StyleSinger, and SPSinger, and is described as achieving best or near-best subjective quality, best timbre similarity, lowest F0-RMSE, and strong SingMOS and SECS. The reported headline values include F0-RMSE of 0.042, SECS of 0.912, and SingMOS of 4.32. In zero-shot evaluation on unseen singers, the reported values are F0-RMSE = 0.048, SECS = 0.897, SingMOS = 4.25, and SMOS = 4.14. The ablations indicate that removing contrastive learning hurts pitch accuracy and speaker consistency, removing frame-level contrastive learning hurts pitch accuracy more strongly than removing sequence-level contrastive learning, and removing SVT causes the largest degradation (Zhao et al., 24 Sep 2025).

6. Position within adjacent research

CoMelSinger belongs to a broader landscape of systems that couple singing, melody, and controllable generation, but its target problem is narrower and more technically specific than several neighboring lines of work. SingingSDS, for example, is a singing-capable spoken dialogue system organized as an ASR $\rightarrow$ LLM $\rightarrow$ melody control $\rightarrow$ SVS pipeline for conversational roleplay. It is “speech-in, singing-out,” persona-grounded, and oriented toward interactive entertainment, whereas CoMelSinger is a zero-shot SVS framework centered on score-faithful melody control and prompt disentanglement rather than dialogue generation (Han et al., 26 Nov 2025).

Other adjacent systems operate on different representational levels. Harmonizer is an end-to-end prototype for melodic harmonization that allows melody input by singing, instrument performance, MIDI, or manual transcription and generates chord accompaniments using Schoenberg’s chart of regions, Giant Steps, church modes, secondary dominants, and ii–V–I turnarounds. Its problem is melodic harmonization rather than singing voice synthesis (Fernandes, 5 Jan 2025). CM-HRNN is a conditional melody generation model that produces symbolic melodies with long-term structure from chord accompaniments, focusing on hierarchical sequence modeling rather than voice rendering (Guo et al., 2021). “Melody-Lyrics Matching with Contrastive Alignment Loss” introduces melody-lyrics matching as a retrieval task with a dual-encoder, soft-DTW-based contrastive alignment loss, and a sylphone representation, again targeting compatibility rather than acoustic synthesis (Wang et al., 31 Jul 2025). cMelGAN is a genre-conditional Mel-spectrogram generator that demonstrates trade-offs between speed and fidelity in audio-like music generation, but it does not model lyrics, phonemes, pitch contours, or vocal timbre explicitly (Qian et al., 2022).

Against that background, CoMelSinger is most precisely understood as a discrete codec-based zero-shot singing synthesizer whose distinct contribution is not merely discrete token generation, but the combination of frame-aligned pitch conditioning, coarse-to-fine contrastive regularization, and SVT-based guidance to mitigate prompt prosody leakage (Zhao et al., 24 Sep 2025).

7. Limitations and implications

The description indicates several limitations. Evaluation is limited to Mandarin corpora. The method still depends on prompt quality for timbre cloning. The SVT and contrastive design add training complexity. Melody control is improved, but perfect disentanglement is not guaranteed (Zhao et al., 24 Sep 2025).

These constraints delimit the scope of the reported results. A plausible implication is that the framework’s strongest claims concern structured melody control under the tested data regime rather than universal zero-shot singing across languages and performance styles. Even so, the reported ablations and zero-shot results suggest that prompt-based discrete synthesis for singing is not principally constrained by codec modeling alone; it is also constrained by whether the system can explicitly separate score information from prompt prosody. In that sense, CoMelSinger reframes zero-shot SVS as a disentanglement problem as much as a generation problem (Zhao et al., 24 Sep 2025).