YingMusic-Singer: Audio-Guided Singing Synthesis

• YingMusic-Singer is a reference-audio-conditioned singing synthesis system that extracts melody directly from audio using weak supervision.
• The system employs diffusion transformers with implicit CKA-based alignment and reinforcement learning to balance melodic fidelity and lyric clarity.
• It uses a multi-stage training curriculum, combining supervised fine-tuning and GRPO to enhance zero-shot synthesis, controllable lyric editing, and robustness.

YingMusic-Singer is a melody-guided family of singing voice synthesis systems that targets zero-shot synthesis, lyric adaptation, and controllable lyric editing without manual phoneme-level alignment or manually annotated melody contours. In the cited literature, the name is associated with two closely related formulations: a 2025 system for zero-shot singing voice synthesis and editing with annotation-free melody guidance, and a 2026 fully diffusion-based system for controllable singing voice synthesis with flexible lyric manipulation and a dedicated lyric-editing benchmark (Zheng et al., 4 Dec 2025, Hao et al., 25 Mar 2026). Across both formulations, the defining premise is that a model should accept reference singing audio and target lyrics, infer melody directly from audio, and generate intelligible singing that preserves melody and timbre under practical editing conditions.

1. Problem setting and scope

YingMusic-Singer is positioned against two recurrent deployment bottlenecks in singing voice synthesis: dependence on accurate phoneme-level alignment and dependence on manually annotated melody or pitch contours. The 2025 formulation argues that conventional SVS pipelines usually require precise duration labels, score or MIDI annotations, and often inference-time control signals that are expensive to obtain and brittle when lyrics, languages, or song structure change. The 2026 formulation sharpens the target task further: given an optional timbre reference, a melody-providing singing clip, and modified lyrics, regenerate singing that preserves the original melody and rhythm while faithfully rendering substitutions, deletions, insertions, translations, and code-mixing, again without manual alignment (Zheng et al., 4 Dec 2025, Hao et al., 25 Mar 2026).

A common simplification is to treat YingMusic-Singer as an ordinary score-conditioned SVS model. The papers instead define it as a reference-audio-conditioned system: melody is extracted from a sung prompt rather than supplied as hand-labeled MIDI, and alignment behavior is learned from weak or coarse supervision rather than from phoneme-level duration annotations. This suggests a shift from symbolic-score dependence toward audio-native controllability.

Version	Core focus	Distinctive elements
2025 YingMusic-Singer	Zero-shot singing voice synthesis and editing	DiT backbone, online melody extraction, weakly annotated duration learning, Flow-GRPO
2026 YingMusic-Singer	Controllable SVS with flexible lyric manipulation	Fully diffusion-based CFM, IPA tokenizer, sentence-level alignment, LyricEditBench

2. Architectural organization

The 2025 system is an end-to-end melody-driven SVS framework built around a Diffusion Transformer backbone plus an online melody extraction module. It takes an audio prompt, lyrics, and a melody representation extracted directly from reference audio, then generates singing as a conditional denoising diffusion process. Its decoder follows an F5-TTS-style DiT initialized from a pretrained DiT TTS model, with 12 decoder layers, hidden size 1024, 16-head self-attention with 64 dimensions per head, and about 0.3B parameters. The melody module is an online encoder $E_{\phi}$ that maps raw audio $x$ to a frame-level melody sequence $m_e = E_{\phi}(x)$, with $m_e \in \mathbb{R}^{T \times D_m}$, and is trained jointly with the synthesis model rather than frozen as preprocessing (Zheng et al., 4 Dec 2025).

The 2026 system is explicitly described as fully diffusion-based and is organized around four components: a Stable Audio 2-style VAE for latent audio representation, a Melody Extractor, an IPA tokenizer for lyrics, and a DiT-based conditional flow matching backbone. The waveform input is stereo at 44.1 kHz, encoded to $$\mathbf{z} = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{T' \times D}$$ with downsampling factor 2048. Melody guidance is produced by a pretrained MIDI-extraction encoder, yielding $\mathbf{h} = \mathcal{M}(\mathbf{M})$, then interpolated to the latent frame rate. Lyric control is injected through a frame-level IPA-aligned embedding $\mathbf{e}$, while partially masked latent frames $\mathbf{z}_{\text{ctx}}$ act as timbre context. The final condition is the concatenation $$\mathbf{c} = [\tilde{\mathbf{h}};\mathbf{e};\mathbf{z}_{\text{ctx}}]$$. Implementation details specify an F5-TTS-derived DiT with 22 layers, 16 heads, hidden dimension 1024, lyric embedding dimension $D_e = 512$, about 727.3M parameters, 32 ODE steps, and classifier-free guidance scale 3 (Hao et al., 25 Mar 2026).

These two formulations are architecturally different in scale and latent representation, but they share the same conditioning doctrine: melody is extracted from audio, text is aligned coarsely rather than by phoneme durations, and timbre is preserved through prompt-derived context rather than through manually curated control tracks.

3. Melody representation, alignment, and duration inference

A central technical feature of YingMusic-Singer is annotation-free melody guidance. In the 2025 system, melody representation learning is teacher-guided: a frozen pre-trained teacher MIDI melody model from openvpi/SOME supervises the student melody extractor through

$x$0

with $x$1. The full base objective is

$x$2

where $x$3 and $x$4 is scheduled from 0.3 down to 0.01 over the first 2.5k steps. To enforce actual melody following rather than weak conditioning, the model introduces an implicit alignment mechanism based on linear CKA between the extracted melody representation and an intermediate flow feature $x$5: $x$6 The paper describes this as a similarity-distribution constraint that improves melodic stability and coherence (Zheng et al., 4 Dec 2025).

The same section of the 2025 design also treats duration modeling as an implicit inference problem. Training uses weakly annotated song data; lyrics are padded with the DiffRhythm strategy, and at inference a single timestamp separates prompt and generated content. Rather than learning a separate duration predictor from phoneme labels, the system learns lyric-to-time allocation from sentence-level timestamps and melodic context, which the paper presents as a way to avoid lyric squeezing, beat drift, and abrupt phrasing.

The 2026 formulation retains the CKA theme but embeds it in a different control stack. Lyrics are converted to a unified IPA phoneme sequence for Chinese and English, then placed into a padded frame-level sequence by sentence-level alignment inspired by DiffRhythm. Melody conditioning is regularized by temporal dropout, introduced specifically to prevent the melody extractor from leaking semantic content from the melody clip into the generation path. During supervised singing adaptation, melody correlation is again reinforced through CKA: $x$7 The accompanying ablation is conceptually important: removing temporal dropout sharply worsens phoneme error rate, which the authors interpret as evidence that unregularized melody representations can carry residual linguistic content and undermine genuine lyric generation (Hao et al., 25 Mar 2026).

4. Training curriculum and reinforcement learning

The 2025 YingMusic-Singer couples supervised diffusion training with post-training reinforcement learning via Flow-GRPO. The flow ODE is reinterpreted as a stochastic policy,

$x$8

with randomness injected at a single uniformly sampled timestep to reduce credit-assignment ambiguity. The reward is explicitly multi-objective: $x$9 with equal weights $m_e = E_{\phi}(x)$0. The two concrete rewards are content accuracy and melody similarity. Content reward is defined from ASR word error rate by $m_e = E_{\phi}(x)$1, and melody reward is the Pearson correlation between generated and target $m_e = E_{\phi}(x)$2 contours. The stated purpose of this stage is to jointly enhance pronunciation clarity and melodic fidelity, particularly in zero-shot synthesis and lyric editing (Zheng et al., 4 Dec 2025).

The 2026 system implements a more explicit curriculum. Stage 1 is TTS pretraining on Emilia Chinese and English speech data without melody conditioning, intended to build phonetic and articulatory priors. Stage 2 is singing supervised fine-tuning in two phases: first, singing adaptation with sentence-level alignment and melody conditioning disabled; second, activation of melody conditioning plus CKA. Stage 3 applies Group Relative Policy Optimization rather than PPO or DPO. The model computes normalized advantages across multiple reward models,

$m_e = E_{\phi}(x)$3

and optimizes a GRPO objective with KL regularization to the reference model. In the authors’ interpretation, supervised fine-tuning improves either melody or intelligibility depending on the balance of losses, whereas GRPO is what resolves that trade-off at the perceptual level (Hao et al., 25 Mar 2026).

Taken together, the two versions indicate a stable design principle: alignment-free conditioning is not sufficient by itself, because melody preservation and lyric fidelity compete. YingMusic-Singer therefore uses RL not as an auxiliary refinement, but as a mechanism for balancing incompatible optimization targets.

5. Data, benchmarks, and evaluation protocol

The 2025 paper trains on about 3.7K hours of Mandarin singing vocals extracted from internet songs, processed at 24 kHz and segmented into roughly 30-second clips. It further selects about 500 hours of higher-quality data using DNSMOS, WER, and aesthetic score filtering. Evaluation uses 60 clips from GTSinger for SVS and five unseen singers for zero-shot testing. For editing, the authors build two specialized datasets: one for lyric editing while keeping structure and character count fixed, and another allowing both structure and character-count changes; the edited lyrics are generated with DeepSeek-V3.2. Objective metrics are WER, speaker similarity SIM from cosine similarity on WavLM-TDNN embeddings, and F0 Pearson correlation FPC. Subjective metrics are N-CMOS, Melody-MOS, and comparative aesthetic scores CE, CU, PC, and PQ. Baselines include TCSinger and Vevo (Zheng et al., 4 Dec 2025).

The 2026 paper broadens the data regime substantially. TTS pretraining again uses Emilia Chinese and English speech subsets. Singing supervised fine-tuning uses internally licensed music tracks segmented by SongFormer, with non-vocal regions removed, vocal stems isolated by Mel-band RoFormer, and clips retained if 2–30 seconds; the final singing training set is 33,562.6 hours. GRPO uses a further curated subset of about 20,240 clips filtered by ASR transcript verification with WER $m_e = E_{\phi}(x)$4, single-speaker diarization via pyannote, and DNSMOS P808 quality threshold $m_e = E_{\phi}(x)$5. The paper also introduces LyricEditBench, described as the first benchmark for melody-preserving lyric modification evaluation. Constructed from GTSinger with DeepSeek-V3.2-generated lyric edits, it covers six scenarios—PSub, FSub, Del, Ins, Trans, and Mix—and yields 11,535 valid samples, later downsampled to 7,200 balanced test instances. Evaluation uses PER, SIM, F0-CORR, Vocal Score, N-MOS, and M-MOS, with Vevo2 as the main comparable baseline (Hao et al., 25 Mar 2026).

This benchmarking strategy is methodologically significant. Rather than evaluating only generic SVS quality, YingMusic-Singer is measured on lyric manipulation under melody constraints, which makes lyric intelligibility and melody adherence first-class criteria rather than secondary observations.

6. Empirical results, ablations, limitations, and context

The 2025 YingMusic-Singer reports its strongest quantitative gains in zero-shot and editing settings. In zero-shot SVS, WER is 1.28 for YingMusic-Singer versus 3.47 for TCSinger and 9.83 for Vevo, with SIM 93.95 and FPC 81.28. In singing voice editing, WER drops from Vevo’s 29.89 to 16.58 in lyric editing and from 30.63 to 18.44 in structural editing. In zero-shot singing voice editing, WER drops to 15.18 for lyric editing and 12.62 for structural editing, versus Vevo’s 67.31 and 73.97. Subjectively, zero-shot editing improves from $m_e = E_{\phi}(x)$6 to $m_e = E_{\phi}(x)$7 in N-CMOS and from $m_e = E_{\phi}(x)$8 to $m_e = E_{\phi}(x)$9 in Melody-MOS. Ablation results indicate that removing post-training worsens WER from 15.18 to 16.75 and FPC from 82.84 to 76.64, while removing CKA raises WER to 16.49 and slows convergence to melody guidance; the authors further note that overly strong CKA can improve F0 correlation at the expense of intelligibility (Zheng et al., 4 Dec 2025).

The 2026 paper reports consistent superiority over Vevo2 on LyricEditBench across all six editing types, in both Chinese and English, under both Melody Control and Sing Edit settings. The reported pattern is large PER reduction, higher F0-CORR, and substantially better Vocal Score, with subjective gains as well: for example, under Melody Control in Chinese, N-MOS rises from 4.25 to 4.31 and M-MOS from 4.28 to 4.44; under Sing Edit in English, N-MOS rises from 4.44 to 4.55 and M-MOS from 4.50 to 4.58. The ablation trajectory is also structurally informative: TTS pretraining provides phonetic priors but almost no singing capability, supervised singing adaptation drives major PER improvement, melody-conditioned SFT pushes F0-CORR above 0.92 while partially degrading PER, and GRPO restores lyric fidelity while preserving or improving melody adherence and Vocal Score. Removing temporal dropout produces a marked PER regression, reinforcing the claim that annotation-free melody guidance must be semantically sanitized to function as control rather than as covert text leakage (Hao et al., 25 Mar 2026).

Several limitations are explicit or strongly implied. The 2025 results are demonstrated mainly on Mandarin singing and on tasks derived from existing singing data, so multilingual robustness and studio-grade audio fidelity remain future work. The same paper also acknowledges a persistent trade-off between melody adherence and intelligibility. The 2026 formulation narrows that trade-off more effectively, but it does so with a much larger model and a multi-stage pipeline dependent on reward design, curated RL data, and a melody-providing clip. Speaker similarity is also not uniformly maximal; the paper notes that Vevo2 can sometimes retain an advantage because of its modular separation of timbre and content (Zheng et al., 4 Dec 2025, Hao et al., 25 Mar 2026).

Within the broader SVS literature, YingMusic-Singer occupies a specific position. TCSinger emphasizes zero-shot style transfer and multi-level style control, including singing method, emotion, rhythm, technique, and pronunciation, whereas DiTSinger emphasizes scaling, score conditioning, and implicit alignment under large Chinese SVS corpora. YingMusic-Singer differs by centering reference-audio melody extraction and lyric manipulation without manual alignment, making editing rather than only synthesis the primary application target (Zhang et al., 2024, Du et al., 10 Oct 2025).