LyricEditBench: Melody-Preserving Edit Evaluation

• LyricEditBench is a specialized test-only benchmark for melody-preserving lyric modification in singing synthesis, requiring a fixed melody, timbre prompt, and modified lyrics as input.
• It evaluates models using objective metrics such as PER for lyric adherence, F0-CORR for melody fidelity, SIM for timbre consistency, and vocal quality scores, with additional subjective MOS ratings.
• The benchmark leverages a diverse corpus from GTSinger covering Chinese and English across multiple singer genders, editing types, and singing techniques under Melody Control and Sing Edit protocols.

LyricEditBench is a test-only benchmark for evaluating melody-preserving lyric modification in singing voice synthesis. Each instance requires regeneration of a singing voice from three inputs—an optional timbre reference, a melody-providing singing clip, and modified lyrics—without manual alignment, and the target output must preserve the reference melody and rhythm while faithfully rendering the edited lyrics. It was introduced alongside YingMusic-Singer as the first benchmark “for lyric modification evaluation under matched melody conditions”, addressing a gap left by SVS evaluations that assume full musical scores and aligned phoneme-level annotations or that assess unconditional or score-based synthesis rather than editing (Hao et al., 25 Mar 2026).

1. Definition and motivation

LyricEditBench was created to evaluate a specific task: changing the phoneme sequence of a sung utterance while keeping the original melodic and rhythmic structure. In each test case, the model receives a melody reference clip, a timbre prompt, and modified lyrics. It must output a new singing clip whose melody stays aligned with the reference, whose timbre follows the prompt, and whose phoneme sequence corresponds to the edited text (Hao et al., 25 Mar 2026).

The benchmark is motivated by a gap in prior evaluation practice. Most SVS evaluations either assume full musical score + aligned phoneme-level annotations or focus on unconditional/score-based synthesis quality, not editing. For lyric editing under fixed melody, existing systems either rely on in-context editing in a local region with limited melody control or require manual alignment of modified lyrics to MIDI or timestamps. LyricEditBench was therefore designed to exercise a broad range of lyric edit types under fixed melody and to evaluate both melody preservation and lyric adherence across languages and singing techniques (Hao et al., 25 Mar 2026).

A central premise is that melody-preserving lyric modification couples prosody, phoneme timing, and pitch contour. This is especially acute when edits are large, such as translation and code-mixing. This suggests that generic SVS quality evaluation is insufficient for the task targeted by the benchmark.

2. Corpus construction and benchmark structure

LyricEditBench is derived from GTSinger, described as “A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks.” For benchmark construction, all “Paired Speech Group” content is removed, audio is deduplicated via MD5 hashing, and clips longer than 15 seconds are excluded. The resulting source pool contains singing-only vocal stems under 15 seconds in Chinese (ZH) and English (EN), with multiple singing techniques (Hao et al., 25 Mar 2026).

Modified lyrics are created with DeepSeek V3.2 from the original lyrics and a modification instruction. Outputs that do not comply with the instruction are discarded. After filtering, the construction process yields 11,535 valid samples with compliant modified lyrics (Hao et al., 25 Mar 2026).

The benchmark organizes these samples by singer gender and language, producing four top-level categories: Chinese–male, Chinese–female, English–male, and English–female. Within each category, samples are further organized by modification type. Selection is then constrained by singing techniques: GTSinger defines six singing techniques, plus a technique-free category. For each combination of gender–language category and modification type, the benchmark selects 30 samples per singing technique and 120 samples for the technique-free category, yielding 300 samples per category × type. Since there are 4 categories × 6 types = 24 combinations, the final benchmark contains 7,200 test instances (Hao et al., 25 Mar 2026).

Abbreviation	Task type	Description
PSub	Partial Substitution	Substitute part of the words
FSub	Full Substitution	Completely rewrite the song
Del	Deletion	Delete some words
Ins	Insertion	Insert some words
Trans	Translation	CN ⇔ EN translation
Mix	Code-Mixing	Mixed-language lyrics

For each instance, the timbre prompt is “randomly drawn from the remaining audio pool” and is disjoint from the melody clip. The benchmark therefore encodes the evaluation item as

$$\text{Instance} = (\text{melody reference clip},\ \text{timbre prompt},\ \text{modified lyrics}).$$

The expected output is a new singing audio clip whose melody and rhythm follow the melody reference, whose timbre follows the timbre prompt, and whose phoneme sequence corresponds to the modified lyrics (Hao et al., 25 Mar 2026).

3. Task settings and evaluation protocol

LyricEditBench supports two evaluation protocols. In Melody Control, timbre and melody references can come from different clips; the stress is on decoupling melody from timbre and applying the melody to a timbre prompt while using modified lyrics. In Sing Edit, timbre and melody references may be the same clip, or at least share the same vocal identity, which approximates editing an existing recording. The benchmark itself supports both; these modes are evaluation protocols / model usage rather than separate datasets (Hao et al., 25 Mar 2026).

Objective evaluation uses four metrics, denoted P, S, F, V:

1. Phoneme Error Rate (PER) for lyric adherence and phoneme-level intelligibility.
2. Speaker Similarity (SIM) for timbre consistency with the prompt.
3. F0 Pearson Correlation (F0-CORR) for melody adherence.
4. Vocal Score (VS) for overall vocal quality aligned with human preferences (Hao et al., 25 Mar 2026).

For PER, both Chinese and English clips are transcribed with Qwen3-ASR-1.7B, a singing-trained ASR model. Transcriptions are converted to phoneme sequences with tone markers removed, and PER is computed between the phoneme sequence of the generated audio and the phoneme sequence corresponding to the modified lyrics:

$\text{PER} = \frac{S + D + I}{N}.$

Lower PER indicates better lyric rendering (Hao et al., 25 Mar 2026).

For SIM, speaker embeddings are extracted using a WavLM-large-based speaker verification model, and cosine similarity is computed between the embeddings of the generated clip and the timbre prompt. Higher SIM indicates better timbre consistency. For F0-CORR, frame-wise F0 is extracted from the reference melody clip and generated audio using RMVPE, and Pearson correlation is computed between the two F0 contours. Higher F0-CORR reflects better melody preservation. VS is produced by VocalVerse2, which outputs a normalized scalar score for generated vocal audio, with higher values corresponding to better perceived vocal quality (Hao et al., 25 Mar 2026).

Subjective evaluation complements the objective metrics. The protocol samples 120 samples uniformly across task types and languages, and 30 listeners score each sample on two 5-point scales: Naturalness MOS (N-MOS) for overall perceptual quality and Melody MOS (M-MOS) for faithfulness to the reference melody. Scores are reported as mean ± 95% confidence interval or similar (Hao et al., 25 Mar 2026).

The benchmark is explicitly designed so that no single metric is sufficient. PER targets lyric adherence, F0-CORR melody preservation, SIM timbre alignment, VS perceived vocal quality, and N-MOS/M-MOS human perceptual corroboration. By comparing PER and F0-CORR, one can directly quantify the trade-off between lyric faithfulness and melody fidelity, which the benchmark treats as central to melody-preserving lyric editing (Hao et al., 25 Mar 2026).

4. Coverage, difficulty, and benchmark semantics

LyricEditBench covers Chinese and English, male and female singers, six singing techniques plus a technique-free category, and six editing scenarios: partial substitution, full substitution, deletion, insertion, translation, and code-mixing. This breadth permits evaluation across small edits, large structural rewrites, prosodic changes, cross-lingual adaptation, and mixed-language complexity (Hao et al., 25 Mar 2026).

The benchmark is explicitly challenging for several reasons. First, the lyric edits can be complex: FSub, Trans, and Mix may alter length, language, and phonotactics substantially. Second, the melody is fixed, so a model must fit a potentially very different phoneme sequence into the same melody and rhythm. Third, Trans and Mix require alignment of very different syllabic structures to the same melodic contour. Fourth, the inclusion of multiple singing techniques yields substantial diversity in timbre, range, articulation, and vibrato. Fifth, in Melody Control, timbre and melody may come from different clips, making timbre–melody disentanglement non-trivial (Hao et al., 25 Mar 2026).

The benchmark is test-only. There is no explicit train/validation split defined for the benchmark itself, and the YingMusic-Singer paper states that the authors explicitly exclude the LyricEditBench test set from training. This is important for interpretation: LyricEditBench is framed as an evaluation substrate rather than a supervised training corpus (Hao et al., 25 Mar 2026).

A recurrent misconception is that LyricEditBench is simply a collection of edited lyrics. In fact, each instance is a multimodal test item defined by the triple of melody reference, timbre prompt, and modified lyrics, and the output is audio, not text. Another possible misconception is that the two modes are separate datasets; the paper instead defines them as evaluation protocols / model usage layered over the same benchmark (Hao et al., 25 Mar 2026).

5. Use in YingMusic-Singer and empirical behavior

The YingMusic-Singer paper evaluates YingMusic-Singer and Vevo2 on all 7,200 LyricEditBench instances under both Melody Control and Sing Edit, using PER, SIM, F0-CORR, VS, and subjective N-MOS and M-MOS (Hao et al., 25 Mar 2026).

Vevo2 is described there as a token-based autoregressive model with disentangled timbre and melody control. Other potential baselines are excluded because in-context song editors require manual boundary alignment and are restricted to local edits, while SoulX-Singer needs precise character-level timestamps, which are described as impractical or as conferring an unfair advantage if mined (Hao et al., 25 Mar 2026).

The benchmark reveals a characteristic performance pattern. Qualitatively, YingMusic-Singer shows dramatically lower PER across tasks and languages, consistently high F0-CORR at approximately 0.93–0.96, and higher VS, while Vevo2 often achieves slightly higher SIM. The paper interprets this SIM behavior as a consequence of Vevo2’s architecture, in which an autoregressive LLM handles content and melody while a separate CFM handles timbre. The largest performance gaps appear on Trans and Mix, which are identified as the hardest tasks. The authors also note that some PER inflation on Mix may come from ASR hallucinations, though the overall trend still favors YingMusic-Singer (Hao et al., 25 Mar 2026).

The subjective results follow the same direction. For both Melody Control and Sing Edit, in both ZH and EN, YingMusic-Singer obtains higher N-MOS and especially M-MOS than Vevo2. The paper additionally reports that listeners observed fewer artifacts such as unfaithful lyric rendering and melodic misalignment in YingMusic-Singer outputs (Hao et al., 25 Mar 2026).

LyricEditBench is also used for ablation analysis. The reported ablations examine curriculum training, CKA loss, GRPO, and melody latent perturbation via temporal dropout. The benchmark is sensitive enough to expose several distinct regimes: TTS pretraining alone gives very poor melody adherence; SFT Phase 1 improves PER and VS but leaves poor F0-CORR in Melody Control; SFT Phase 2 adds melody conditioning and CKA, producing F0-CORR above 0.92 but worsening PER; GRPO improves both PER and F0-CORR simultaneously; and w/o Dist yields high F0-CORR but PER collapses, which the paper interprets as semantic leakage from the melody latent rather than proper following of modified lyrics. This suggests that the benchmark is not only comparative but also diagnostically useful (Hao et al., 25 Mar 2026).

6. Relation to adjacent work, accessibility, and limitations

LyricEditBench is distinguished in the YingMusic-Singer paper from prior SVS evaluation protocols by its task specificity, its lack of manual alignment requirements, its explicit coverage of partial/full substitutions, insertions, deletions, translation, and code-mixing, and its combined use of PER, F0-CORR, SIM, VS, and MOS to expose trade-offs intrinsic to editing under fixed melody (Hao et al., 25 Mar 2026).

This makes it adjacent to, but distinct from, text-side lyric-editing frameworks such as REFFLY, which formulates melody-constrained lyric editing as revision of a plain-text draft into melody-aligned lyrics and evaluates properties such as prominent word–note Match Rate, BERTScore similarity, and human ratings for prosody, singability, intelligibility, coherence, creativity, structural clarity, and translation quality (Zhao et al., 2024). REFFLY operates at the level of textual lyric revision under melody constraints, whereas LyricEditBench evaluates singing voice synthesis outputs under melody-preserving lyric modification. A plausible implication is that the two works occupy complementary positions: one defines a text editing problem and associated heuristics, while the other defines an audio-generation benchmark with timbre and melody control.

LyricEditBench is also distinct from resources such as LyCon, which reconstructs copyright-free lyrics from bag-of-words and metadata and releases a dataset of reconstructed lyrics aligned with metadata from the Million Song Dataset, Deezer Mood Detection Dataset, and AllMusic Genre Dataset (Kim et al., 2024). LyCon supports constrained lyric generation and could underpin future lyric-editing resources, but it does not define a benchmark for matched-melody lyric modification. This suggests that LyricEditBench occupies a more specialized niche at the intersection of lyric editing and controllable singing synthesis.

The benchmark is announced as publicly available in the YingMusic-Singer repository, which releases code, weights, benchmark, and demos. The paper states that LyricEditBench is part of the repository along with scripts for inference and evaluation, and indicates that users can access metadata for all 7,200 instances, audio files or references for melody and timbre prompts, corresponding original and modified lyric texts, and evaluation scripts for PER, SIM, F0-CORR, and VS (Hao et al., 25 Mar 2026).

The stated limitations are equally specific. LyricEditBench covers only Chinese and English; it inherits the distribution of GTSinger and therefore may not cover all genres or noisy/live performance conditions; PER depends on Qwen3-ASR-1.7B, so code-mixed or stylistically extreme singing may distort PER, especially in Mix; only clips of 15 seconds or less are used; and the benchmark assumes a fixed melody, so it does not test flexible melody modification or composition-level editing (Hao et al., 25 Mar 2026).

Within those boundaries, LyricEditBench functions as a specialized evaluation instrument for controllable SVS systems that must jointly satisfy lyric adherence, melody preservation, timbre consistency, and perceptual vocal quality under realistic multilingual and multi-technique editing scenarios.