MeloDISinger: Singing Voice Editing Model

• MeloDISinger is a text-based singing voice editing model that precisely revises lyrics while retaining the original melody, rhythm, overall duration, and non-edited regions.
• It employs a melody-aware fixed-budget duration ratio predictor (MeloDRP) and a conditional flow-matching mel decoder to synthesize only the edited spans through targeted audio infilling.
• The system integrates rigorous alignment, duration-aware evaluation metrics, and contextual preservation techniques to outperform previous SVE methods in intelligibility and melodic accuracy.

Searching arXiv for MeloDISinger and closely related singing voice editing papers. MeloDISinger is a text-based singing voice editing (SVE) model that revises sung lyrics while preserving the original melody, total duration, and non-edited regions through audio infilling. Formulated as an editing system rather than a de novo singing synthesizer, it takes original singing audio $S_{\mathrm{orig}}$, original lyrics $L_{\mathrm{orig}}$, and edited lyrics $L_{\mathrm{edit}}$, and generates edited singing audio $S_{\mathrm{edit}}$ whose lyrics follow the edit while retaining the source performance’s melody, rhythm, and temporal structure (Park et al., 29 Jun 2026). Its defining components are MeloDRP, a melody-aware fixed-budget duration-ratio predictor, and a conditional flow-matching mel decoder that synthesizes only the edited spans and copies the remaining mel frames exactly (Park et al., 29 Jun 2026).

1. Task definition and formal constraints

MeloDISinger addresses the SVE problem under four simultaneous constraints: the output must follow the edited lyrics, preserve the original melody and rhythm, exactly preserve total duration, and leave non-edited regions unchanged (Park et al., 29 Jun 2026). This combination distinguishes it from systems that regenerate an entire sequence or that treat duration implicitly.

The model operates over $N$ disjoint edited spans. For each span $i$, $T_i$ denotes the duration budget derived from the original timing, and $E_i$ denotes the number of edited phonemes. MeloDRP predicts duration ratios $r_{ij}$ satisfying

$$\sum_{j=1}^{E_i} r_{ij} = 1, \quad \hat d_{ij} = T_i r_{ij},$$

so that span duration is preserved by construction:

$L_{\mathrm{orig}}$0

This fixed-budget formulation is central to the system. Prior SVE approaches described in the source material either learn duration implicitly or predict absolute durations without melodic context, whereas MeloDISinger allocates a fixed span-wise time budget across edited phonemes (Park et al., 29 Jun 2026).

The paper defines operation-dependent budgets. In replacement, $L_{\mathrm{orig}}$1 is the duration of the original replaced span. In insertion, $L_{\mathrm{orig}}$2 is defined from the local span around the insertion and reallocated between neighboring and inserted phonemes. In deletion, the removed span’s duration is assigned to a silence phoneme so that timing remains unchanged (Park et al., 29 Jun 2026). A phoneme-level budget sequence $L_{\mathrm{orig}}$3 is then formed with $L_{\mathrm{orig}}$4 inside span $L_{\mathrm{orig}}$5 and $L_{\mathrm{orig}}$6 elsewhere.

A common source of confusion is to treat MeloDISinger as a general singing synthesis model. It is instead an editing system whose output remains anchored to an existing sung performance. The melody is not newly composed, and non-edited regions are not regenerated; they are preserved explicitly at the mel level (Park et al., 29 Jun 2026).

2. System architecture and signal representations

The overall pipeline comprises feature extraction, a parsing operation, acoustic modeling, and waveform synthesis (Park et al., 29 Jun 2026). From $L_{\mathrm{orig}}$7, the system extracts mel-spectrogram $L_{\mathrm{orig}}$8, a speaker embedding via Resemblyzer, frame-level F0 and voiced/unvoiced flags via Parselmouth, and a pseudo-MIDI score obtained from F0 through MIDI quantization and note segmentation. From $L_{\mathrm{orig}}$9, it derives phoneme sequences and phoneme-level durations using Montreal Forced Aligner. From $L_{\mathrm{edit}}$0, it derives a phoneme sequence using g2p-en together with phoneme-level linguistic features: start flags and coarse phoneme type (Park et al., 29 Jun 2026).

The parsing stage compares $L_{\mathrm{edit}}$1 and $L_{\mathrm{edit}}$2 to identify replacement, insertion, and deletion regions and to construct a phoneme-level edit mask $L_{\mathrm{edit}}$3 (Park et al., 29 Jun 2026). This mask determines which parts of the signal are to be regenerated.

The acoustic model then combines three modules. MeloDRP predicts edited-span phoneme durations through duration ratios. FPIP, inherited from EditSinger, predicts edited-region F0 contours. A flow-matching mel decoder performs infilling conditioned on edited phonemes, predicted durations, predicted F0, speaker embedding, and the original mel context (Park et al., 29 Jun 2026). The final mel-spectrogram is merged as

$L_{\mathrm{edit}}$4

This merge rule is a precise statement of the preservation guarantee: unedited mel frames are copied directly rather than approximated (Park et al., 29 Jun 2026).

The vocoder stage uses an external PC-NSF HiFi-GAN configuration to convert the edited mel-spectrogram into waveform (Park et al., 29 Jun 2026). Training uses GTSinger-En, consisting of 13 hours of English singing from 3 singers, with audio segmented into chunks up to 11.6 s while preserving word boundaries. Preprocessing uses 44.1 kHz audio, window 2048, hop 512, and 128 mel bins (Park et al., 29 Jun 2026).

3. MeloDRP: melody-aware fixed-budget duration ratio prediction

MeloDRP is the model’s principal duration-control mechanism. Rather than regressing absolute phoneme durations, it predicts a normalized distribution over phonemes inside each edited span and then rescales that distribution by the span budget $L_{\mathrm{edit}}$5 (Park et al., 29 Jun 2026). This makes duration preservation a hard architectural property rather than a soft training tendency.

Its input representation for each edited phoneme includes phoneme identity, phoneme type, start flag, and budget-related information. The edited phoneme sequence is encoded by a 4-layer Transformer encoder with hidden size 256 and 2 heads. In parallel, pseudo-MIDI notes extracted from the source performance are embedded and passed through another 4-layer Transformer encoder (Park et al., 29 Jun 2026). A 6-layer Transformer decoder then performs cross-attention from phoneme queries to pseudo-MIDI keys and values, followed by projection to duration logits and a span-wise softmax that enforces $L_{\mathrm{edit}}$6 within each edited span (Park et al., 29 Jun 2026).

The melodic conditioning is not symbolic-score conditioning in the conventional SVS sense. MeloDISinger extracts pseudo-MIDI from the actual sung F0 contour, so the duration model is tied to performance-specific melodic structure rather than to an external score (Park et al., 29 Jun 2026). This suggests that duration allocation is shaped by the realized melody, not merely by text or generic prosody.

To supervise phoneme–note interaction, the model uses a temporal-overlap-based guided-attention mechanism. A binary phoneme-by-note mask is constructed from temporal overlap between ground-truth phoneme durations and pseudo-MIDI note durations: overlap is assigned 1, non-overlap 0 (Park et al., 29 Jun 2026). An $L_{\mathrm{edit}}$7 guided-attention loss encourages cross-attention maps to match this overlap mask rather than a diagonal prior. The choice is important because singing often exhibits many-to-many relations between phonemes and notes.

The full MeloDRP loss is

$L_{\mathrm{edit}}$8

with $L_{\mathrm{edit}}$9 (Park et al., 29 Jun 2026). The component losses are a phoneme-level KL divergence over span-wise ratios, a word-level duration $S_{\mathrm{edit}}$0 term, a penalty for predicted durations below a minimum threshold $S_{\mathrm{edit}}$1, and the guided-attention loss. Using ground-truth phoneme durations $S_{\mathrm{edit}}$2, the target ratios are defined as

$S_{\mathrm{edit}}$3

Ablation analysis shows that removing duration budget conditioning causes the largest degradation in WER and CER, while removing melody conditioning harms performance particularly in replacement, insertion, and mixed-edit settings (Park et al., 29 Jun 2026). The paper interprets this as evidence that both fixed-budget normalization and melodic context are structurally necessary rather than optional refinements.

4. Conditional flow matching and audio infilling

The second defining component is a flow-matching mel decoder for edited-region synthesis (Park et al., 29 Jun 2026). The model adopts conditional flow matching in the sense of learning a time-dependent velocity field from Gaussian noise to the target mel-spectrogram, but constrains training to masked regions so that the task is explicitly one of infilling rather than full-sequence generation.

Let $S_{\mathrm{edit}}$4 denote the target mel-spectrogram and $S_{\mathrm{edit}}$5 Gaussian noise. The linear interpolation path is

$S_{\mathrm{edit}}$6

with target velocity

$S_{\mathrm{edit}}$7

Conditioning $S_{\mathrm{edit}}$8 includes edited phoneme embeddings, durations predicted by MeloDRP, F0 predicted by FPIP, speaker embedding, and original mel context (Park et al., 29 Jun 2026). During training, random frame-level masks are sampled with mask ratio $S_{\mathrm{edit}}$9, including both continuous and discontinuous spans, and the conditional flow-matching loss is applied only to the masked frames:

$N$0

At inference, the model integrates the corresponding ODE from noise using 100 Euler steps to obtain $N$1, then merges generated and untouched frames through the edit mask (Park et al., 29 Jun 2026). The decoder backbone is a non-causal WaveNet with 20 residual layers and 256 channels, jointly trained with a mel encoder and a phoneme encoder under the CFM objective (Park et al., 29 Jun 2026).

This design has two practical implications. First, continuity across edit boundaries is learned as a conditional infilling problem rather than imposed post hoc. Second, the requirement that non-edited regions remain unchanged is exact in mel space, because those frames are copied rather than decoded (Park et al., 29 Jun 2026).

5. Duration-aware lyric generation, training procedure, and evaluation

The evaluation protocol is itself a substantive contribution. Earlier SVE evaluations described in the source material use LLM-based lyric rewriting without duration feasibility constraints, which can produce edits that do not fit the available singing time (Park et al., 29 Jun 2026). MeloDISinger introduces a duration-aware lyric generation pipeline based on WhisperX and an LLM.

WhisperX provides word-level onset and offset times. For each editable word or phrase, the available duration $N$2 is converted to a syllable-capacity constraint

$N$3

with $N$4 and $N$5 (Park et al., 29 Jun 2026). Gemini-2.5-flash then generates edited lyrics subject to per-slot syllable capacities, and outputs are automatically rejected and regenerated if they violate the constraint. The resulting test conditions include insertion, deletion, mixed edits, phoneme-matched replacement (Rep-P), syllable-matched replacement (Rep-S), and syllable-mismatched replacement (Rep-SM) (Park et al., 29 Jun 2026).

Training uses Adam with $N$6, $N$7, learning rate $N$8, batch size 16, and MultiStepLR halving at 10k, 20k, and 30k steps (Park et al., 29 Jun 2026). Phoneme identity and type embeddings are dropped out with probability 0.3 to improve robustness (Park et al., 29 Jun 2026).

Representative results reported in the paper are as follows.

Aspect	Representative finding	Scenario
Duration preservation	MeloDISinger reports DDUR $N$9 s and DC $i$0	Across scenarios
Intelligibility	WER/CER $i$1, versus $i$2 for EditSinger and $i$3 for Vevo2	Rep-P
Melody following	Cut/DTW FPC $i$4, versus $i$5 for EditSinger and $i$6 for Vevo2	Del

Subjective evaluation with 22 listeners rates lyric following, melody following, and naturalness on a 1–5 scale with 0.5 increments (Park et al., 29 Jun 2026). In the most difficult Rep-SM setting, Vevo2 scores 2.92, 3.08, and 2.86, whereas MeloDISinger scores 4.05, 3.99, and 3.65. In Mix, Vevo2 scores 2.13, 2.31, and 2.39, whereas MeloDISinger scores 4.12, 3.64, and 3.48. In Del, EditSinger scores 4.03, 3.75, and 3.45, while MeloDISinger scores 4.21, 4.05, and 3.87 (Park et al., 29 Jun 2026). The reported pattern is consistent: the model performs best when edits alter phoneme and syllable structure, precisely the regime where span-wise duration redistribution matters most.

6. Position within singing research, limitations, and future directions

Within the broader singing literature, MeloDISinger occupies the intersection of SVE, melody-conditioned timing control, and neural audio infilling. It differs from melody-unsupervision singing synthesis, where a model is trained from unaligned audio-and-lyrics pairs and later synthesizes singing from lyrics and melody (Choi et al., 2021), because MeloDISinger edits an existing performance rather than synthesizing a new one from scratch. It also differs from pronunciation-oriented non-autoregressive SVS such as N-Singer, which uses separate linguistic and pitch pathways to synthesize Korean singing from lyrics and score (Lee et al., 2021); MeloDISinger instead preserves a source performance’s timing and melody through fixed-budget editing constraints. It is likewise distinct from speech-to-singing and singer-style transfer pipelines such as SingIt!, which convert source content into a target singer identity via zero-shot many-to-many transfer (Eliav et al., 2024). Its pseudo-MIDI front end places it adjacent to melody extraction research concerned with robust F0 estimation under domain shift (Saxena et al., 2024), although MeloDISinger uses extracted melodic structure as an editing scaffold rather than as an end task.

The paper explicitly characterizes MeloDISinger as the first SVE system that simultaneously provides explicit melody-aware duration control with strict preservation per edit span, high-quality audio infilling that preserves context and melody, and a duration-aware evaluation protocol grounded in singing-specific temporal feasibility (Park et al., 29 Jun 2026). A plausible implication is that the system’s main advance is not merely generative quality, but the conversion of several informal desiderata of singing editing into hard architectural or procedural constraints.

The reported limitations are also specific. The system depends on alignment quality from Montreal Forced Aligner and WhisperX. Its melody-aware behavior depends on accurate F0 tracking and pseudo-MIDI quantization, which can be challenged by vibrato and slides. Training uses 13 hours from 3 English singers, so generalization to other languages, timbres, and genres remains open. The flow-matching decoder uses 100 ODE steps and a non-causal WaveNet backbone, which is heavier than simpler decoders (Park et al., 29 Jun 2026). The paper identifies future work in melody-aware metrics and broader SVE settings, including more varied datasets, languages, and scenarios with larger edits (Park et al., 29 Jun 2026).

In that sense, MeloDISinger can be understood as a constrained editing framework for sung audio: it treats lyric replacement, insertion, and deletion as operations that must remain synchronized with an already realized musical performance, and it enforces that requirement through fixed-budget duration ratios, melody-aware phoneme–note fusion, and masked flow-matching infilling (Park et al., 29 Jun 2026).