MeloDISinger: Melody-Aware & Duration-Preserving Singing Voice Editing with Audio Infilling

Abstract: Text-based singing voice editing (SVE) aims to revise sung lyrics while preserving the original melody, total duration, and non-edited regions. In this paper, we propose MeloDISinger, a flow-matching-based SVE model for melody-aware and duration-preserving editing. Its core module, MeloDRP, predicts fixed-budget duration ratios, enabling explicit span-wise duration control. For melody-aware duration allocation, MeloDRP fuses phonetic cues with pseudo-MIDI melodic context through cross-attention, while temporal-overlap supervision encourages soft phoneme--note correspondences. We further use a flow-matching mel decoder for audio infilling to synthesize edited regions while preserving surrounding context. In addition, we introduce a duration-aware edited-lyric generation pipeline using WhisperX and an LLM to construct feasible evaluation scenarios. Experiments demonstrate state-of-the-art performance in both objective and subjective evaluations.

• The paper introduces MeloDISinger, a text-based editing system that explicitly preserves melody and duration using a novel ratio-based duration predictor.
• It employs a flow-matching-based audio infilling architecture to regenerate only the edited regions, ensuring seamless transitions with the unaltered segments.
• Experimental results demonstrate lower error rates, superior MOS ratings, and improved intelligibility and melody fidelity compared to existing methods.

MeloDISinger: Melody-Aware & Duration-Preserving Singing Voice Editing with Audio Infilling

Introduction

Text-based Singing Voice Editing (SVE) is a demanding task that seeks to regenerate vocal segments according to revised lyrics, while strictly retaining the original melody, total duration, and non-edited audio. Unlike speech editing systems, SVE must consider complex musical constraints—particularly melodic adherence and precise rhythmic alignment with musical accompaniment. This paper introduces MeloDISinger, a novel SVE system that addresses these requirements through explicit, melody-aware, and duration-preserving mechanisms. The core innovations are the introduction of the MeloDRP duration predictor—fusing acoustic and melodic cues under a fixed-duration allocation—and a flow-matching-based audio infilling architecture that selectively regenerates edited regions while maintaining seamless transitions and surrounding audio context.

Figure 1: Overview of MeloDISinger—including (a) the text-based SVE pipeline and (b) the MeloDRP architecture for melody-aware duration-ratio prediction.

System Architecture

MeloDISinger is structured into three main pipeline stages: feature extraction, parsing operations to localize edits, and a generative model that synthesizes revised audio. The methodological advancements are primarily located in two components: MeloDRP (Melody-aware Duration Ratio Predictor) and an infilling acoustic model driven by conditional flow-matching.

Feature Extraction and Parsing

Audio features (mel-spectrograms, F0 contours, speaker embeddings, pseudo-MIDI derived from F0) and phoneme-aligned durations are extracted from the source. The edit regions are determined by aligning original and edited lyrics, resulting in a phoneme-level edit mask that guides subsequent modeling.

MeloDRP: Span-wise Duration Ratio Prediction

Prior SVE approaches either use implicit duration control, risking misalignment, or reuse original phoneme durations, which precludes edits that alter phoneme structures. MeloDISinger addresses this by introducing MeloDRP, which predicts normalized duration ratios (subject to a fixed total per edit-span budget), enabling explicit and precise control. This is formulated so that, given a desired total duration $T_i$ and $E_i$ target phonemes:

$$\sum_{j=1}^{E_i} r_{ij} = 1,\quad \hat{d}_{ij} = T_i r_{ij}$$

This ensures the reconstructed span matches the original region's timing precisely.

MeloDRP injects melody-awareness by fusing phonetic representations with pseudo-MIDI context through cross-attention. Supervision is provided by losses on KL divergence for phoneme durations, word-level aggregation, penalties for undershoot durations, and a guided-attention loss anchored by actual phoneme–note overlap in time. Ablation analyses verify that each component contributes to edit robustness, intelligibility, and melody faithfulness.

Flow-Matching-Based Audio Infilling

The system's mel-spectrogram decoder is non-autoregressive and employs continuous flow-matching. It is conditioned on phoneme, pitch, speaker, and context embeddings. Only the edited regions are resynthesized, with their boundaries informed by the predicted durations and F0. At inference, the decoder generates edited spans from noise via a learned ODE (solved with 100 Euler steps), and merges the results with the original, unaltered segments for perfect temporal and timbral continuity at edit boundaries.

Evaluation Protocol

A duration-aware edited-lyric generation pipeline is proposed, combining WhisperX-based word alignment with LLM-driven text generation to synthesize feasible edit scenarios. This ensures edits conform to musical timing constraints—addressing the limitations in prior work that might induce unattainable or unnatural edits.

Experimental Results

Experiments were conducted using GTSinger-En, a multi-technique English singing corpus. Objective metrics include WER/CER (intelligibility), duration accuracy (Duration Consistency and Duration Difference), and melody tracking (F0 Pearson correlation). Subjective metrics were obtained via MOS ratings across Lyric Following, Melody Following, and Naturalness.

Objective Evaluation

MeloDISinger yields consistently lower WER/CER and strictly preserves overall duration (Duration Difference ≈ 0.00s), outperforming both EditSinger and Vevo2 in all non-trivial edit scenarios, and achieves the highest F0 correlation with the original melody in most cases.

Subjective Evaluation

Listeners rated MeloDISinger highest for all criteria and edit types. Particularly, the system was superior in challenging cases—where phoneme and syllable structures change—demonstrating the capacity of MeloDRP to flexibly reallocate durations while maintaining prosodic and melodic naturalness.

Ablation Study

Ablation results demonstrate that removing duration conditioning causes the largest degradations (WER/CER), confirming the necessity of strict duration budgeting. Melody conditioning and guided attention further contribute to fidelity in melody and articulation, especially in edits that alter phonetic structure. The qualitative analysis reveals that melody conditioning ensures that edited spans temporally align with underlying note structures, while removing phoneme cues harms scenarios with inserted material.

Theoretical and Practical Implications

MeloDISinger showcases a general framework for controlled, context-aware regeneration in musical audio. The explicit duration allocation mechanism mediated through ratio prediction and cross-attention with melodic context is particularly notable, as it facilitates edit flexibility without sacrificing temporal and melodic fidelity. This approach redefines feasible SVE operations: it enables insertion, deletion, and complex replacements beyond simple phoneme-matched scenarios.

Practically, this system extends the capabilities of interactive audio production, automated lyric revision, and content localization tools, offering high-quality and context-conserving vocal edits suitable for production workflows. The duration-aware edit generation pipeline has ramifications for music- and speech-synthesis evaluation protocols, providing a template for time-feasible edit synthesis in multi-modal generation tasks.

Future Directions

Potential future developments include the integration of end-to-end neural lyric alignment, more sophisticated melodic-context modeling, and the formulation of new melody-aware evaluation metrics tailored to SVE. MeloDISinger's core architecture—melding ratio-based duration modeling with flow-matching-based infilling—could also be adapted to other generative editing tasks involving temporally-structured signals, including instrumental music or expressive speech synthesis.

Conclusion

MeloDISinger establishes a robust, efficient solution to text-based singing voice editing, combining explicit melody-contextual duration modeling with advanced infilling via flow-matching. The system sets a new standard for naturalness, lyric intelligibility, and musical alignment in SVE, as supported by both objective and subjective assessment. The architecture and evaluation pipeline presented have immediate applications for automatic vocal editing and broader generative audio modeling tasks.