MeloDRP: Melody-aware Duration Predictor

• MeloDRP is a melody-aware duration predictor that reallocates fixed time budgets across phoneme spans to preserve musical timing in edited singing voices.
• It employs a span-wise softmax over duration logits and cross-modal attention with temporal-overlap supervision to align phonetic cues with melodic context.
• Integration with a conditional flow-matching mel decoder ensures only edited regions are resynthesized, resulting in seamless transitions and exact duration preservation.

Searching arXiv for MeloDISinger/MeloDRP and closely related singing voice editing work. MeloDRP, short for Melody-aware Duration Ratio Predictor, is the core timing module in MeloDISinger, a text-based singing voice editing (SVE) system that revises sung lyrics while preserving the original melody, the total duration of the edited audio, and all non-edited regions. Within MeloDISinger, MeloDRP addresses the central duration-control problem by predicting span-wise fixed-budget duration ratios conditioned on phonetic and melodic context, and by supervising cross-modal attention with temporal-overlap signals. Its duration outputs drive a conditional flow-matching mel decoder that infills only edited regions, enabling seamless boundary transitions and exact preservation elsewhere (Park et al., 29 Jun 2026).

1. Role within text-based singing voice editing

Text-based SVE is defined by three simultaneous constraints. Given original audio $S_{\mathrm{orig}}$, original lyrics $L_{\mathrm{orig}}$, and edited lyrics $L_{\mathrm{edit}}$, the system must synthesize $S_{\mathrm{edit}}$ such that the generated singing follows the original melodic contour, the total duration and each edited span’s duration are strictly preserved, and only edited regions are resynthesized while non-edited regions remain unchanged. The source paper explicitly characterizes this as fundamentally harder than speech editing because the edited content must remain synchronized to musical time and pitch (Park et al., 29 Jun 2026).

Within MeloDISinger’s pipeline, MeloDRP is placed in the acoustic model stage. After feature extraction and edit parsing, the acoustic model predicts edited phoneme durations $D_{\mathrm{edit}}$ with MeloDRP and edited-region pitch $P_{\mathrm{edit}}$ with FPIP. These signals condition a flow-matching mel decoder, which infills the masked regions and merges them into the original mel-spectrogram. The system design therefore makes duration prediction an explicit, constrained subproblem rather than an implicit byproduct of acoustic generation.

A key implication is that MeloDRP is not merely a duration estimator. It is the mechanism by which SVE’s locality and synchronization constraints are enforced at span level, before mel synthesis occurs.

2. Span-wise fixed-budget duration modeling

MeloDRP reformulates duration prediction as reallocation of a fixed time budget within each edited span. Conventional duration predictors output absolute durations and do not enforce a prescribed total duration; in SVE, this is inadequate because each edited span must fit the original time slot exactly (Park et al., 29 Jun 2026).

Let edited spans be indexed by $i=1,\dots,N$. Each span $i$ has a time budget $T_i$ and a target count $E_i$ of phonemes to render inside that span. Instead of predicting absolute phoneme durations directly, MeloDRP predicts a ratio vector $L_{\mathrm{orig}}$0 satisfying nonnegativity and unit sum, and reconstructs phoneme durations as

$L_{\mathrm{orig}}$1

By construction,

$L_{\mathrm{orig}}$2

so span-wise duration preservation is guaranteed.

The span budget $L_{\mathrm{orig}}$3 depends on the edit operation:

• Replacement: the original replaced span’s duration is reallocated to the new phonemes.
• Insertion: duration is collected from a local neighborhood and reallocated across the inserted and neighboring phonemes.
• Deletion: the deleted span’s duration is assigned to a silence phoneme.

These budgets are provided to MeloDRP as a phoneme-level budget sequence $L_{\mathrm{orig}}$4, where $L_{\mathrm{orig}}$5 for phonemes inside span $L_{\mathrm{orig}}$6 and $L_{\mathrm{orig}}$7 elsewhere.

Fixed-budget enforcement is achieved through a span-wise softmax over duration logits:

$L_{\mathrm{orig}}$8

Multiplication by $L_{\mathrm{orig}}$9 then yields $L_{\mathrm{edit}}$0. The paper emphasizes that this satisfies the fixed-budget constraint in both training and inference, and that no extra constraints such as Dirichlet distributions are required (Park et al., 29 Jun 2026).

This formulation decouples how a span’s time is divided from how much total time is available, which focuses learning on relative timing structure rather than absolute duration scale.

3. Input representations and cross-modal architecture

MeloDRP uses two synchronized input streams: a phonetic stream and a melodic stream.

The phonetic stream begins with conversion of $L_{\mathrm{edit}}$1 to phonemes via g2p-en. During training, phoneme identities and features for the original sequence are available from MFA (Montreal Forced Aligner). Linguistic features include start flags with values 2/1/0 for word-initial, syllable-initial but non-word-initial, and others, as well as coarse phoneme types defined by broad classes of manner of articulation, with vowels including stress markers. These are embedded and combined with positional encodings. The system uses dropout of phoneme identity and type embeddings with $L_{\mathrm{edit}}$2 to improve robustness (Park et al., 29 Jun 2026).

The melodic stream is built from pseudo-MIDI context derived from the original singing. Frame-level $L_{\mathrm{edit}}$3 with voiced/unvoiced flags is extracted from $L_{\mathrm{edit}}$4, then converted into pseudo-MIDI notes through MIDI quantization of $L_{\mathrm{edit}}$5, note segmentation, and post-processing. The note tokens carry pitch, duration, onset times, and optional beat or positional cues implied by note ordering and timing. Because MFA supplies phoneme-level time intervals for the original audio, phonemes and pseudo-MIDI notes share the same timeline and can be matched through temporal overlap. The system explicitly does not enforce one-to-one phoneme–note mapping; instead, it learns soft correspondences (Park et al., 29 Jun 2026).

The architecture is a two-stream Transformer with cross-attention fusion:

Module	Configuration	Function
Phoneme encoder	4-layer Transformer, hidden size 256, 2 attention heads	Encodes edited phoneme sequence and linguistic cues
Melody encoder	4-layer Transformer, hidden size 256, 2 heads	Encodes pseudo-MIDI note sequence
Duration-ratio decoder	6-layer Transformer decoder, hidden size 256, 2 heads	Produces span-local duration logits

Cross-attention uses the phoneme-side representation as queries and the melody-side representation as keys and values:

$L_{\mathrm{edit}}$6

Here, $L_{\mathrm{edit}}$7 comes from encoded phoneme tokens within each edited span, while $L_{\mathrm{edit}}$8 come from encoded pseudo-MIDI note tokens covering the same global time window. The stated rationale is that duration must be allocated among phonemes according to both linguistic role and alignment to performed melodic events such as note onsets, sustains, and rhythm. Using phonemes as queries allows each phoneme to retrieve the most relevant melodic context for its ratio decision (Park et al., 29 Jun 2026).

4. Temporal-overlap supervision and objective design

To make cross-attention reflect musically meaningful alignments, MeloDRP introduces temporal-overlap supervision. If phoneme $L_{\mathrm{edit}}$9 covers $S_{\mathrm{edit}}$0 and note $S_{\mathrm{edit}}$1 covers $S_{\mathrm{edit}}$2, a general overlap weight could be written as

$S_{\mathrm{edit}}$3

In MeloDRP, the supervised target is simplified to a binary overlap mask:

$S_{\mathrm{edit}}$4

The cross-attention matrix $S_{\mathrm{edit}}$5, aggregated over heads, is encouraged to match $S_{\mathrm{edit}}$6 through an $S_{\mathrm{edit}}$7 guided-attention loss. This promotes higher attention on time-overlapping phoneme–note pairs and lower attention otherwise (Park et al., 29 Jun 2026).

Ground-truth duration supervision is also span-normalized. If $S_{\mathrm{edit}}$8 is the ground-truth phoneme duration within span $S_{\mathrm{edit}}$9 and $D_{\mathrm{edit}}$0 is the span duration, the target ratios are

$D_{\mathrm{edit}}$1

The MeloDRP objective is

$D_{\mathrm{edit}}$2

Its components are:

• Phoneme-level ratio loss ($D_{\mathrm{edit}}$3): KL divergence per span between target and predicted ratios.
• Word-level aggregation loss ($D_{\mathrm{edit}}$4): $D_{\mathrm{edit}}$5 loss after summing phoneme ratios within each word.
• Penalty loss ($D_{\mathrm{edit}}$6): penalizes phonemes whose predicted duration $D_{\mathrm{edit}}$7 falls below a minimum threshold $D_{\mathrm{edit}}$8.
• Guided-attention loss ($D_{\mathrm{edit}}$9): $P_{\mathrm{edit}}$0 loss between attention $P_{\mathrm{edit}}$1 and the binary overlap mask $P_{\mathrm{edit}}$2.

The paper’s formulation makes the modeling priorities explicit: span-level duration conservation is enforced algebraically; ratio learning captures intra-span timing structure; word aggregation regularizes higher-level rhythm; and overlap supervision anchors cross-modal attention to the shared timeline (Park et al., 29 Jun 2026).

5. Integration with flow-matching audio infilling

MeloDRP’s outputs are used to convert the edited phoneme sequence into a frame-level conditioning stream for a non-autoregressive conditional flow-matching mel decoder. Each phoneme embedding is repeated for its predicted frame count, so the decoder receives timing-conditioned phoneme features at the intended rhythm within each span’s fixed budget (Park et al., 29 Jun 2026).

The decoder conditions on the sum of frame-level embeddings from phoneme, pitch, speaker, and context mel. Training uses random edit masks, and the conditional flow-matching objective is

$P_{\mathrm{edit}}$3

where $P_{\mathrm{edit}}$4, $P_{\mathrm{edit}}$5, $P_{\mathrm{edit}}$6, and $P_{\mathrm{edit}}$7.

At inference, the decoder starts from Gaussian noise, solves the learned ODE to sample $P_{\mathrm{edit}}$8 for masked frames, and merges the result into the original mel-spectrogram as

$P_{\mathrm{edit}}$9

Because non-edited regions never pass through the generative pathway, they remain bit-identical to the original, while edited regions are synthesized under strong conditioning from surrounding mel context, predicted durations, and pitch (Park et al., 29 Jun 2026).

The full inference pipeline combines alignment, edited-lyric generation, edit parsing, duration budgeting, MeloDRP prediction, pitch conditioning, mel infilling, and waveform generation. Alignment uses WhisperX for word-level onset/offset times and MFA for phoneme-level durations. Duration-aware edited-lyric generation converts each word slot duration $i=1,\dots,N$0 into a syllable capacity

$i=1,\dots,N$1

with $i=1,\dots,N$2 and $i=1,\dots,N$3, and an LLM rewrites lyrics subject to these capacities. The final waveform is produced by PC-NSF HiFi-GAN at 44.1 kHz with window 2048, hop 512, and 128 mel bins (Park et al., 29 Jun 2026).

6. Empirical characteristics and comparative significance

The reported experiments use GTSinger-En, a dataset of 13 hours of English singing from three singers spanning six singing techniques. Audio is segmented into chunks of at most 11.6 s or 1000 mel frames, with word boundaries preserved via MFA durations. Feature extraction uses Resemblyzer for speaker embeddings and Parselmouth for $i=1,\dots,N$4; frames with $i=1,\dots,N$5 are treated as unvoiced. Training uses Adam with $i=1,\dots,N$6, $i=1,\dots,N$7, learning rate $i=1,\dots,N$8, batch size 16, and MultiStepLR with 0.5 decay at 10k/20k/30k steps. Edit masks are sampled with $i=1,\dots,N$9, and inference uses 100 Euler steps (Park et al., 29 Jun 2026).

Evaluation covers intelligibility, duration preservation, and melody following. Objective metrics are WER and CER from Whisper-large-v3, Duration Consistency (DC), Duration Difference (DDUR), and F0 Pearson Correlation (FPC) with Cut and DTW variants. Across insertion, deletion, replacement, and mixed edits, the system reports DDUR $i$0 s and DC $i$1, alongside improved intelligibility and melody following relative to prior baselines. Representative examples include Rep-S with WER/CER 21.88/15.26, Ins with 18.57/11.62, and Del with 24.88/15.74. For melody following, the insertion setting reports FPC-Cut/DTW 77.71/71.14. Subjective evaluation with 22 listeners gives, for Rep-SM, Lyric 4.05±0.23, Melody 3.99±0.20, and Naturalness 3.65±0.19, with significant improvement over the baselines (Park et al., 29 Jun 2026).

The ablation study isolates the contribution of MeloDRP’s design choices. Removing total-duration conditioning produces the largest degradation, for example on mixed edits with WER/CER 44.7/31.6 versus 39.4/27.6 in the full system. Removing melody conditioning degrades replacement and insertion settings; removing phoneme features notably hurts insertion; and removing guided attention weakens sensitivity to melodic context. The paper interprets these results as evidence for three main novelties: fixed-budget, span-wise control, melody-aware allocation, and soft alignment without one-to-one phoneme–note assumptions (Park et al., 29 Jun 2026).

7. Limitations, failure modes, and terminology

The source paper identifies several limitations. MeloDRP is sensitive to alignment errors in MFA phoneme timings, WhisperX word boundaries, and the $i$2-to-MIDI conversion pipeline. Such errors can misguide attention and duration allocation, particularly in rapid ornaments or vibrato-heavy segments. Extreme edits that introduce large syllable-count changes within very short budgets may force many sub-threshold phonemes; the penalty loss mitigates this but does not eliminate intelligibility trade-offs. The binary overlap mask used for guided attention is deliberately simple; a length-weighted overlap could provide finer supervision, but it is not used. Computationally, the architecture itself is lightweight, but inference with 100 Euler steps in the flow-matching decoder is the main cost, although infilling only edited spans reduces practical latency (Park et al., 29 Jun 2026).

The acronym is also potentially ambiguous across literatures. In the singing voice editing work, MeloDRP denotes Melody-aware Duration Ratio Predictor. By contrast, in last-mile logistics the standard acronym is MDRP, standing for Meal Delivery Routing Problem, and that paper explicitly states that it does not use “MeloDRP” (Giraldo-Herrera et al., 2024). A plausible implication is that acronym expansion should be checked carefully when moving between music generation and operations research contexts.