---
title: 'MeloDRP: Melody-aware Duration Predictor'
url: https://www.emergentmind.com/topics/melodrp
type: topic
---

# MeloDRP: Melody-aware Duration Predictor

Searching arXiv for MeloDISinger/MeloDRP and closely related singing voice editing work.
MeloDRP, short for **Melody-aware Duration Ratio Predictor**, is the core timing module in **MeloDISinger**, a text-based singing voice editing (SVE) system that revises sung lyrics while preserving the original melody, the total duration of the edited audio, and all non-edited regions. Within MeloDISinger, MeloDRP addresses the central duration-control problem by predicting **span-wise fixed-budget duration ratios** conditioned on phonetic and melodic context, and by supervising cross-modal attention with temporal-overlap signals. Its duration outputs drive a conditional flow-matching mel decoder that infills only edited regions, enabling seamless boundary transitions and exact preservation elsewhere [2606.30580].

## 1. Role within text-based singing voice editing

Text-based SVE is defined by three simultaneous constraints. Given original audio $S_{\mathrm{orig}}$, original lyrics $L_{\mathrm{orig}}$, and edited lyrics $L_{\mathrm{edit}}$, the system must synthesize $S_{\mathrm{edit}}$ such that the generated singing follows the original melodic contour, the total duration and each edited span’s duration are strictly preserved, and only edited regions are resynthesized while non-edited regions remain unchanged. The source paper explicitly characterizes this as fundamentally harder than speech editing because the edited content must remain synchronized to musical time and pitch [2606.30580].

Within MeloDISinger’s pipeline, MeloDRP is placed in the acoustic model stage. After feature extraction and edit parsing, the acoustic model predicts edited phoneme durations $D_{\mathrm{edit}}$ with MeloDRP and edited-region pitch $P_{\mathrm{edit}}$ with FPIP. These signals condition a flow-matching mel decoder, which infills the masked regions and merges them into the original mel-spectrogram. The system design therefore makes duration prediction an explicit, constrained subproblem rather than an implicit byproduct of acoustic generation.

A key implication is that MeloDRP is not merely a duration estimator. It is the mechanism by which SVE’s locality and synchronization constraints are enforced at span level, before mel synthesis occurs.

## 2. Span-wise fixed-budget duration modeling

MeloDRP reformulates duration prediction as **reallocation of a fixed time budget** within each edited span. Conventional duration predictors output absolute durations and do not enforce a prescribed total duration; in SVE, this is inadequate because each edited span must fit the original time slot exactly [2606.30580].

Let edited spans be indexed by $i=1,\dots,N$. Each span $i$ has a time budget $T_i$ and a target count $E_i$ of phonemes to render inside that span. Instead of predicting absolute phoneme durations directly, MeloDRP predicts a ratio vector $r_i=\{r_{ij}\}_{j=1}^{E_i}$ satisfying nonnegativity and unit sum, and reconstructs phoneme durations as

$$
\sum_{j=1}^{E_i} r_{ij} = 1, \qquad \hat d_{ij} = T_i r_{ij}.
$$

By construction,

$$
\sum_j \hat d_{ij} = T_i,
$$

so span-wise duration preservation is guaranteed.

The span budget $T_i$ depends on the edit operation:

- **Replacement**: the original replaced span’s duration is reallocated to the new phonemes.
- **Insertion**: duration is collected from a local neighborhood and reallocated across the inserted and neighboring phonemes.
- **Deletion**: the deleted span’s duration is assigned to a silence phoneme.

These budgets are provided to MeloDRP as a phoneme-level budget sequence $b$, where $b_p=T_i$ for phonemes inside span $i$ and $b_p=0$ elsewhere.

Fixed-budget enforcement is achieved through a span-wise softmax over duration logits:

$$
r_{ij}=\frac{\exp(z_{ij})}{\sum_{k=1}^{E_i}\exp(z_{ik})}.
$$

Multiplication by $T_i$ then yields $\hat d_{ij}=T_i r_{ij}$. The paper emphasizes that this satisfies the fixed-budget constraint in both training and inference, and that no extra constraints such as Dirichlet distributions are required [2606.30580].

This formulation decouples **how a span’s time is divided** from **how much total time is available**, which focuses learning on relative timing structure rather than absolute duration scale.

## 3. Input representations and cross-modal architecture

MeloDRP uses two synchronized input streams: a **phonetic stream** and a **melodic stream**.

The phonetic stream begins with conversion of $L_{\mathrm{edit}}$ to phonemes via **g2p-en**. During training, phoneme identities and features for the original sequence are available from **MFA (Montreal Forced Aligner)**. Linguistic features include start flags with values **2/1/0** for word-initial, syllable-initial but non-word-initial, and others, as well as coarse phoneme types defined by broad classes of manner of articulation, with vowels including stress markers. These are embedded and combined with positional encodings. The system uses dropout of phoneme identity and type embeddings with $p=0.3$ to improve robustness [2606.30580].

The melodic stream is built from **pseudo-MIDI context** derived from the original singing. Frame-level $F_0$ with voiced/unvoiced flags is extracted from $S_{\mathrm{orig}}$, then converted into pseudo-MIDI notes through MIDI quantization of $F_0$, note segmentation, and post-processing. The note tokens carry pitch, duration, onset times, and optional beat or positional cues implied by note ordering and timing. Because MFA supplies phoneme-level time intervals for the original audio, phonemes and pseudo-MIDI notes share the same timeline and can be matched through temporal overlap. The system explicitly does **not** enforce one-to-one phoneme–note mapping; instead, it learns soft correspondences [2606.30580].

The architecture is a two-stream Transformer with cross-attention fusion:

| Module | Configuration | Function |
|---|---|---|
| Phoneme encoder | 4-layer Transformer, hidden size 256, 2 attention heads | Encodes edited phoneme sequence and linguistic cues |
| Melody encoder | 4-layer Transformer, hidden size 256, 2 heads | Encodes pseudo-MIDI note sequence |
| Duration-ratio decoder | 6-layer Transformer decoder, hidden size 256, 2 heads | Produces span-local duration logits |

Cross-attention uses the phoneme-side representation as queries and the melody-side representation as keys and values:

$$
A = \operatorname{softmax}(QK^\top / \sqrt{d}), \qquad Y = AV.
$$

Here, $Q$ comes from encoded phoneme tokens within each edited span, while $K,V$ come from encoded pseudo-MIDI note tokens covering the same global time window. The stated rationale is that duration must be allocated among phonemes according to both linguistic role and alignment to performed melodic events such as note onsets, sustains, and rhythm. Using phonemes as queries allows each phoneme to retrieve the most relevant melodic context for its ratio decision [2606.30580].

## 4. Temporal-overlap supervision and objective design

To make cross-attention reflect musically meaningful alignments, MeloDRP introduces **temporal-overlap supervision**. If phoneme $i$ covers $[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}]$ and note $j$ covers $[T_j^{\mathrm{start}}, T_j^{\mathrm{end}}]$, a general overlap weight could be written as

$$
W_{i,j} =
\frac{\left|[t_i^{\mathrm{start}}, t_i^{\mathrm{end}}] \cap [T_j^{\mathrm{start}}, T_j^{\mathrm{end}}]\right|}
{t_i^{\mathrm{end}}-t_i^{\mathrm{start}}}.
$$

In MeloDRP, the supervised target is simplified to a binary overlap mask:

$$
W_{i,j}=1 \text{ if the intervals overlap by any positive length, else } 0.
$$

The cross-attention matrix $A$, aggregated over heads, is encouraged to match $W$ through an $L_1$ guided-attention loss. This promotes higher attention on time-overlapping phoneme–note pairs and lower attention otherwise [2606.30580].

Ground-truth duration supervision is also span-normalized. If $d^*_{ij}$ is the ground-truth phoneme duration within span $i$ and $D_i^*=\sum_{k=1}^{E_i} d^*_{ik}$ is the span duration, the target ratios are

$$
r^*_{ij} = \frac{d^*_{ij}}{D_i^*}.
$$

The MeloDRP objective is

$$
L_{\mathrm{MeloDRP}} = \sum_{q\in Q}\lambda_q L_q, \qquad Q=\{ph, wd, pen, ga\}.
$$

Its components are:

- **Phoneme-level ratio loss ($ph$)**: KL divergence per span between target and predicted ratios.
- **Word-level aggregation loss ($wd$)**: $L_1$ loss after summing phoneme ratios within each word.
- **Penalty loss ($pen$)**: penalizes phonemes whose predicted duration $\hat d_{ij}=T_i r_{ij}$ falls below a minimum threshold $d_{\min}$.
- **Guided-attention loss ($ga$)**: $L_1$ loss between attention $A$ and the binary overlap mask $W$.

The paper’s formulation makes the modeling priorities explicit: span-level duration conservation is enforced algebraically; ratio learning captures intra-span timing structure; word aggregation regularizes higher-level rhythm; and overlap supervision anchors cross-modal attention to the shared timeline [2606.30580].

## 5. Integration with flow-matching audio infilling

MeloDRP’s outputs are used to convert the edited phoneme sequence into a frame-level conditioning stream for a non-autoregressive conditional flow-matching mel decoder. Each phoneme embedding is repeated for its predicted frame count, so the decoder receives timing-conditioned phoneme features at the intended rhythm within each span’s fixed budget [2606.30580].

The decoder conditions on the sum of frame-level embeddings from phoneme, pitch, speaker, and context mel. Training uses random edit masks, and the conditional flow-matching objective is

$$
L_{\mathrm{CFM}} = \mathbb{E}_{x_1,t,x_0}
\left[
\left\|
\left(v_\theta(x_t,t,c)-u\right)\odot m_{\mathrm{edit}}
\right\|_2^2
\right],
$$

where $x_0\sim N(0,I)$, $x_1=x_{\mathrm{orig}}$, $u=x_1-x_0$, and $x_t=(1-t)x_0+t x_1$.

At inference, the decoder starts from Gaussian noise, solves the learned ODE to sample $\hat x_{\mathrm{gen}}$ for masked frames, and merges the result into the original mel-spectrogram as

$$
x_{\mathrm{edit}} =
m_{\mathrm{edit}}\odot \hat x_{\mathrm{gen}} +
(1-m_{\mathrm{edit}})\odot x_{\mathrm{orig}}.
$$

Because non-edited regions never pass through the generative pathway, they remain **bit-identical** to the original, while edited regions are synthesized under strong conditioning from surrounding mel context, predicted durations, and pitch [2606.30580].

The full inference pipeline combines alignment, edited-lyric generation, edit parsing, duration budgeting, MeloDRP prediction, pitch conditioning, mel infilling, and waveform generation. Alignment uses **WhisperX** for word-level onset/offset times and **MFA** for phoneme-level durations. Duration-aware edited-lyric generation converts each word slot duration $\Delta t$ into a syllable capacity

$$
C = \left\lfloor \alpha \Delta t / \tau_{\min} \right\rfloor,
$$

with $\tau_{\min}=0.3\,\mathrm{s}$ and $\alpha=0.8$, and an LLM rewrites lyrics subject to these capacities. The final waveform is produced by **PC-NSF HiFi-GAN** at **44.1 kHz** with window **2048**, hop **512**, and **128 mel bins** [2606.30580].

## 6. Empirical characteristics and comparative significance

The reported experiments use **GTSinger-En**, a dataset of **13 hours** of English singing from **three singers** spanning **six singing techniques**. Audio is segmented into chunks of at most **11.6 s** or **1000 mel frames**, with word boundaries preserved via MFA durations. Feature extraction uses **Resemblyzer** for speaker embeddings and **Parselmouth** for $F_0$; frames with $F_0<3$ are treated as unvoiced. Training uses Adam with $\beta_1=0.9$, $\beta_2=0.999$, learning rate $10^{-4}$, batch size **16**, and MultiStepLR with **0.5** decay at **10k/20k/30k** steps. Edit masks are sampled with $r\sim U(0.3,0.7)$, and inference uses **100 Euler steps** [2606.30580].

Evaluation covers intelligibility, duration preservation, and melody following. Objective metrics are **WER** and **CER** from Whisper-large-v3, **Duration Consistency (DC)**, **Duration Difference (DDUR)**, and **F0 Pearson Correlation (FPC)** with Cut and DTW variants. Across insertion, deletion, replacement, and mixed edits, the system reports **DDUR $\approx 0.00$ s** and **DC $\approx 99.93\%$**, alongside improved intelligibility and melody following relative to prior baselines. Representative examples include **Rep-S** with **WER/CER 21.88/15.26**, **Ins** with **18.57/11.62**, and **Del** with **24.88/15.74**. For melody following, the insertion setting reports **FPC-Cut/DTW 77.71/71.14**. Subjective evaluation with **22 listeners** gives, for **Rep-SM**, **Lyric 4.05±0.23**, **Melody 3.99±0.20**, and **Naturalness 3.65±0.19**, with significant improvement over the baselines [2606.30580].

The ablation study isolates the contribution of MeloDRP’s design choices. Removing total-duration conditioning produces the largest degradation, for example on mixed edits with **WER/CER 44.7/31.6** versus **39.4/27.6** in the full system. Removing melody conditioning degrades replacement and insertion settings; removing phoneme features notably hurts insertion; and removing guided attention weakens sensitivity to melodic context. The paper interprets these results as evidence for three main novelties: **fixed-budget, span-wise control**, **melody-aware allocation**, and **soft alignment** without one-to-one phoneme–note assumptions [2606.30580].

## 7. Limitations, failure modes, and terminology

The source paper identifies several limitations. MeloDRP is sensitive to alignment errors in MFA phoneme timings, WhisperX word boundaries, and the $F_0$-to-MIDI conversion pipeline. Such errors can misguide attention and duration allocation, particularly in rapid ornaments or vibrato-heavy segments. Extreme edits that introduce large syllable-count changes within very short budgets may force many sub-threshold phonemes; the penalty loss mitigates this but does not eliminate intelligibility trade-offs. The binary overlap mask used for guided attention is deliberately simple; a length-weighted overlap could provide finer supervision, but it is not used. Computationally, the architecture itself is lightweight, but inference with **100 Euler steps** in the flow-matching decoder is the main cost, although infilling only edited spans reduces practical latency [2606.30580].

The acronym is also potentially ambiguous across literatures. In the singing voice editing work, **MeloDRP** denotes **Melody-aware Duration Ratio Predictor**. By contrast, in last-mile logistics the standard acronym is **MDRP**, standing for **Meal Delivery Routing Problem**, and that paper explicitly states that it does not use “MeloDRP” [2408.06353]. A plausible implication is that acronym expansion should be checked carefully when moving between music generation and operations research contexts.

Source: https://www.emergentmind.com/topics/melodrp