Melodia: Training-Free Music Editing

• Melodia is a training-free music editing methodology that leverages diffusion models and selective self-attention to modify attributes while preserving temporal structure.
• It employs an innovative attention repository to reapply SA maps during the editing process, ensuring that key melodic and rhythmic elements remain intact.
• Melodia introduces composite metrics (ASB and AMB) that balance prompt adherence and musical integrity, achieving state-of-the-art performance on diverse datasets.

Melodia is a training-free music editing methodology that achieves attribute modification—such as instrument, genre, or mood change—on audio via diffusion models, while preserving the original temporal structure (melody, rhythm) of a source recording. It is characterized by its attention-probing analysis in the AudioLDM 2 backbone, the selective manipulation of self-attention (SA) maps, and the use of an attention repository for structure preservation, obviating the need for source text descriptions. Melodia contributes new metrics for quantifying the fidelity of music edits and demonstrates state-of-the-art performance in both objective and subjective evaluations across multiple music datasets (Yang et al., 11 Nov 2025).

1. Background and Motivation

Conventional text-to-music editing and generation methods, including those based on cross-attention (CA) intervention or edit-based inversion pipelines such as MusicMagus (Zhang et al., 2024) and MusRec (Boudaghi et al., 6 Nov 2025), often suffer from a trade-off between text-prompt adherence and the preservation of the source audio's melodic and rhythmic integrity. Prior approaches predominantly focus on conditioning edits via CA layers, which, as Melodia demonstrates, are highly entangled with prompt semantics and editing attributes but ill-suited for conserving fine-grained temporal structure.

Melodia addresses a key limitation: the inability of CA-targeted interventions to preserve temporal structure during music editing. The central insight, derived from rigorous attention-probing, is that SA maps, rather than CA, are responsible for encoding and maintaining musical structure. This observation underpins Melodia’s architectural and algorithmic design (Yang et al., 11 Nov 2025).

2. Mathematical Foundations and Attention Manipulation

Melodia operates within the latent diffusion paradigm using AudioLDM 2 as a backbone. The denoising UNet incorporates both cross- and self-attention modules at various layers and timesteps. Let $z_t \in \mathbb{R}^{N \times d_e}$ be the latent feature at diffusion step $t$.

• Cross-attention (CA):

$$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$

with $Q^c = W_{Q^c} \phi(z_t)$ and $K^c, V^c$ derived from the text embedding $\tau(y)$.

• Self-attention (SA):

$$\phi^{s}(z_t) = \text{SelfAttention}(Q^s, K^s, V^s) = M^s V^s,\ \text{where}\ M^s = \text{Softmax}(Q^s {K^s}^\top / \sqrt{d^s})$$

with $Q^s, K^s, V^s$ obtained from $\phi(z_t)$ itself.

Melodia’s central manipulation occurs during editing: SA maps from specific layers (8–14 by default) and up to a user-chosen horizon $T_{\mathrm{start}}$ are extracted from a partial inversion of the source audio and then reapplied at the corresponding diffusion steps during the edit. Mathematically, at each $t$0,

$t$1

where $t$2 and $t$3 are from the source inversion, while $t$4 is from the current edited latent. This operation ensures that the temporal dependencies (e.g., melody, rhythm) from the source are maintained throughout the edit trajectory.

3. Attention Probing and Repository Construction

A comprehensive probing analysis was conducted on both CA and SA maps using classification accuracy to determine the amount of information retained about prompt attributes (instrument, style, mood):

• CA maps display high prompt classification accuracy (70–100%), confirming that they capture semantic attribute information and control the attribute-editing locus.
• SA maps yield low classification accuracy (<40%), showing that they do not encode attribute information but, as results confirm, encode temporal structure.

This empirical distinction motivates the construction of an attention repository: during a partial DDIM inversion phase, at each step $t$5 to $t$6, the corresponding $t$7 from SA layers 8–14 are stored. During the editing (“reverse denoising”) process, these keys and queries are reapplied at each $t$8 as prescribed in the pseudocode:

$$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$5

4. Evaluation Metrics

Melodia introduces two composite metrics for music editing assessment, designed to jointly reward textual adherence and structure retention, penalizing unbalanced trade-offs:

• Adherence–Structure Balance (ASB):

$t$9

where $$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$0 is prompt-adherence; $$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$1 (lower is better) measures perceptual structure loss.

• Adherence–Musicality Balance (AMB):

$$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$2

with $$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$3 quantifying harmony and pitch-contour preservation.

These metrics, alongside conventional CLAP, Chroma similarity, LPAPS, and Fréchet Audio Distance (FAD), provide a multi-faceted assessment across datasets such as MusicDelta, zoME-Bench, and a supplementary real/synthesized mixed set.

5. Experimental Validation

Quantitative evaluations on MusicDelta and zoME-Bench demonstrate that Melodia achieves the best or near-best scores on CLAP (semantic adherence), LPAPS (structure preservation), FAD, and especially on ASB/AMB, indicating minimal trade-off between attribute transfer and musical integrity. For example, on MusicDelta:

Metric	Melodia	Best Baseline
CLAP	0.34	0.35
LPAPS	4.01	4.01
Chroma	0.32	0.32
FAD	0.56	0.56
ASB	1.00	1.00
AMB	1.00	1.00

Subjective results, aggregated across $$\phi^{c}(z_t, y) = \text{CrossAttention}(Q^c, K^c, V^c) = M^c V^c,\ \text{where}\ M^c = \text{Softmax}(Q^c {K^c}^\top / \sqrt{d^c})$$4 listening participants, indicate Melodia yields the highest mean scores in relevance to the target prompt (REL), structural consistency (CON), and music-editing balance (MEB), with REL ≈ 3.2–3.4/5 and CON ≈ 3.5–3.7/5.

An ablation study demonstrates optimal ASB/AMB balance when SA map replacement is performed in layers 8–14. Furthermore, generalization to other diffusion backbones (e.g., Stable Audio Open) shows consistent improvements in all major metrics.

6. Positioning Among Related Methods

Melodia is distinguished from prior training-free music editing systems by its exclusive and principled manipulation of SA maps, as opposed to CA or latent-embedding shifts. MusicMagus employs Δ-editing in text embeddings and a cross-attention consistency penalty, which controls attributes but is less effective for structure preservation (Zhang et al., 2024). MusRec utilizes rectified-flow inversion and attention-feature injection, primarily targeting self-attention in transformer architectures for zero-shot text-driven editing (Boudaghi et al., 6 Nov 2025). AudioEditor (Jia et al., 2024) and self-attention-based style transfer (Kim et al., 2024) manipulate latent space or SA features, but Melodia's layer- and timestep-specific repository mechanism and new balance metrics provide improved control and assessment of editing trade-offs.

7. Limitations and Future Directions

Melodia relies on the structure and invertibility of the AudioLDM 2 backbone; extension to other architectures (e.g., transformers, autoregressive decoders) depends on the availability of comparable SA modules. The scope of demonstrated edits is attribute-focused (timbre, genre, mood), and the approach has not been explicitly validated for local or time-varying edits (e.g., masking, segment-based operations). Possible future directions include adaptive SA layer/timestep selection, hybrid attention interventions, expansion to multi-stem and long-form editing, and application to tasks with more complex structure–attribute interactions (Yang et al., 11 Nov 2025).