Melody-Guided Music Generation (MG²)

• Melody-Guided Music Generation (MG²) is a family of techniques that use melody or its abstractions as explicit control signals for synthesizing and editing music.
• MG² methods employ structured conditioning objects such as melodic skeletons, chord embeddings, and form templates to guide both symbolic and audio-domain generation.
• The approach factorizes music creation into stages—first establishing a control framework and then refining details—achieving improved coherence and balance between melody preservation and creative neural refinement.

Melody-Guided Music Generation (MG²) denotes a family of music generation and editing methods in which melody, or a melody-derived abstraction, functions as an explicit control signal for synthesis, accompaniment, refinement, or cross-modal generation. In symbolic settings, the guide may be a melodic skeleton, a basic melody, a motif-to-section form template, or a bar-level chord summary extracted from melody; in audio settings, it may be a retrieved melody embedding or a time-varying top-$k$ constant-Q Transform prompt. Across these formulations, MG² replaces monolithic left-to-right note generation with factorized procedures such as $P(M)=P(S)\cdot P(D\mid S)$, melody-first harmony generation, expert-system form enforcement followed by neural refinement, or text-conditioned diffusion augmented by melody control (Zhang et al., 2023, Dai et al., 2021, Goren et al., 2021, Lu et al., 2022, Wei et al., 2024, Hou et al., 2024).

1. Conditioning Objects and Structural Abstractions

A central property of MG² is that “melody” is not restricted to a literal note sequence. In WuYun, the conditioning object is a melodic skeleton $S=\{s_1,\dots,s_K\}$, extracted from a full melody $M=\{n_1,\dots,n_T\}$ by combining a rhythmic skeleton and a tonal skeleton. The rhythmic skeleton selects notes on strong metrical accents or agogic accents; the tonal skeleton segments the melody into small rhythmic cells and retains, in each cell, the note minimizing tension in the spiral array model. The resulting skeleton notes are represented in the same MeMIDI event vocabulary as full notes, but the sequence contains only the tokens for notes in $S$ plus bar and position tokens (Zhang et al., 2023).

MusicFrameworks uses a broader hierarchical abstraction. A melody is decomposed into sections $S_1,\dots,S_J$ and phrases $P_{j,1},\dots,P_{j,K_j}$, with phrase-level conditioning variables comprising a Basic Melody $BM_{j,k}$, a Basic Rhythm Form $BRF_{j,k}$, and a repetition token $R_{j,k}$ indicating approximate phrase repetition. The Basic Melody is a scale-degree pitch sequence sampled at a fixed half-note rhythm, while the Basic Rhythm Form stores one of 256 possible two-beat rhythm-pattern labels together with a numerical complexity score (Dai et al., 2021).

MeloForm pushes the abstraction one level higher by making musical form itself the guide. A musical work $P(M)=P(S)\cdot P(D\mid S)$0 is defined as a sequence of sections $P(M)=P(S)\cdot P(D\mid S)$1; each section contains phrases; each phrase is developed from a motif $P(M)=P(S)\cdot P(D\mid S)$2 that is 1–2 bars long. A form $P(M)=P(S)\cdot P(D\mid S)$3 prescribes phrase repetitions and variations inside sections, for example $P(M)=P(S)\cdot P(D\mid S)$4. The expert system realizes $P(M)=P(S)\cdot P(D\mid S)$5 exactly by construction before a Transformer-based refinement model improves melodic richness without changing the form (Lu et al., 2022).

In A-Muze-Net, the guide is neither a skeleton nor a formal template but a bar-level chord representation derived from the generated right-hand melody. Given the set of pitch-classes $P(M)=P(S)\cdot P(D\mid S)$6 in bar $P(M)=P(S)\cdot P(D\mid S)$7, the system selects $P(M)=P(S)\cdot P(D\mid S)$8, where $P(M)=P(S)\cdot P(D\mid S)$9 and $S=\{s_1,\dots,s_K\}$0 is a 253-chord dictionary. The left-hand harmony network conditions on this bar-level chord embedding, so the melody serves as a compressed guide for accompaniment generation (Goren et al., 2021).

Recent audio-domain MG² systems generalize the conditioning object further. In the diffusion-based MG² model for text-to-music, melody enters both implicitly and explicitly: implicitly through Contrastive Language–Music Pretraining, which aligns text, waveform, and melody embeddings in a common space, and explicitly through retrieval of a nearest-neighbor melody embedding conditioned on the input text (Wei et al., 2024). In ControlNet-augmented Diffusion Transformer editing, the melody prompt is represented as a stereo top-$S=\{s_1,\dots,s_K\}$1 CQT sequence, where for each frame and each channel the top 4 CQT bins are retained and embedded before being injected into a ControlNet branch (Hou et al., 2024).

2. Factorized Symbolic Generation Pipelines

WuYun formalizes symbolic MG² as a two-stage probabilistic decomposition. The full melody probability is factorized as

$S=\{s_1,\dots,s_K\}$2

where $S=\{s_1,\dots,s_K\}$3 is the melodic skeleton and $S=\{s_1,\dots,s_K\}$4 is the decorative infill. Stage 1 trains a decoder-only Transformer-XL for

$S=\{s_1,\dots,s_K\}$5

with top-$S=\{s_1,\dots,s_K\}$6 temperature sampling at inference ($S=\{s_1,\dots,s_K\}$7, $S=\{s_1,\dots,s_K\}$8). Stage 2 uses a Transformer encoder–decoder to model

$S=\{s_1,\dots,s_K\}$9

copying skeleton notes through and generating only the non-skeleton notes. Training minimizes

$M=\{n_1,\dots,n_T\}$0

with $M=\{n_1,\dots,n_T\}$1 (Zhang et al., 2023).

MusicFrameworks adopts a three-step phrase-level factorization. First, a basic melody generator predicts half-note scale-degree pitches from positional and chord context. Second, a realized-rhythm generator predicts two-beat rhythm patterns conditioned on rhythm-form descriptors and positional features. Third, a realized-melody generator predicts final pitches conditioned on durations, chords, the basic melody, and positional indicators including sixteenth-note offset. Repetition is handled explicitly: when a phrase is labeled as repeated, generation enforces a Dynamic Time Warping contour similarity $M=\{n_1,\dots,n_T\}$2 (Dai et al., 2021).

MeloForm separates structural correctness from surface richness. Its expert system first builds a melody from motif to phrase to section according to a prescribed form $M=\{n_1,\dots,n_T\}$3. Phrase development uses sequence, transformation, and ending operators; section assembly reuses earlier phrases either as exact repetitions $M=\{n_1,\dots,n_T\}$4 or as variations $M=\{n_1,\dots,n_T\}$5. The refinement stage is an encoder-attention-decoder Transformer that operates phrasewise. Training masks similar phrases in a MASS-style scheme, and the decoder reconstructs the original pitch tokens while conditioning on cadence, rhythm tokens, chord tokens, and special $M=\{n_1,\dots,n_T\}$6 and $M=\{n_1,\dots,n_T\}$7 tokens (Lu et al., 2022).

These systems share a common MG² principle: the guide is generated or specified first, and subsequent modules are constrained to elaborate rather than replace it. In WuYun, the decoder emits decorative notes around fixed skeleton notes; in MusicFrameworks, the final melody realizes a precomputed contour and rhythm scaffold; in MeloForm, the refinement model is designed not to rewrite musical form. A plausible implication is that MG² methods treat melody not only as content but also as a carrier of long-range structural invariants.

3. Melody-First Harmony and Accompaniment

A-Muze-Net instantiates MG² in accompaniment generation rather than single-line melody realization. The architecture contains two networks: a right-hand melody network and a left-hand harmony network, each with two stacked LSTM layers of hidden size 128 and dropout 0.5 on the second layer’s outputs. The right-hand network predicts the next token from a vocabulary of approximately 1,161 discrete pitch-duration tokens using an embedding layer $M=\{n_1,\dots,n_T\}$8 and a linear-softmax output layer. The left-hand network augments the same note embedding with a chord embedding $M=\{n_1,\dots,n_T\}$9 for $S$0 chord classes, so that

$S$1

Both networks are trained with teacher-forcing and cross-entropy losses $S$2 and $S$3 (Goren et al., 2021).

The representation is scale-invariant. If $S$4 is the MIDI note number and $S$5 is the tonic, the relative pitch is

$S$6

Durations are quantized into nine common lengths, giving a joint pitch-duration vocabulary of size $S$7 plus break tokens. This design is used both for right-hand generation and for conditioning left-hand accompaniment on bar-level melodic content (Goren et al., 2021).

After both hands are generated, A-Muze-Net optionally applies a heuristic enrichment step. For each generated right-hand token, the system may add a simultaneous perfect fifth, or a major/minor third chosen to remain inside the scale; for out-of-scale melody notes, it may duplicate the pitch in a neighboring octave. For the left hand, there is only a $S$8 chance of adding an additional simultaneous note. This enrichment is explicitly described as random note addition to “flesh out” the harmony (Goren et al., 2021).

Within the MG² landscape, this system illustrates a narrower but important meaning of melody guidance: melody need not guide note infilling alone; it can also guide harmonic realization. The reported ablations support this interpretation: removing chord conditioning lowers UPC from 9.54 to 7.30 and raises TD from 0.86 to 0.95, while replacing chord embeddings with sums of note embeddings lowers UPC to 6.80 and yields TD of approximately 0.90 (Goren et al., 2021).

4. Audio-Domain MG²: Retrieval, Diffusion, and ControlNet

The 2024 MG² model extends melody guidance to text-to-music generation. Its first stage, Contrastive Language–Music Pretraining, aligns waveform, text, and melody into a shared embedding space using six pairwise InfoNCE losses:

$S$9

Audio features are derived from a frozen HTS-AT encoder followed by a trainable 2-layer MLP, text features from frozen RoBERTa followed by a trainable 2-layer MLP, and melody features from a small randomly initialized MLP pooler. Retrieval is then performed by nearest-neighbor search in the learned embedding space, using FAISS/HNSW, and the retrieved melody embedding is concatenated with the text embedding to form a diffusion conditioning vector $S_1,\dots,S_J$0 (Wei et al., 2024).

The generative backbone is a latent diffusion model similar to AudioLDM. It uses a DDIM schedule,

$S_1,\dots,S_J$1

with a denoising network $S_1,\dots,S_J$2 trained by per-step denoising MSE. Classifier-free guidance is applied with weight $S_1,\dots,S_J$3, and inference begins from pure noise $S_1,\dots,S_J$4 before decoding with an AudioLDM-style VAE and HiFi-GAN vocoder (Wei et al., 2024).

ControlNet-based Diffusion Transformer editing addresses a different MG² problem: joint control by text and melody for long-form and variable-length generation or editing. The backbone is StableAudio-Open’s DiT, operating on codec latents rather than Mel-spectrograms. Melody is injected through a ControlNet-style branch formed by cloning the first $S_1,\dots,S_J$5 transformer blocks of the pre-trained DiT. Control features are merged into the frozen path via a zero-initialized linear layer $S_1,\dots,S_J$6, so the melody branch perturbs the denoising trajectory in proportion to the control signal (Hou et al., 2024).

Its melody representation is a stereo top-$S_1,\dots,S_J$7 CQT. Using $S_1,\dots,S_J$8, $S_1,\dots,S_J$9 bins, and hop length 512, the model computes a CQT and, for each frame and each stereo channel, retains the indices of the top 4 magnitude bins. These indices are embedded, downsampled by 1D convolutions to match latent spatial resolution, and fed into the ControlNet branch. Training uses a progressive curriculum masking strategy over masking ratios $P_{j,1},\dots,P_{j,K_j}$0 so that early training emphasizes text-only generation and later training exposes more melody information. The paper states that training with full melody prompts from the start causes the model to ignore text, whereas gradual unmasking yields more stable learning (Hou et al., 2024).

Together, these two systems broaden MG² from symbolic note generation to cross-modal control. One model uses retrieved symbolic melody embeddings to improve text-to-music diffusion; the other uses time-varying pitch prompts to edit or generate audio directly. The shared principle is that melody provides a control channel orthogonal to text semantics.

5. Data, Representations, and Reported Results

The empirical literature on MG² spans distinct datasets, representations, and evaluation regimes. Symbolic systems emphasize phrase structure, formal consistency, or human subjective judgments; accompaniment systems report note validity and tonal coherence; audio-domain systems emphasize FAD, KL, IS, CLAP-based alignment, and melody preservation.

System	Data / representation	Reported result
WuYun (Zhang et al., 2023)	Wikifonia, MeMIDI, 2,921 pieces	WuYun-RS outperforms the best end-to-end baseline by 0.51 points on average across five subjective metrics
MeloForm (Lu et al., 2022)	Musical form templates, expert system + Transformer refinement	formAcc = 97.79%; avg-pitchAcc = 92.5%; pitch-spanAcc = 99.71%
A-Muze-Net (Goren et al., 2021)	Scale-invariant MIDI, two-hand piano	QN = 100%; UPC = 9.54; TD = 0.86; OOS = 29%
MusicFrameworks (Dai et al., 2021)	POP909, section/phrase structure, BM/BRF	Basic-Melody Accuracy = 38.7%; Rhythm Accuracy = 50.1%; Realized-Melody Accuracy = 55.2%
MG² (Wei et al., 2024)	MusicCaps + MusicBench, 132 h, latent diffusion	FAD/KL best on both datasets: 1.91/1.21 on MusicCaps and 0.99/1.07 on MusicBench
ControlNet DiT (Hou et al., 2024)	59,955 instrumental clips, 2,239.7 h	Edit Melody Accuracy = 56.6%; Edit FD = 97.73; KL = 0.265; CLAP = 0.396

WuYun’s evaluation is notable because its objective feature-distribution overlaps between generated and real melodies show “no conclusive trend,” whereas subjective evaluation on Rhythm, Richness, Structure, Expectation, and Overall favors the rhythmic-skeleton variant. In its ablation over nine skeleton settings, the best performance is obtained with the true rhythmic skeleton, comprising 33.8% of notes; one-tailed $P_{j,1},\dots,P_{j,K_j}$1-tests yield $P_{j,1},\dots,P_{j,K_j}$2 in all comparisons except real-skeleton inpainting (Zhang et al., 2023).

MeloForm evaluates musical form directly. Its expert-system-plus-refinement design achieves 97.79% form accuracy without any labelled musical form data, and its subjective improvements over Music Transformer, MELONS, and a simple POP909_lm baseline are reported as +0.75 in Structure, +0.50 in Thematic, +0.86 in Richness, and +0.89 in Overall Quality. It also supports multiple explicit forms, including verse and chorus, rondo, variational, and sonata (Lu et al., 2022).

MusicFrameworks reports objective gains over a vanilla Transformer in all three subtasks: 38.7% versus 30.9% for basic-melody prediction, 50.1% versus 25.8% for rhythm prediction, and 55.2% versus 39.3% for realized-melody prediction. Ablations without music-framework conditioning fall to 33% rhythm accuracy and 37% melody accuracy. In a listening test with 196 listeners evaluating 105 one-minute song sections, roughly 50% of generated melodies are judged as good as or better than human-composed excerpts (Dai et al., 2021).

In audio generation, MG² reports 416 M parameters and approximately 132 h of training audio, yet achieves the best FAD and KL on both MusicCaps and MusicBench relative to open-source baselines listed in the paper. It also reports stronger cross-modal retrieval than CLAP: for example, on MusicCaps, CLMP attains 0.741 / 0.820 for waveform-to-text and 0.705 / 0.798 for text-to-waveform, compared with 0.523 / 0.627 and 0.515 / 0.626 for CLAP. Removing explicit melody conditioning degrades FAD and KL by approximately 10–30% (Wei et al., 2024).

The ControlNet DiT system reports stronger editing-oriented melody preservation than MusicGen baselines. On the “Song Describer” benchmark, it attains text-to-music FD 158.53, KL 0.623, and CLAP 0.375, and in editing it attains melody accuracy 56.6%, FD 97.73, KL 0.265, and CLAP 0.396. Subjective MOS scores are 3.4/3.6 for text fit and overall quality in text-to-music, and 3.7/3.8 in editing (Hou et al., 2024).

6. Scope, Misconceptions, and Open Problems

A common misconception is that MG² refers to a single architecture. The literature instead shows a family of decompositions. In some systems, melody is the target to be scaffolded by skeletons or form; in others, melody is the source signal that drives harmony, retrieval, or editing. The conditioning object can be white-box and symbolic, as in WuYun and MeloForm, or learned and latent, as in the diffusion systems (Zhang et al., 2023, Lu et al., 2022, Wei et al., 2024).

Another misconception is that “knowledge-enhanced” MG² necessarily means extensive rule injection into neural networks. WuYun explicitly states that the only explicit knowledge injected is the skeleton $P_{j,1},\dots,P_{j,K_j}$3 itself; no additional hand-coded rules or bias terms are used inside the Transformer. Conversely, MeloForm relies on a deterministic expert system for form enforcement before neural refinement. These two cases show that MG² can place musical knowledge either outside the network as a preprocessing or generation scaffold, or inside the overall pipeline as an explicit symbolic module (Zhang et al., 2023, Lu et al., 2022).

The literature also distinguishes between guidance and copying. WuYun copies skeleton notes through but generates decorative notes in the gaps; MusicFrameworks allows users to swap in new chords, basic melodies, and rhythm forms so that the same models reinterpret the constraints; the ControlNet DiT paper explicitly introduces curriculum masking because full melody exposure from the first training steps leads to over-reliance on melody and underuse of text (Dai et al., 2021, Hou et al., 2024). This suggests that effective MG² often depends on carefully balancing invariance and flexibility rather than maximizing direct adherence to a prompt.

Open problems are stated explicitly across the cited works. WuYun proposes integrating more advanced masked-language pretraining for the inpainting stage, jointly learning the skeleton extractor, and injecting additional domain knowledge such as harmonic functions and phrase boundaries. A-Muze-Net identifies extension beyond two hands, replacement of hard bar-hash chord selection with a differentiable “soft-chord” encoder, and exploration of attention or Transformer-based variants while retaining scale invariance and melody-to-harmony decomposition. The diffusion-based MG² literature, by construction, raises further questions about how text relevance, melody preservation, and long-form coherence should be jointly optimized under retrieval and ControlNet conditioning (Zhang et al., 2023, Goren et al., 2021, Wei et al., 2024, Hou et al., 2024).