Melody-CoT: Token-Based Music Reasoning

• Melody-CoT is a class of methods that employs explicit musical tokens—such as pitch, duration, and quantized CLAP embeddings—for intermediate reasoning in music tasks.
• Approaches like MusiCoT and CoMelSinger use structured token chains to plan music generation and enable precise control in singing synthesis.
• The methods report improvements in both objective metrics (e.g., FAD, MOS) and subjective evaluations, while facing challenges on complex tasks due to token representation discrepancies.

Melody-Token-Based Chain-of-Thought (Melody-CoT) denotes a class of music and audio modeling approaches in which intermediate reasoning, planning, or control is carried by explicit musical representations rather than being left entirely implicit in direct next-token generation. In the literature summarized here, the term spans three closely related settings: Chain-of-Thought (CoT) reasoning in Large Audio-LLMs (LALMs) for music questions, where current systems still rely on audio and natural-language descriptions rather than symbolic melody tokens; analyzable music generation, where a model first predicts quantized “musical thoughts” derived from CLAP embeddings before generating audio tokens; and zero-shot singing voice synthesis, where lyric, pitch, and duration tokens provide structured and disentangled melody control (Ma et al., 13 Jan 2025, Lam et al., 25 Mar 2025, Zhao et al., 24 Sep 2025).

1. Conceptual scope

A common misconception is that CoT in music must be verbal. The reported systems instead use three distinct intermediate forms. In Audio-CoT, music inputs are audio, and any intermediate representations are natural language captions or reasoning steps. In MusiCoT, the chain consists of quantized representations of CLAP audio embeddings using residual vector quantization (RVQ). In CoMelSinger, lyric tokens, pitch tokens, duration tokens, and regulated frame-aligned pitch tokens provide the structured control path.

This distinction matters because the phrase “Melody-CoT” does not refer to a single canonical architecture. In one line of work, CoT is a prompting strategy for answering music questions; in another, it is a planning phase before audio token generation; in a third, it is a structured token-conditioning mechanism for singing synthesis. This suggests that Melody-CoT is best understood as a family of token-mediated intermediate reasoning or control mechanisms rather than a single formalism.

2. Audio-language reasoning antecedents

"Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio LLM" is described as the first exploration into integrating Chain-of-Thought reasoning into LALMs to enhance their reasoning ability across auditory modalities, with the music modality explicitly evaluated alongside sound and speech (Ma et al., 13 Jan 2025). The paper evaluates three CoT methods on the MMAU benchmark using Qwen2-Audio-7B-Instruct: Manual-CoT, Zero-Shot-CoT, and Desp-CoT.

The generic formulations are:

$\mathcal{E} = \{(A_i, I_i, C_i)\}_{i=1}^N$

$$C_\text{output} = f_\text{LALM}(\mathcal{E}, A_\text{input}, I_\text{input})$$

for Manual-CoT,

$$C_\text{output} = f_\text{LALM}(A_\text{input}, I_\text{input} + P)$$

for Zero-Shot-CoT, and

$$D = f_\text{Caption}(A_\text{input}), \quad C_\text{output} = f_\text{LALM}(A_\text{input}, D + I_\text{input})$$

for Desp-CoT.

For the music subset of MMAU, the reported results are as follows:

Method	Music Extraction	Music Reasoning
Baseline (normalized)	56.35%	53.59%
Manual-CoT	60.77%	50.33%
Zero-Shot-CoT	59.12%	52.94%
Desp-CoT	57.45%	53.59%

The corresponding music totals are 55.09% for the baseline, 55.99% for Manual-CoT, 56.29% for Zero-Shot-CoT, and 55.69% for Desp-CoT. Table 2 further shows small but consistent improvement from Self-Consistency, with music gains of +1–2%. The paper reports that CoT methods significantly improve performance on easy and medium tasks but encounter challenges with hard tasks, where reasoning chains can confuse the model rather than improve accuracy. It also reports a positive correlation between reasoning path length and accuracy, including in music tasks. At the same time, there is no explicit mention of using melody tokens or specialized symbolic melody representations as reasoning primitives; all representations remain at the level of audio features, automatically generated captions, or free-form language. This suggests a direct motivation for later work that moves the intermediate chain into discrete music-related token spaces.

3. MusiCoT and quantized “musical thoughts”

"Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generation" introduces MusiCoT, a chain-of-thought prompting technique tailored for music generation in autoregressive (AR) models (Lam et al., 25 Mar 2025). The central change is architectural rather than verbal: instead of predicting only the next audio token, the model first outlines an overall music structure through a chain of “musical thoughts,” and only then generates fine-level audio tokens. The work is built atop the MeLoDy framework, which combines a semantic LM and a diffusion model.

The intermediate chain is constructed from CLAP audio embeddings. The audio encoder processes music in fixed-length (10s) segments, outputting continuous embeddings, and each segment embedding is quantized via RVQ with multiple codebooks. The result is a flattened sequence of coarse-to-fine tokens:

$$\left[ c_\text{clap}^{(1,1)}, ..., c_\text{clap}^{(M,1)}, c_\text{clap}^{(1,2)}, ..., c_\text{clap}^{(M,L)} \right]$$

The conditioning sequence is reported as

$$\mathbf{C}_\text{clap} \oplus \operatorname{Emb}_\theta(\operatorname{Tokenizer}(\mathbf{c}_\text{tags})) \oplus \operatorname{Emb}_\theta(\operatorname{Tokenizer}(\mathbf{c}_\text{lyrics}))$$

and the prompting protocol uses special tokens <cot_bos> and <cot_eos> to separate the structure-token prediction phase from the audio-token prediction phase. The CoT tokens are trained as additional next-token prediction targets with cross-entropy loss. Because the structural tokens are temporally aligned at a fixed resolution, such as every 10s, the approach supports post hoc analysis of music structure and independent control of structure and fine detail.

Within the terminology used in the summarized literature, this is a Melody-CoT formulation because the chain is not natural language but a sequence of interpretable, structure-representing tokens generated before lower-level audio tokens. The reported interpretation is that the chain can encode melody, harmony, instrumentation, and section-level organization in an analyzable form.

4. Sampling, analyzability, and music generation performance

MusiCoT augments the planning phase with two dedicated sampling mechanisms: Dual-Temperature Sampling and Dual-Scale Classifier-Free Guidance (CFG) (Lam et al., 25 Mar 2025). The former applies separate softmax temperatures to structure tokens and audio tokens, with the example values 0.65 for MusiCoT tokens and 0.75 for audio. The latter rescales conditional and unconditional log probabilities differently for structure and audio stages, with tunable guidance scales $\lambda_1$ and $\lambda_2$.

The practical consequences are reported in three areas. First, the explicit structure-planning phase sets global aspects such as instrumental arrangements and style reference before detailing the audio, promoting long-term musical coherence and intentionality absent in standard AR next-token models. Second, MusiCoT supports music referencing by accepting variable-length audio inputs as optional style references and encoding them as CLAP tokens. Third, because the reference is abstracted as quantized structure tokens rather than raw waveform or semantic audio tokens, the method is reported to address copying issues.

The paper evaluates MusiCoT with objective and subjective metrics. FAD is CLAP-based and lower is better; MOS is rated by expert musicians on a 1–5 scale; Meta Audiobox-Aesthetic provides content scores CE, CU, PC, and PQ; and RTF measures efficiency. The reported comparison between the base model and the MusiCoT variant is exact: MeLoDy (baseline) has RTF 0.27, MOS 3.35, FAD 0.112, CE 7.49, CU 7.85, PC 6.38, and PQ 8.11, whereas MeLoDy + MusiCoT has RTF 0.27, MOS 3.72, FAD 0.102, CE 7.49, CU 7.87, PC 6.21, and PQ 8.11. The paper further states that MusiCoT-enabled models outperform both classic open-source models and top commercial systems in FAD (best), and are top-2 in MOS. Structural analyzability is also quantified: cosine similarity between generated CLAP tokens and instrument text anchors correlates with actual source track volumes, with Pearson $R$ values 0.5–0.7.

5. CoMelSinger and structured melody control in zero-shot singing synthesis

"CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance" transfers the Melody-CoT idea into zero-shot Singing Voice Synthesis (SVS) (Zhao et al., 24 Sep 2025). The problem setting differs from MusiCoT: the objective is not general music generation from a planning chain, but precise and disentangled melody control in a discrete codec modeling paradigm. The model replaces conventional text inputs with lyric and pitch tokens and is built on the non-autoregressive MaskGCT architecture.

The token inventory is explicit. Lyrics are converted to phoneme tokens, yielding a lyric sequence from a lyric vocabulary. Pitch information is quantized into discrete tokens from a pitch vocabulary. Note or phoneme durations are also converted to discrete tokens. To obtain frame-level alignment, each pitch token is repeated according to duration, producing regulated pitch tokens aligned with the acoustic sequence. The two-stage pipeline comprises T2S (Text-to-Semantic), which converts lyrics and an optional reference semantic prompt to semantic tokens, and S2A (Semantic-to-Acoustic), which converts semantic tokens, regulated pitch tokens, and an acoustic prompt to acoustic tokens. The core conditioning equation is

$\mathbf{e}_c = \mathbf{e}_p + \mathbf{e}_s$

where $$C_\text{output} = f_\text{LALM}(\mathcal{E}, A_\text{input}, I_\text{input})$$0 and $$C_\text{output} = f_\text{LALM}(\mathcal{E}, A_\text{input}, I_\text{input})$$1 are the pitch and semantic token embeddings.

A central technical problem is prosody leakage, in which pitch information becomes entangled within the timbre prompt. CoMelSinger addresses this with a coarse-to-fine contrastive learning strategy. At the sequence level, the model enforces that outputs generated from the same lyrics and melody but different timbre prompts retain the specified global melody. At the frame level, it perturbs a proportion of the pitch tokens and uses cosine-similarity-based supervision to enforce local pitch alignment. The total contrastive loss is

$$C_\text{output} = f_\text{LALM}(\mathcal{E}, A_\text{input}, I_\text{input})$$2

with tunable coefficients. The model also incorporates a lightweight, encoder-only Singing Voice Transcription (SVT) module that is pre-trained and then frozen during S2A training, supplying frame-level supervision for acoustic-to-melody alignment as well as segment and duration regularization.

The reported results state that CoMelSinger achieves notable improvements in pitch accuracy, timbre consistency, and zero-shot transferability over competitive baselines. F0-RMSE is consistently lower than prior works, high SECS and SMOS scores indicate preserved timbre cloning, and efficient LoRA-based tuning achieves best performance with only approximately 6% of parameters trainable. In the terminology used by the summary, melody tokens act as explicit thought-steps, and the contrastive and SVT regularization make this token sequence a robust reasoning path for synthesis.

6. Limitations, misconceptions, and open problems

The most important limitation is that melody-token reasoning is not yet uniformly realized across the field. Audio-CoT evaluates music reasoning but explicitly reports no melody-token-based intermediate representation; its reasoning chains remain textual (Ma et al., 13 Jan 2025). MusiCoT moves the chain into a latent, tokenized structure space derived from CLAP RVQ rather than into symbolic note sequences (Lam et al., 25 Mar 2025). CoMelSinger uses explicit pitch and duration tokens, but in a singing synthesis framework centered on controllability and disentanglement rather than on general-purpose question answering (Zhao et al., 24 Sep 2025). This suggests that current Melody-CoT work spans multiple technical regimes with different operational meanings of “thought.”

Another recurring issue is failure on difficult tasks. In Audio-CoT, CoT methods improve easy and medium music tasks but can reduce accuracy on hard music reasoning tasks because reasoning chains may confuse the model rather than help. The proposed future directions there are dynamic reasoning strategies, enhanced foundational capabilities, and adaptive, context-aware reasoning mechanisms. The same paper explicitly identifies Melody-Token-Based CoT as an open problem: although not implemented, symbolic melody representations such as note sequences or MIDIs are proposed as potential intermediate reasoning steps. A plausible implication is that future Melody-CoT systems may combine the analyzability of MusiCoT’s structure tokens with the explicit pitch-and-duration control of CoMelSinger, while avoiding the confusion effects observed when long or inappropriate reasoning paths are imposed on models that lack sufficiently deep musical understanding.