MuCPT: Domain-Adapted Music LLM Pretraining

• MuCPT is a domain-adaptive continued pretraining framework for LLMs that specializes in music by integrating a large-scale, high-quality music corpus with token-level quality control.
• The framework employs a reference model-normalized soft scoring mechanism to amplify in-domain signals and dynamically weight tokens, reducing noise during training.
• MuCPT achieves state-of-the-art performance on the MusicSimpleQA benchmark, surpassing larger models by leveraging proprietary, entity-aligned Chinese music data.

MuCPT is a domain-adaptive continued pretraining framework for LLMs targeting the music and music-entertainment domain. It addresses the limitations of general-purpose LLMs in music-specific applications by integrating a high-relevance, large-scale music corpus with a domain-first data pipeline and reference model-based objective for token-level quality control. MuCPT achieves state-of-the-art performance in music factual question answering, supported by the MusicSimpleQA benchmark and rigorous ablation and compositional analyses (Tian et al., 18 Nov 2025).

1. Music Domain Corpus Construction

MuCPT constructs a 40B-token music-related natural language corpus by merging open-source and proprietary data, then applying rigorous filtering, cleaning, and metadata integration to maximize relevance and structural integrity.

Data sources:

• Matrix-music: ≈20B tokens, mined from the 4.5T-token public Matrix corpus. After domain filtering and cleaning, primary sub-sources are Common Crawl (≈17.4B), instruction data (2.7B), academic papers (1.3B), books (0.33B), and wiki (0.27B).
• WeChat-music: ≈20B tokens, curated from billions of WeChat articles and comments. Using the top ~10M song names as seeds, ≈20.5M documents were retrieved, with entity-level alignment across singers, songs, and text fragments.

Domain filtering and weighting relies on a lightweight Qwen2.5-0.5B-based binary classifier trained on 250K positive/negative music examples. Classifier confidence both gates in-domain text and assigns sampling weights at corpus construction time.

Cleaning and de-duplication follows a multi-stage pipeline:

1. Language identification and normalization.
2. Heuristic quality scoring (removal of templated/low-salience content).
3. Near-duplicate elimination via MinHash/locality-sensitive hashing.
4. Privacy-preserving masking of personal identifiers.

Metadata integration in the WeChat-music set yields:

• 3.5M “song–snippet” pairs (1.7M distinct songs),
• ≈0.86M songs with reliable comment-based descriptors (mood, genre, instrumentation),
• Weakly supervised song tags (mood, genre, instrument, BPM) via a “rules + statistics + LLM” pipeline.

This design establishes a corpus of unparalleled scale and purity for the music domain, prioritizing verifiable content and dense entity alignment (Tian et al., 18 Nov 2025).

2. Reference Model-Normalized Continued Pretraining Objective

MuCPT replaces uniform autoregressive next-token prediction with a reference model (RM)-normalized, token-level soft scoring to amplify domain alignment during continued pretraining.

Reference Model (RM) construction involves training a small LM from the same architecture family as the base model on a high-quality, curated music-domain seed set (notably, a WeChat wiki of singers and songs).

For each token $x_t$ in context $x_{<t}$, the RM computes a per-token cross-entropy:

$$\mathrm{CE}_{\mathrm{RM}}(x_t) = -\log p_{\mathrm{RM}}(x_t\,|\,x_{<t}).$$

Unified loss-ratio criterion for a music-domain token:

$$L_{d}(x_t) = -\,\alpha \frac{\log p(x_t \mid x_{<t})} {\log p_{\mathrm{RM}}(x_t \mid x_{<t})}, \quad \alpha>0,$$

while for general-domain tokens, the standard autoregressive loss is used:

$L_{g}(x_t) = -\log p(x_t \mid x_{<t}).$

Functional effects:

• “Soft filtering”: Tokens with high RM loss (out-of-domain or low-quality) are down-weighted in the training objective; tokens that the RM deems high-quality preserve full weight.
• Signal amplification: In-distribution music tokens influence the optimization process more, reducing gradient noise from off-domain content.
• The same criterion is employed for both hard data selection (filtering out tokens above a certain RM loss threshold) and for dynamic loss weighting during training.

This objective cleanly fuses data selection with fine-grained optimization control and allows precise, per-token regulation of learning dynamics (Tian et al., 18 Nov 2025).

3. Model Training Protocol

The MuCPT framework is instantiated atop the Qwen2.5 series autoregressive Transformer architecture, with the Qwen2.5-32B model as its largest public release.

Training configuration highlights:

• Corpus: 2 epochs over the 40B-token music corpus, plus a carefully mixed ratio of general-domain tokens to minimize catastrophic forgetting.
• Optimizer and schedule: Adam with initial learning rate $6\times10^{-5}$, cosine decay to $3\times10^{-5}$, and warmup over the first 0.05% of steps.
• Hardware: 256 H20 GPUs, batch size scaled per hardware to complete 2 full epochs.
• Data sampling: “Domain-first” classifier-based sampling elevates music token prevalence, while WeChat-music is upsampled slightly on tail/new releases to enrich recent content.

Mixture ratios of music to general tokens are tuned per model to optimally balance specialization and generalization (Tian et al., 18 Nov 2025).

4. Evaluation Methodology and Empirical Results

MuCPT introduces the MusicSimpleQA benchmark to quantitatively assess factual music knowledge.

Benchmark features:

• 500 short-form, single-answer factual questions on artists and songs (300 from current artists, 200 from diverse genres/eras).
• Each item has a single, unambiguous, verifiable answer.

Automated agreement scoring uses DeepSeek-v3 as an evaluator to judge the semantic agreement between model responses and reference answers. Accuracy is computed as the fraction of questions where agreement exceeds a set threshold.

Table 1: Leading Model Results on MusicSimpleQA

Models	MusicSimpleQA
GPT-4o	0.6632
DeepSeek-v3	0.7539
Qwen3-235B-A22B-Instruct	0.6719
Qwen2.5-32B-Instruct	0.3599
Qwen2.5-32B-MuCPT	0.7759

MuCPT achieves the highest accuracy (0.7759), surpassing DeepSeek-v3 (+2.2 pts), GPT-4o, and the much larger 235B-parameter Qwen3 model.

Table 2: Ablation on Token-Level Scoring (MusicSimpleQA)

Model / Method	1.5B	7B
NextTokenPrediction	0.4439	0.5499
RHO-1 (hard filter)	0.4759	0.5699
MuCPT (soft scoring)	0.5259	0.6119

RM-based soft scoring outperforms both hard filtering and naïve prediction across all model sizes.

Table 3: Data Recipe Comparison, 1.5B Model, 7B Tokens

Recipe	MusicSimpleQA
IndustryCorpus2 (film)	0.2899
Matrix-music	0.3659
WeChat-music	0.4579

High-relevance, entity-aligned WeChat-music data yields +9.2 pts over open Matrix-music, showing the primacy of targeted domain text.

5. Comparative Analysis and Key Insights

MuCPT’s key empirical advances and methodological contributions are summarized as follows:

• State-of-the-art factual music QA with 32B parameters, surpassing larger, general-purpose and instruction-tuned models as well as GPT-4o.
• Data composition critically impacts performance: proprietary, entity-aligned Chinese music text is substantially more valuable than generic entertainment corpora or open-source domain-filtered data.
• RM-based, token-level loss normalization simultaneously enables robust data selection and training signal amplification, reducing supervision noise and enabling effective alignment without catastrophic forgetting (≤3% drop on general benchmarks such as C-Eval, CMMLU).
• Current limitations include a focus on single-turn, short-answer factual QA; directions for extension include support for complex reasoning, generative tasks (lyrics, melody), and multimodal objectives (text and audio).

A plausible implication is that domain-specialized data pipelines with per-token quality control are necessary to push LLM factuality and retention in specialized fields (Tian et al., 18 Nov 2025).

6. Relations to Symbolic Music Pretraining and Broader Context

While MuCPT focuses on natural language and factual question answering for music, it is complementary to generative symbolic music LLMs such as MuPT (Qu et al., 2024). MuPT demonstrates that LLMs pre-trained on ABC notation, using synchronized multi-track representations and informed by scaling laws tailored to symbolic music, can outperform MIDI-based and general LLM baselines in human preference and structural similarity assessments.

Resources, preprocessing approaches, and evaluation tools from MuPT (e.g., ABC corpus construction, Synchronized Multi-Track ABC Notation, Symbolic Music Scaling Law) provide a foundation for extending MuCPT towards structured musical text generation and integrating symbolic modeling capacities (Qu et al., 2024).

7. Open-Source Commitment and Reproducibility

MuCPT authors commit to scalable, reusable domain LLM construction by providing the MusicSimpleQA benchmark, detailed corpus construction and scoring scripts, as well as public release of model checkpoints and data pipelines. Due to synergy with the MuPT project, MuCPT stands as a template for domain-adaptive LLM development, emphasizing corpus specificity, per-token quality control, and specialized benchmarks for factual assessment (Tian et al., 18 Nov 2025, Qu et al., 2024).