---
title: Three-Stage Music Knowledge Pre-training
url: https://www.emergentmind.com/topics/three-stage-music-knowledge-pre-training
type: topic
---

# Three-Stage Music Knowledge Pre-training

Three-stage Music Knowledge Pre-training is a curriculum-based large language model (LLM) pre-training paradigm designed to inculcate robust, multifaceted music-domain capabilities in transformer-based architectures. Developed and evaluated in the MuChator framework, this scheme incrementally exposes the LLM to progressively complex music-related data distributions—objective music knowledge, subjective user-contextual knowledge, and personalized user trajectory data—to support active music discovery through natural language interaction. Empirical ablations demonstrate significant improvements at each stage in task-relevant benchmarks such as domain-specific factual knowledge, query-to-item musical relevance, and user-personalization perplexity [2605.27103].

## 1. Pre-training Curriculum Overview

Three-stage Music Knowledge Pre-training is structured as an “easy-to-hard” knowledge injection process, where each successive stage builds upon previously acquired competencies:

1. **Stage 1: Objective Music Knowledge:** Factual grounding in fundamental music information, including song and artist metadata, lyrical content, and contextual domain facts.
2. **Stage 2: Subjective Music Knowledge:** Exposure to community-generated content such as user comments and playlists, modeling subjective associations and collaborative patterns.
3. **Stage 3: Personalized Music Preferences:** Sequence modeling over anonymized user context histories, enabling tailored, situation-aware music recommendations.

This curriculum enables a monolithic Qwen3-8B transformer to develop from generic LLM capabilities into a music-specialized expert, showing cumulative, monotonic gains in essential music discovery and recommendation tasks.

## 2. Stage 1: Objective Music Knowledge

### Goals and Rationale

Stage 1 is intended to "ground" the model in the foundational vocabulary and semantics of the music domain. This includes factual associations between songs, artists, albums, lyrics, genres, and background knowledge.

### Data Sources

- **Music Metadata (~1B tokens):** Structured fields such as song titles, artist names, lyrics, genres, and album identifiers.
- **Online Resources (~1.2B tokens):** Web-scraped encyclopedia entries, artist biographies, and cultural articles from Douyin’s music platform.
- **Lyric Descriptions (~0.5B tokens):** LLM-generated paraphrases and explanations of lyrics.

### Modeling and Loss

A standard causal language modeling objective is used:
\[
\mathcal{L}_{\text{obj}} = -\sum_{t=1}^N \log p_\theta(x_t | x_{<t})
\]
where \( x_{1:N} \) are music-domain token sequences.

### Architecture and Modifications

- The base transformer (Qwen3-8B) is architecturally unchanged.
- The tokenizer vocabulary is extended to encompass music-specific entities.
- No adapters, additional output heads, or special modules are introduced.

### Impact and Gains

After Stage 1, compared with vanilla Qwen3:

- Music Knowledge (multiple-choice accuracy): **74.6%** (+22.1%)
- Q2I Relevance: **84.6%** (+12.2%)
- U2I Personalization (perplexity): **8.43** (–19.8%)
- Mild loss in general knowledge (76.4%).

*This suggests that focused domain grounding can substantially improve music domain performance, albeit at slight expense to general capabilities*.

## 3. Stage 2: Subjective Music Knowledge

### Goals and Rationale

Stage 2 targets the acquisition of "subjective" music reasoning—mapping between users' colloquial or situational descriptions and musical items via exposure to real-world listening behaviors and opinions.

### Data Sources

- **Music Comments (~2B tokens):** User-generated comments and professional reviews.
- **Music Playlists (~0.8B tokens):** Playlists reflecting co-listening and item co-occurrence.
- **Music Relations (~1.8B tokens):** Collaborative filtering outputs and a music knowledge graph encoding song/artist/tag associations.

### Modeling and Loss

Continued causal LM optimized over subjective data:
\[
\mathcal{L}_{\text{subj}} = -\sum_{t=1}^M \log p_\theta(y_t | y_{<t})
\]
with \( y_{1:M} \) including comments, playlist sequences, and graph-serialized relations.

### Architecture and Modifications

- No architectural changes; input sequences reflect subjective relationships using a flat text serialization.
- No explicit contrastive or relational losses are applied.

### Impact and Gains

After Stage 2:

- Music Knowledge: **80.4%** (+31.6%)
- Q2I Relevance: **91.2%** (+21.0%)
- U2I Personalization: **8.12** (–22.7%)
- General Knowledge: 75.3% (–2.3%)

*The cumulative objective-to-subjective pre-training sequence enhances reasoning about musical relevance within less strictly factual, interactional discourse*.

## 4. Stage 3: Personalized Music Preferences

### Goals and Rationale

Stage 3 equips the model with the ability to model, anticipate, and recommend in alignment with individual user histories—a pre-requisite for adaptive, context-aware music recommendation.

### Data Sources

- **User Context (~16B tokens):** Sequences structured as \(\tau_t = (\text{profile}; s_1,i_1,f_1, ..., s_t,i_t,f_t)\), encoding user profile, situational state (e.g., time, weather, mood), played item, and user feedback labels (like/skip/dislike).

### Modeling and Loss

Joint modeling of next-item and feedback given cumulative context:
\[
\mathcal{L}_{\text{pref}} = -\sum_t \big[ \log p_\theta(i_{t+1} | \mathcal{C}_t, s_{t+1}) + \log p_\theta(f_{t+1} | \mathcal{C}_t, s_{t+1}, i_{t+1}) \big]
\]
Both outputs use a shared output projection layer.

### Architecture and Modifications

- The transformer stack remains unchanged.
- Vocabulary is expanded to include item and feedback tokens.
- Sequence structure is exploited through position embeddings.

### Impact and Gains

After Stage 3:

- Music Knowledge: **82.4%** (+34.9%)
- Q2I Relevance: **93.2%** (+23.6%)
- U2I Personalization: **3.64** (–65.4%)
- General Knowledge: 74.3% (–3.6%)

*A plausible implication is that explicit serialized user context enables the model to anticipate future listening and feedback with high fidelity, yielding strong personalization metrics.*

## 5. Comparative Performance across Pre-training Stages

| Stage      | Music Knowledge (%) | Q2I Relevance (%) | U2I Perplexity | General Knowledge (%) |
|------------|--------------------|-------------------|----------------|----------------------|
| Vanilla    | 61.1               | 75.4              | 10.51          | 77.1                 |
| Stage 1    | 74.6               | 84.6              | 8.43           | 76.4                 |
| Stage 2    | 80.4               | 91.2              | 8.12           | 75.3                 |
| Stage 3    | 82.4               | 93.2              | 3.64           | 74.3                 |

The table summarizes cumulative improvements in each domain-relevant metric as the pre-training proceeds. The monotonic increase across stages for all music-domain focused tasks validates the staged, curriculum-inspired approach in the context of large-scale interactive recommendation [2605.27103].

## 6. Significance and Mechanisms of Curriculum Design

Each stage in Three-stage Music Knowledge Pre-training targets distinct but complementary competencies:

- **Stage 1 (Anchoring):** Embeds core music facts and terminology.
- **Stage 2 (Subjectivity):** Supports alignment between naturalistic user descriptions and musical content via exposure to real-world opinions and associations.
- **Stage 3 (Personalization):** Internalizes prototypical user behavior sequences, conferring adaptive, context-sensitive recommendation ability.

This incremental structure is motivated by the need to scaffold learning: grounding factual concepts precedes subjective reasoning, which itself precedes reliable personalization over user trajectory data. The approach capitalizes on token-level flexibility and the inherent sequence modeling strength of transformer-based LLMs without requiring bespoke architectural modifications at any stage.

## 7. Implications for Interactive Music LLMs

The three-stage scheme underpins the MuChator framework’s ability to facilitate active music discovery via conversational interfaces on Douyin Music. By demonstrating monotonically increasing accuracy in music knowledge, query-to-item relevance, and user-to-item personalization—with a moderate tradeoff in general knowledge—this curriculum emerged as a critical enabler for next-generation music recommender systems employing LLMs in industrial contexts [2605.27103]. The ablation evidence confirms the curriculum’s essential role in advancing situational, colloquial, and personalized interaction paradigms for large-scale music platforms.

Source: https://www.emergentmind.com/topics/three-stage-music-knowledge-pre-training