Three-Stage Music Knowledge Pre-training

• The paper introduces a curriculum-based pre-training paradigm that incrementally builds music domain expertise from factual grounding to personalized user preferences.
• Objective Music Knowledge leverages large-scale metadata and lyric data, significantly boosting accuracy in factual and relevance benchmarks.
• Personalized Music Preferences stage utilizes extensive user context and sequential modeling to markedly improve recommendation and personalization performance.

Three-stage Music Knowledge Pre-training is a curriculum-based LLM pre-training paradigm designed to inculcate robust, multifaceted music-domain capabilities in transformer-based architectures. Developed and evaluated in the MuChator framework, this scheme incrementally exposes the LLM to progressively complex music-related data distributions—objective music knowledge, subjective user-contextual knowledge, and personalized user trajectory data—to support active music discovery through natural language interaction. Empirical ablations demonstrate significant improvements at each stage in task-relevant benchmarks such as domain-specific factual knowledge, query-to-item musical relevance, and user-personalization perplexity (Liang et al., 26 May 2026).

1. Pre-training Curriculum Overview

Three-stage Music Knowledge Pre-training is structured as an “easy-to-hard” knowledge injection process, where each successive stage builds upon previously acquired competencies:

1. Stage 1: Objective Music Knowledge: Factual grounding in fundamental music information, including song and artist metadata, lyrical content, and contextual domain facts.
2. Stage 2: Subjective Music Knowledge: Exposure to community-generated content such as user comments and playlists, modeling subjective associations and collaborative patterns.
3. Stage 3: Personalized Music Preferences: Sequence modeling over anonymized user context histories, enabling tailored, situation-aware music recommendations.

This curriculum enables a monolithic Qwen3-8B transformer to develop from generic LLM capabilities into a music-specialized expert, showing cumulative, monotonic gains in essential music discovery and recommendation tasks.

2. Stage 1: Objective Music Knowledge

Goals and Rationale

Stage 1 is intended to "ground" the model in the foundational vocabulary and semantics of the music domain. This includes factual associations between songs, artists, albums, lyrics, genres, and background knowledge.

Data Sources

• Music Metadata (~1B tokens): Structured fields such as song titles, artist names, lyrics, genres, and album identifiers.
• Online Resources (~1.2B tokens): Web-scraped encyclopedia entries, artist biographies, and cultural articles from Douyin’s music platform.
• Lyric Descriptions (~0.5B tokens): LLM-generated paraphrases and explanations of lyrics.

Modeling and Loss

A standard causal language modeling objective is used: $$\mathcal{L}_{\text{obj}} = -\sum_{t=1}^N \log p_\theta(x_t | x_{<t})$$ where $x_{1:N}$ are music-domain token sequences.

Architecture and Modifications

• The base transformer (Qwen3-8B) is architecturally unchanged.
• The tokenizer vocabulary is extended to encompass music-specific entities.
• No adapters, additional output heads, or special modules are introduced.

Impact and Gains

After Stage 1, compared with vanilla Qwen3:

• Music Knowledge (multiple-choice accuracy): 74.6% (+22.1%)
• Q2I Relevance: 84.6% (+12.2%)
• U2I Personalization (perplexity): 8.43 (–19.8%)
• Mild loss in general knowledge (76.4%).

This suggests that focused domain grounding can substantially improve music domain performance, albeit at slight expense to general capabilities.

3. Stage 2: Subjective Music Knowledge

Goals and Rationale

Stage 2 targets the acquisition of "subjective" music reasoning—mapping between users' colloquial or situational descriptions and musical items via exposure to real-world listening behaviors and opinions.

Data Sources

• Music Comments (~2B tokens): User-generated comments and professional reviews.
• Music Playlists (~0.8B tokens): Playlists reflecting co-listening and item co-occurrence.
• Music Relations (~1.8B tokens): Collaborative filtering outputs and a music knowledge graph encoding song/artist/tag associations.

Modeling and Loss

Continued causal LM optimized over subjective data: $$\mathcal{L}_{\text{subj}} = -\sum_{t=1}^M \log p_\theta(y_t | y_{<t})$$ with $y_{1:M}$ including comments, playlist sequences, and graph-serialized relations.

Architecture and Modifications

• No architectural changes; input sequences reflect subjective relationships using a flat text serialization.
• No explicit contrastive or relational losses are applied.

Impact and Gains

After Stage 2:

• Music Knowledge: 80.4% (+31.6%)
• Q2I Relevance: 91.2% (+21.0%)
• U2I Personalization: 8.12 (–22.7%)
• General Knowledge: 75.3% (–2.3%)

The cumulative objective-to-subjective pre-training sequence enhances reasoning about musical relevance within less strictly factual, interactional discourse.

4. Stage 3: Personalized Music Preferences

Goals and Rationale

Stage 3 equips the model with the ability to model, anticipate, and recommend in alignment with individual user histories—a pre-requisite for adaptive, context-aware music recommendation.

Data Sources

• User Context (~16B tokens): Sequences structured as $$\tau_t = (\text{profile}; s_1,i_1,f_1, ..., s_t,i_t,f_t)$$, encoding user profile, situational state (e.g., time, weather, mood), played item, and user feedback labels (like/skip/dislike).

Modeling and Loss

Joint modeling of next-item and feedback given cumulative context: $$\mathcal{L}_{\text{pref}} = -\sum_t \big[ \log p_\theta(i_{t+1} | \mathcal{C}_t, s_{t+1}) + \log p_\theta(f_{t+1} | \mathcal{C}_t, s_{t+1}, i_{t+1}) \big]$$ Both outputs use a shared output projection layer.

Architecture and Modifications

• The transformer stack remains unchanged.
• Vocabulary is expanded to include item and feedback tokens.
• Sequence structure is exploited through position embeddings.

Impact and Gains

After Stage 3:

• Music Knowledge: 82.4% (+34.9%)
• Q2I Relevance: 93.2% (+23.6%)
• U2I Personalization: 3.64 (–65.4%)
• General Knowledge: 74.3% (–3.6%)

A plausible implication is that explicit serialized user context enables the model to anticipate future listening and feedback with high fidelity, yielding strong personalization metrics.

5. Comparative Performance across Pre-training Stages

Stage	Music Knowledge (%)	Q2I Relevance (%)	U2I Perplexity	General Knowledge (%)
Vanilla	61.1	75.4	10.51	77.1
Stage 1	74.6	84.6	8.43	76.4
Stage 2	80.4	91.2	8.12	75.3
Stage 3	82.4	93.2	3.64	74.3

The table summarizes cumulative improvements in each domain-relevant metric as the pre-training proceeds. The monotonic increase across stages for all music-domain focused tasks validates the staged, curriculum-inspired approach in the context of large-scale interactive recommendation (Liang et al., 26 May 2026).

6. Significance and Mechanisms of Curriculum Design

Each stage in Three-stage Music Knowledge Pre-training targets distinct but complementary competencies:

• Stage 1 (Anchoring): Embeds core music facts and terminology.
• Stage 2 (Subjectivity): Supports alignment between naturalistic user descriptions and musical content via exposure to real-world opinions and associations.
• Stage 3 (Personalization): Internalizes prototypical user behavior sequences, conferring adaptive, context-sensitive recommendation ability.

This incremental structure is motivated by the need to scaffold learning: grounding factual concepts precedes subjective reasoning, which itself precedes reliable personalization over user trajectory data. The approach capitalizes on token-level flexibility and the inherent sequence modeling strength of transformer-based LLMs without requiring bespoke architectural modifications at any stage.

7. Implications for Interactive Music LLMs

The three-stage scheme underpins the MuChator framework’s ability to facilitate active music discovery via conversational interfaces on Douyin Music. By demonstrating monotonically increasing accuracy in music knowledge, query-to-item relevance, and user-to-item personalization—with a moderate tradeoff in general knowledge—this curriculum emerged as a critical enabler for next-generation music recommender systems employing LLMs in industrial contexts (Liang et al., 26 May 2026). The ablation evidence confirms the curriculum’s essential role in advancing situational, colloquial, and personalized interaction paradigms for large-scale music platforms.