MuChator: Interactive MusicLLM Framework

• MuChator is an interactive MusicLLM framework that uses a three-phase curriculum to integrate music metadata, user feedback, and session logs.
• It employs context-aware instruction tuning with semantic clustering and candidate retrieval to generate ranked song lists based on user queries.
• Hybrid reward reinforcement learning and group-wise policy optimization enhance recommendation accuracy, significantly boosting user engagement.

MuChator is an interactive MusicLLM-based framework deployed at scale in the Douyin Music App, designed to enable active and situational music discovery via conversational interfaces. Unlike conventional feed-driven or keyword-based music recommender systems, MuChator employs a three-phase curriculum for LLM pre-training, followed by context-aware instruction tuning and preference alignment via hybrid reward reinforcement learning. The architecture is centered on the Qwen3-8B backbone and integrates extensive music-domain, behavioral, and platform-specific supervision. The platform demonstrates significant improvements in user engagement metrics and is architected for industrial-scale, low-latency deployment (Liang et al., 26 May 2026).

1. Music Knowledge Pre-training

MuChator employs a three-stage music-centric pre-training pipeline over approximately 23.5 billion tokens. The process continues training of the Qwen3-8B LLM backbone, utilizing transformer networks with segment tokens that mark information such as "songID," "artist," "user_state," and "feedback." The architecture and objective remain standard autoregressive cross-entropy language modeling, with modifications in Stage 3 to introduce item and feedback prediction in a joint loss.

• Stage 1 (Objective Music Knowledge): Trained on 2.7B tokens of music metadata (Douyin catalog), encyclopedic articles, and GPT-derived lyric paraphrases. Data is structured in templated formats such as “Song: <title>. Artist: <artist>. Genre: <genre>. Lyrics: <lyrics>.” The objective is standard next-token modeling:

$L_1(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$

• Stage 2 (Subjective Music Knowledge): Incorporates 4.6B tokens from user comments, professional reviews, playlists, and knowledge graph-style relations (e.g., song-artist-mood). Loss remains standard language modeling:

$L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$

• Stage 3 (Personalized Music Preference): Focuses on 16B tokens of anonymized user session logs (state→item→feedback sequences). The data is structured as:

$\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$

Joint objective:

$$L_3(\theta) = -\sum_t \left[\log p_\theta(i_{t+1} | C_t, s_{t+1}) + \log p_\theta(f_{t+1} | C_t, s_{t+1}, i_{t+1})\right]$$

with $C_t = (\text{profile}, \tau_t)$.

All three stages use a packed token window of 7,168 tokens, a learning rate of $2 \times 10^{-5}$, and are trained for one epoch each with 512 A100 GPUs using mixed-precision (Liang et al., 26 May 2026).

2. Context-aware Instruction Tuning

The context-aware supervised fine-tuning (SFT) phase adapts MuChator to generate a ranked list of 10 song IDs in response to (user context, query) pairs. Given that authentic user–query–item triplets are rare, MuChator uses an automated synthetic data pipeline:

1. Semantic Clustering: Aggregates approximately 1 million (query, song) pairs, utilizing both GPT-generated situational queries and Douyin search logs. These are clustered into around 40,000 intent clusters in embedding space (e.g., “study background,” “walking by the sea”).
2. Candidate Retrieval: For each user, their "favorited" songs are paired with sampled query clusters. All songs within the same cluster form the candidate recommendation pool (50–100 items per pool).
3. Personalization-guided Filtering: Applies a frozen, multi-objective dot-product ranker to score candidates. The top 10 items form the ground-truth recommendation list.

Supervised objective:

$$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(C,q,y) \sim D_{\text{SFT}}} \sum_{t=1}^{T} \log p_\theta(y_t | y_{<t}, C, q)$$

Hyperparameters for SFT include batch size 256, learning rate $2 \times 10^{-5}$, two epochs, maximum decode length of 10 items, and a 7,168-token context window. Example prompt construction is fully specified (incorporating user profiles, session history, and free-text queries) (Liang et al., 26 May 2026).

3. Preference Alignment Through Hybrid Reward Reinforcement Learning

Post-SFT, MuChator applies reinforcement learning to align recommendations with both query intent and fine-grained personalization, while enforcing practical constraints through a hybrid reward model ($R_{\text{hyb}}$). The reward comprises both model-based and rule-based components:

• Model-based:
  • $R_{\text{rel}} \in \{0, 1\}$: Semantic relevance (scored by a finetuned LLM “judge,” thresholded at 0.5)
  • $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$0: Satisfaction score from Douyin’s online ranking model
• Rule-based:
  • $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$1: Validity of parseable output
  • $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$2: Existence of item in platform catalog
  • $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$3: Diversity penalty for repeated artists/titles in the list
  • $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$4: Penalty for recent item repetition

Weighted hybrid reward:

$L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$5

with empirically tuned weights ($L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$60.5, $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$70.05, $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$80.15, $L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$90.15, $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$00.15).

Group-wise Policy Optimization (GRPO) is performed over groups of $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$1 sampled lists per user–query context. The clipped surrogate loss is:

$\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$2

where $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$3 is the policy ratio, $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$4 is the advantage over in-group baseline $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$5, $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$6, $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$7, $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$8. RL training runs for one epoch with batch size 1024, rollout $\tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t)$9, and learning rate $$L_3(\theta) = -\sum_t \left[\log p_\theta(i_{t+1} | C_t, s_{t+1}) + \log p_\theta(f_{t+1} | C_t, s_{t+1}, i_{t+1})\right]$$0 on 512 GPUs. The held-out RL dataset size is approximately 200,000 (user, query, session) tuples (Liang et al., 26 May 2026).

4. Inference Pipeline and System Deployment

MuChator is deployed as a production API serving Douyin Music, with the inference stack optimized for industry-scale, low-latency chat interaction. The system integrates Qwen3-8B and post-SFT/RL weights. The end-to-end inference pipeline proceeds as follows:

1. User Request: Initiated via the Douyin App through HTTP gRPC, traversing an API gateway and authentication layer.
2. LLM-based Recommendation: The MuChator service consumes the static user profile, recent session history, and current natural language query, generating a 10-item songID list in a single pass.
3. Post-processing: Maps songIDs to metadata (e.g., title, cover), enforces deduplication, and prefetches streaming URLs/snippets.
4. Client Rendering: The chat interface renders interactive controls, and audio streaming requests are initiated as needed.

The system achieves an end-to-first-token latency of 0.09 s and produces the complete 10-song list within 1.87 s on a p3.8xlarge server. Pre-training, SFT, and RL phases are all executed with 512 × A100 GPUs (80GB), requiring several hours for full corpus completion (Liang et al., 26 May 2026).

5. Data Scales, Evaluation, and Engagement Outcomes

Phase	Data Volume	Notable Statistics
Pre-training (Stage 1–3)	23.5B tokens	Stage1=2.7B, Stage2=4.6B, Stage3=16B
SFT (UQ2I synthetic)	~400K user-query-item10 examples	~40K intent clusters, ~1M (query, song) seeds
RL	~200K sessions	GRPO with G=8, batch size 1024
Online Deployment	0.1B daily MAU	Up to +46.49% Active Days (A/B test results)

MuChator demonstrated a 46.49% increase in “user active days,” 77.36% increase in session duration, and 11.26% uplift in click-through rate versus baseline (internal search) in an online A/B test on Douyin Music (Liang et al., 26 May 2026).

6. Architectural Significance and Implications

MuChator establishes a modular architecture in which standard LLM pre-training and instruction tuning are deeply aligned to domain-specific knowledge and active user intent. Its pipeline addresses prevalent LLM recommendation limitations: insufficient music knowledge representation, shallow preference modeling, and inability to resolve ambiguous, colloquial, or situational user queries. The system illustrates the impact of large-scale, curriculum-based data curation and reinforcement learning with structured, hybrid rewards in maximizing both item relevance and user satisfaction within industrial-scale interactive music recommendation.

A plausible implication is that the MuChator approach, specifically its multi-stage knowledge injection and groupwise RL paradigm, generalizes to other domains requiring synthesis of rich domain-specific facts, subjective context, and real-world user feedback. Ongoing evaluation, and comparative baselines such as Gemini-3-Pro, situate MuChator at state-of-the-art in conversational music discovery (Liang et al., 26 May 2026).