. Artist: <artist>. Genre: <genre>. Lyrics: <lyrics>.” The objective is standard next-token modeling: $$L_1(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$$ - **Stage 2 (Subjective Music Knowledge):** Incorporates 4.6B tokens from user comments, professional reviews, playlists, and knowledge graph-style relations (e.g., song-artist-mood). Loss remains standard language modeling: $$L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$$ - **Stage 3 (Personalized Music Preference):** Focuses on 16B tokens of anonymized user session logs (state→item→feedback sequences). The data is structured as: $$ \tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t) $$ Joint objective: $$ L_3(\theta) = -\sum_t \left[\log p_\theta(i_{t+1} | C_t, s_{t+1}) + \log p_\theta(f_{t+1} | C_t, s_{t+1}, i_{t+1})\right] $$ with $C_t = (\text{profile}, \tau_t)$. All three stages use a packed token window of 7,168 tokens, a learning rate of $2 \times 10^{-5}$, and are trained for one epoch each with 512 A100 GPUs using mixed-precision [2605.27103]. ## 2. Context-aware Instruction Tuning The context-aware supervised fine-tuning (SFT) phase adapts MuChator to generate a ranked list of 10 song IDs in response to (user context, query) pairs. Given that authentic user–query–item triplets are rare, MuChator uses an automated synthetic data pipeline: 1. **Semantic Clustering:** Aggregates approximately 1 million (query, song) pairs, utilizing both GPT-generated situational queries and Douyin search logs. These are clustered into around 40,000 intent clusters in embedding space (e.g., “study background,” “walking by the sea”). 2. **Candidate Retrieval:** For each user, their "favorited" songs are paired with sampled query clusters. All songs within the same cluster form the candidate recommendation pool (50–100 items per pool). 3. **Personalization-guided Filtering:** Applies a frozen, multi-objective dot-product ranker to score candidates. The top 10 items form the ground-truth recommendation list. Supervised objective: $$ L_{\text{SFT}}(\theta) = -\mathbb{E}_{(C,q,y) \sim D_{\text{SFT}}} \sum_{t=1}^{T} \log p_\theta(y_t | y_{<t}, C, q) $$ Hyperparameters for SFT include batch size 256, learning rate $2 \times 10^{-5}$, two epochs, maximum decode length of 10 items, and a 7,168-token context window. Example prompt construction is fully specified (incorporating user profiles, session history, and free-text queries) [2605.27103]. ## 3. Preference Alignment Through Hybrid Reward Reinforcement Learning Post-SFT, MuChator applies reinforcement learning to align recommendations with both query intent and fine-grained personalization, while enforcing practical constraints through a hybrid reward model ($R_{\text{hyb}}$). The reward comprises both model-based and rule-based components: - **Model-based:** - $R_{\text{rel}} \in \{0, 1\}$: Semantic relevance (scored by a finetuned LLM “judge,” thresholded at 0.5) - $R_{\text{pers}} \in \mathbb{R}$: Satisfaction score from Douyin’s online ranking model - **Rule-based:** - $R_{\text{format}}$: Validity of parseable output - $R_{\text{fact}}$: Existence of item in platform catalog - $R_{\text{div}}$: Diversity penalty for repeated artists/titles in the list - $R_{\text{dedup}}$: Penalty for recent item repetition Weighted hybrid reward: $$ R_{\text{hyb}} = R_{\text{rel}} \times \lambda_{\text{pers}} R_{\text{pers}} + \sum_{r\in \text{Rules}} \lambda_r R_r $$ with empirically tuned weights ($\lambda_{\text{pers}}{=}$0.5, $\lambda_{\text{format}}{=}$0.05, $\lambda_{\text{fact}}{=}$0.15, $\lambda_{\text{div}}{=}$0.15, $\lambda_{\text{dedup}}{=}$0.15). Group-wise Policy Optimization (GRPO) is performed over groups of $G=8$ sampled lists per user–query context. The clipped surrogate loss is: $$ L_{\text{GRPO}}(\theta) = -\mathbb{E}_{(C,q)} \left[\frac{1}{G} \sum_{i=1}^G \min ( r_i A_i, \text{clip}(r_i, 1-\epsilon, 1+\epsilon) A_i )\right] + \beta D_{KL}(p_{\text{ref}} \| p_\theta) $$ where $r_i$ is the policy ratio, $A_i$ is the advantage over in-group baseline $b$, $G=8$, $\epsilon=0.2$, $\beta=0.01$. RL training runs for one epoch with batch size 1024, rollout $G=8$, and learning rate $1 \times 10^{-5}$ on 512 GPUs. The held-out RL dataset size is approximately 200,000 (user, query, session) tuples [2605.27103]. ## 4. Inference Pipeline and System Deployment MuChator is deployed as a production API serving Douyin Music, with the inference stack optimized for industry-scale, low-latency chat interaction. The system integrates Qwen3-8B and post-SFT/RL weights. The end-to-end inference pipeline proceeds as follows: 1. **User Request:** Initiated via the Douyin App through HTTP gRPC, traversing an API gateway and authentication layer. 2. **LLM-based Recommendation:** The MuChator service consumes the static user profile, recent session history, and current natural language query, generating a 10-item songID list in a single pass. 3. **Post-processing:** Maps songIDs to metadata (e.g., title, cover), enforces deduplication, and prefetches streaming URLs/snippets. 4. **Client Rendering:** The chat interface renders interactive controls, and audio streaming requests are initiated as needed. The system achieves an end-to-first-token latency of 0.09 s and produces the complete 10-song list within 1.87 s on a p3.8xlarge server. Pre-training, SFT, and RL phases are all executed with 512 × A100 GPUs (80GB), requiring several hours for full corpus completion [2605.27103]. ## 5. Data Scales, Evaluation, and Engagement Outcomes | Phase | Data Volume | Notable Statistics | |----------------------------|----------------------------------|------------------------------------------------| | Pre-training (Stage 1–3) | 23.5B tokens | Stage1=2.7B, Stage2=4.6B, Stage3=16B | | SFT (UQ2I synthetic) | ~400K user-query-item10 examples | ~40K intent clusters, ~1M (query, song) seeds | | RL | ~200K sessions | GRPO with G=8, batch size 1024 | | Online Deployment | 0.1B daily MAU | Up to +46.49% Active Days (A/B test results) | MuChator demonstrated a 46.49% increase in “user active days,” 77.36% increase in session duration, and 11.26% uplift in click-through rate versus baseline (internal search) in an online A/B test on Douyin Music [2605.27103]. ## 6. Architectural Significance and Implications MuChator establishes a modular architecture in which standard LLM pre-training and instruction tuning are deeply aligned to domain-specific knowledge and active user intent. Its pipeline addresses prevalent LLM recommendation limitations: insufficient music knowledge representation, shallow preference modeling, and inability to resolve ambiguous, colloquial, or situational user queries. The system illustrates the impact of large-scale, curriculum-based data curation and reinforcement learning with structured, hybrid rewards in maximizing both item relevance and user satisfaction within industrial-scale interactive music recommendation. A plausible implication is that the MuChator approach, specifically its multi-stage knowledge injection and groupwise RL paradigm, generalizes to other domains requiring synthesis of rich domain-specific facts, subjective context, and real-world user feedback. Ongoing evaluation, and comparative baselines such as Gemini-3-Pro, situate MuChator at state-of-the-art in conversational music discovery [2605.27103]. Source: https://www.emergentmind.com/topics/muchator-architecture

--- title: 'MuChator: Interactive MusicLLM Framework' url: https://www.emergentmind.com/topics/muchator-architecture type: topic --- # MuChator: Interactive MusicLLM Framework MuChator is an interactive MusicLLM-based framework deployed at scale in the Douyin Music App, designed to enable active and situational music discovery via conversational interfaces. Unlike conventional feed-driven or keyword-based music recommender systems, MuChator employs a three-phase curriculum for large language model (LLM) pre-training, followed by context-aware instruction tuning and preference alignment via hybrid reward reinforcement learning. The architecture is centered on the Qwen3-8B backbone and integrates extensive music-domain, behavioral, and platform-specific supervision. The platform demonstrates significant improvements in user engagement metrics and is architected for industrial-scale, low-latency deployment [2605.27103]. ## 1. Music Knowledge Pre-training MuChator employs a three-stage music-centric pre-training pipeline over approximately 23.5 billion tokens. The process continues training of the Qwen3-8B LLM backbone, utilizing transformer networks with segment tokens that mark information such as "songID," "artist," "user_state," and "feedback." The architecture and objective remain standard autoregressive cross-entropy language modeling, with modifications in Stage 3 to introduce item and feedback prediction in a joint loss. - **Stage 1 (Objective Music Knowledge):** Trained on 2.7B tokens of music metadata (Douyin catalog), encyclopedic articles, and GPT-derived lyric paraphrases. Data is structured in templated formats such as “Song: . Artist: <artist>. Genre: <genre>. Lyrics: <lyrics>.” The objective is standard next-token modeling: $$L_1(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$$ - **Stage 2 (Subjective Music Knowledge):** Incorporates 4.6B tokens from user comments, professional reviews, playlists, and knowledge graph-style relations (e.g., song-artist-mood). Loss remains standard language modeling: $$L_2(\theta) = -\sum_t \log p_\theta(w_t | w_{<t})$$ - **Stage 3 (Personalized Music Preference):** Focuses on 16B tokens of anonymized user session logs (state→item→feedback sequences). The data is structured as: $$ \tau_t = (s_1, i_1, f_1, \ldots, s_t, i_t, f_t) $$ Joint objective: $$ L_3(\theta) = -\sum_t \left[\log p_\theta(i_{t+1} | C_t, s_{t+1}) + \log p_\theta(f_{t+1} | C_t, s_{t+1}, i_{t+1})\right] $$ with $C_t = (\text{profile}, \tau_t)$. All three stages use a packed token window of 7,168 tokens, a learning rate of $2 \times 10^{-5}$, and are trained for one epoch each with 512 A100 GPUs using mixed-precision [2605.27103]. ## 2. Context-aware Instruction Tuning The context-aware supervised fine-tuning (SFT) phase adapts MuChator to generate a ranked list of 10 song IDs in response to (user context, query) pairs. Given that authentic user–query–item triplets are rare, MuChator uses an automated synthetic data pipeline: 1. **Semantic Clustering:** Aggregates approximately 1 million (query, song) pairs, utilizing both GPT-generated situational queries and Douyin search logs. These are clustered into around 40,000 intent clusters in embedding space (e.g., “study background,” “walking by the sea”). 2. **Candidate Retrieval:** For each user, their "favorited" songs are paired with sampled query clusters. All songs within the same cluster form the candidate recommendation pool (50–100 items per pool). 3. **Personalization-guided Filtering:** Applies a frozen, multi-objective dot-product ranker to score candidates. The top 10 items form the ground-truth recommendation list. Supervised objective: $$ L_{\text{SFT}}(\theta) = -\mathbb{E}_{(C,q,y) \sim D_{\text{SFT}}} \sum_{t=1}^{T} \log p_\theta(y_t | y_{<t}, C, q) $$ Hyperparameters for SFT include batch size 256, learning rate $2 \times 10^{-5}$, two epochs, maximum decode length of 10 items, and a 7,168-token context window. Example prompt construction is fully specified (incorporating user profiles, session history, and free-text queries) [2605.27103]. ## 3. Preference Alignment Through Hybrid Reward Reinforcement Learning Post-SFT, MuChator applies reinforcement learning to align recommendations with both query intent and fine-grained personalization, while enforcing practical constraints through a hybrid reward model ($R_{\text{hyb}}$). The reward comprises both model-based and rule-based components: - **Model-based:** - $R_{\text{rel}} \in \{0, 1\}$: Semantic relevance (scored by a finetuned LLM “judge,” thresholded at 0.5) - $R_{\text{pers}} \in \mathbb{R}$: Satisfaction score from Douyin’s online ranking model - **Rule-based:** - $R_{\text{format}}$: Validity of parseable output - $R_{\text{fact}}$: Existence of item in platform catalog - $R_{\text{div}}$: Diversity penalty for repeated artists/titles in the list - $R_{\text{dedup}}$: Penalty for recent item repetition Weighted hybrid reward: $$ R_{\text{hyb}} = R_{\text{rel}} \times \lambda_{\text{pers}} R_{\text{pers}} + \sum_{r\in \text{Rules}} \lambda_r R_r $$ with empirically tuned weights ($\lambda_{\text{pers}}{=}$0.5, $\lambda_{\text{format}}{=}$0.05, $\lambda_{\text{fact}}{=}$0.15, $\lambda_{\text{div}}{=}$0.15, $\lambda_{\text{dedup}}{=}$0.15). Group-wise Policy Optimization (GRPO) is performed over groups of $G=8$ sampled lists per user–query context. The clipped surrogate loss is: $$ L_{\text{GRPO}}(\theta) = -\mathbb{E}_{(C,q)} \left[\frac{1}{G} \sum_{i=1}^G \min ( r_i A_i, \text{clip}(r_i, 1-\epsilon, 1+\epsilon) A_i )\right] + \beta D_{KL}(p_{\text{ref}} \| p_\theta) $$ where $r_i$ is the policy ratio, $A_i$ is the advantage over in-group baseline $b$, $G=8$, $\epsilon=0.2$, $\beta=0.01$. RL training runs for one epoch with batch size 1024, rollout $G=8$, and learning rate $1 \times 10^{-5}$ on 512 GPUs. The held-out RL dataset size is approximately 200,000 (user, query, session) tuples [2605.27103]. ## 4. Inference Pipeline and System Deployment MuChator is deployed as a production API serving Douyin Music, with the inference stack optimized for industry-scale, low-latency chat interaction. The system integrates Qwen3-8B and post-SFT/RL weights. The end-to-end inference pipeline proceeds as follows: 1. **User Request:** Initiated via the Douyin App through HTTP gRPC, traversing an API gateway and authentication layer. 2. **LLM-based Recommendation:** The MuChator service consumes the static user profile, recent session history, and current natural language query, generating a 10-item songID list in a single pass. 3. **Post-processing:** Maps songIDs to metadata (e.g., title, cover), enforces deduplication, and prefetches streaming URLs/snippets. 4. **Client Rendering:** The chat interface renders interactive controls, and audio streaming requests are initiated as needed. The system achieves an end-to-first-token latency of 0.09 s and produces the complete 10-song list within 1.87 s on a p3.8xlarge server. Pre-training, SFT, and RL phases are all executed with 512 × A100 GPUs (80GB), requiring several hours for full corpus completion [2605.27103]. ## 5. Data Scales, Evaluation, and Engagement Outcomes | Phase | Data Volume | Notable Statistics | |----------------------------|----------------------------------|------------------------------------------------| | Pre-training (Stage 1–3) | 23.5B tokens | Stage1=2.7B, Stage2=4.6B, Stage3=16B | | SFT (UQ2I synthetic) | ~400K user-query-item10 examples | ~40K intent clusters, ~1M (query, song) seeds | | RL | ~200K sessions | GRPO with G=8, batch size 1024 | | Online Deployment | 0.1B daily MAU | Up to +46.49% Active Days (A/B test results) | MuChator demonstrated a 46.49% increase in “user active days,” 77.36% increase in session duration, and 11.26% uplift in click-through rate versus baseline (internal search) in an online A/B test on Douyin Music [2605.27103]. ## 6. Architectural Significance and Implications MuChator establishes a modular architecture in which standard LLM pre-training and instruction tuning are deeply aligned to domain-specific knowledge and active user intent. Its pipeline addresses prevalent LLM recommendation limitations: insufficient music knowledge representation, shallow preference modeling, and inability to resolve ambiguous, colloquial, or situational user queries. The system illustrates the impact of large-scale, curriculum-based data curation and reinforcement learning with structured, hybrid rewards in maximizing both item relevance and user satisfaction within industrial-scale interactive music recommendation. A plausible implication is that the MuChator approach, specifically its multi-stage knowledge injection and groupwise RL paradigm, generalizes to other domains requiring synthesis of rich domain-specific facts, subjective context, and real-world user feedback. Ongoing evaluation, and comparative baselines such as Gemini-3-Pro, situate MuChator at state-of-the-art in conversational music discovery [2605.27103]. Source: https://www.emergentmind.com/topics/muchator-architecture