MuQ-MuLan: Joint Music–Text Embedding

• MuQ-MuLan is a joint music–text embedding model that integrates self-supervised audio encoding with contrastive learning for efficient zero-shot music tagging and retrieval.
• It employs a dual-tower architecture with a 12-layer Conformer audio encoder using Mel-RVQ and an xlm-roberta-based text encoder to align high-dimensional audio with natural language descriptors.
• The system demonstrates state-of-the-art performance, achieving significant ROC-AUC gains and high perceptual alignment with human judgments on large-scale music datasets.

MuQ-MuLan denotes a joint music–text embedding model that integrates a self-supervised music representation learning architecture, MuQ, with a contrastively trained multi-modal alignment module. The resulting system establishes a state-of-the-art baseline for zero-shot music tagging and music–text retrieval. MuQ-MuLan is constructed around recent advances in discrete audio tokenization and large-scale, two-tower contrastive representation learning, enabling alignment of high-dimensional music features with natural language descriptors. It achieves strong perceptual alignment and competitive performance on music semantics and retrieval tasks across a variety of corpora (Zhu et al., 2 Jan 2025, Vohra et al., 27 Jan 2026, Deng et al., 16 Oct 2025).

1. Model Architecture and Tokenization

MuQ-MuLan's architecture consists of two principal towers for audio and text. The audio tower employs the MuQ model, which is a 12-layer Conformer encoder trained using masked signal modeling. The input audio is transformed into 128-dimensional Mel-spectrogram frames (sampled at 25 Hz from 24 kHz audio) and subjected to stochastic masking (60% of frames), after which the encoder predicts masked Mel-spectrogram tokens (Zhu et al., 2 Jan 2025). These tokens are produced using Mel Residual Vector Quantization (Mel-RVQ), which quantizes each frame through a series of $N=8$ residual codebooks (each with size $K=1024$), with independent linear projections for encoding and decoding at each stage. The pre-trained MuQ encoder outputs are average-pooled across time and projected into a 512-dimensional embedding space.

The text tower leverages an xlm-roberta-base LLM, with an additional stack of 8 Transformer layers. The output tokens are likewise average-pooled and projected into a 512-dimensional embedding space (Zhu et al., 2 Jan 2025). Prior to similarity scoring, both towers apply $L_2$ normalization to their representations (Vohra et al., 27 Jan 2026).

2. Contrastive Training and Loss Function

MuQ-MuLan is trained using a contrastive objective to jointly align audio and text embeddings. For a batch of $B$ paired music and text samples $(e_m^{(i)}, e_t^{(i)})_{i=1}^B$, the decoupled contrastive loss (DCL) or InfoNCE is used: $$L = -\frac{1}{B}\sum_{i=1}^B \log \frac{\exp(\mathrm{sim}(e_m^{(i)}, e_t^{(i)}) / \tau)}{\sum_{j=1}^B \exp(\mathrm{sim}(e_m^{(i)}, e_t^{(j)}) / \tau)}$$ where $\mathrm{sim}(\cdot, \cdot)$ denotes the dot product of $L_2$-normalized embeddings and $\tau$ is a temperature hyperparameter tuned on validation splits. The same loss is applied both in the audio-to-text and text-to-audio directions, and the final loss is their average (Zhu et al., 2 Jan 2025, Vohra et al., 27 Jan 2026).

3. Data, Training Protocol, and Ablation Insights

MuQ-MuLan is trained on a large-scale corpus: 130,000 hours of in-house music audio, each paired with human-written text descriptions (genre, mood, instrument, tempo, and open-ended keywords). This corpus allows fine-grained grounding of musical audio in naturalistic, user-provided semantic descriptions.

The model is optimized using standard settings for large-scale transformers, including AdamW and large batch sizes (e.g., 768). Audio segments of 10 seconds are used per sample for audio tower training to support extensive batch-wise negative sampling (Zhu et al., 2 Jan 2025).

Ablative comparisons demonstrate that incorporating MuQ as the audio encoder in place of MERT-MuLan or MusicFM-MuLan yields a +3.8 and +3.1 point increase in ROC-AUC, respectively. Training Mel-RVQ projections (as opposed to using random projections) yields downstream improvements of 2–3 percentage points in tagging tasks. Use of 8 residual quantization codebooks in Mel-RVQ is beneficial relative to alternatives, and additional Mel-RVQ depth confers no further gain (Zhu et al., 2 Jan 2025).

4. Zero-Shot Evaluation and Perceptual Alignment

MuQ-MuLan attains state-of-the-art results in zero-shot music tagging. On the MagnaTagATune dataset (top-50 tags), it achieves an ROC-AUC of 79.3 and PR-AUC of 29.3, outperforming MuLan (ROC-AUC 78.2 with nearly three times more pretraining data) and prior models such as LAION-CLAP and Microsoft-CLAP (Zhu et al., 2 Jan 2025). The contrastive methodology supports efficient retrieval across modalities without explicit fine-tuning for individual tasks.

In perceptual similarity tasks, MuQ-MuLan's cosine similarity between audio fragments is highly correlated with human judgments. On the Inst-Sim-ABX perceptual benchmark, MuQ-MuLan obtains a 90.4% agreement rate in the most challenging 6-stem instrument-weighted scenario, improving over the "mix-only" baseline and showing robustness across different instrument separations (Vohra et al., 27 Jan 2026). Ridge regression analysis on stem-level features reveals interpretable, instrument-wise weights that correlate with human perceptual weighting (e.g., drums and mix stems often exert greater influence on perceived similarity).

5. Use in Perceptual and Semantic Retrieval

The pretrained MuQ-MuLan embedding space supports several practical regimes:

• Zero-shot retrieval: Audio–text and audio–audio search can be performed without further adaptation, with strong agreement to perceptual similarity or musical tag ground truth (Vohra et al., 27 Jan 2026).
• Instrument-weighted similarity: When source stems are available (e.g., via Demucs), a small linear regressor trained on human ABX preference data enables customizable, interpretable similarity queries (e.g., stem-level loop or sample search).
• Fine-tuning: For domain adaptation or improved task-specific performance, continued contrastive training or audio-only contrastive learning with margin-based losses can be applied (Vohra et al., 27 Jan 2026).

Normalization and temperature regularization are critical for robust retrieval; hyperparameters are typically tuned on a limited preference-judgment validation set.

6. Limitations, Timbre Semantics, and Comparative Analysis

While MuQ-MuLan excels in music tagging and perceptual similarity, its ability to encode or align with nuanced, human-perceived timbre semantics is limited compared to models trained on broader web-sourced audio–text pairs such as LAION-CLAP (Deng et al., 16 Oct 2025). Quantitative analysis shows that MuQ-MuLan achieves positive Pearson correlation with human adjective ratings for fewer than half of the tested timbre descriptors, and often exhibits negative or unstructured trends in response to controlled DSP manipulations (e.g., reverb or EQ sweeps). This suggests domain specialization restricts its semantic breadth in capturing subjective audio attributes. Fine-tuning or explicit probing for timbral axes is recommended to bridge this gap.

7. Summary Table: Core Properties and Performance Metrics

Component	Architectural Detail	Characteristic Value / Result
Audio Encoder	12-layer Conformer; Mel-RVQ masking/pretrain	310M parameters
Mel-RVQ Tokenizer	$N=8$, $K=1024$ per stage, single linear proj.	Stable, 2–3% gain over baselines
Text Encoder	xlm-roberta-base + 8 Transformer layers	≈320M parameters
Embedding Dim	512 (per modality, $L_2$-normed prior to scoring)	–
Contrastive Loss	DCL/InfoNCE (dot product, temp. τ≈0.07)	Achieves batch-aligned learning
Pretraining Data	130K h music+text	SOTA at moderate scale
Zero-shot ROC-AUC	MagnaTagATune (50 tags): 79.3	SOTA (prior ≈78.2, 73.9, 75.5)
ABX Perceptual Align	Weighted 6-stem: 90.4% agreement	Exceeds prior baselines

MuQ-MuLan exemplifies the integration of highly specialized self-supervised audio encoding with large-scale cross-modal contrastive learning, achieving leading performance in music understanding and retrieval. The architecture’s inductive bias toward music structure, combined with data-efficient design, substantiates its utility for both research and production applications in music information retrieval, automated tagging, and perceptually aligned music search (Zhu et al., 2 Jan 2025, Vohra et al., 27 Jan 2026, Deng et al., 16 Oct 2025).