---
title: 'MuQ: Self-Supervised Music Encoder'
url: https://www.emergentmind.com/topics/muq
type: topic
---

# MuQ: Self-Supervised Music Encoder

Searching arXiv for recent papers on MuQ and related models.
MuQ is a self-supervised music representation learning model and, in subsequent literature, a family of pretrained music encoders and derivatives for music understanding, retrieval, evaluation, and recommendation. In its original formulation, MuQ is distinguished by Mel Residual Vector Quantization (Mel-RVQ), which produces stable discrete targets for masked prediction with a Conformer backbone, with the stated goal of capturing both tonal/acoustic details such as melody, pitch, and timbre and high-level semantic attributes such as genre and mood [2501.01108]. Later work extends or reuses this family as MuQ\(_{iter}\), MuQ-310M, and MuQ-MuLan, and applies frozen MuQ representations to piano performance evaluation, perceptual similarity, AI-music quality prediction, and music recommender systems [2601.19029][2601.19109][2603.22677][2604.23077].

## 1. Foundational formulation

The original MuQ paper defines a masked-prediction SSL pipeline for 30 s music audio sampled at 24 kHz, converted to a 128-dim log-Mel spectrum at 25 Hz, encoded by a 12-layer Conformer of approximately 310 M parameters, and trained to predict discrete Mel-RVQ tokens on randomly masked frames, with mask probability \(p=0.6\) [2501.01108]. The high-level recipe is: extract 128-dim Mel spectrogram frames, randomly mask 60% of frames, encode with a 12-layer Conformer, predict \(N=8\) discrete tokens from Mel-RVQ via \(N\) independent linear heads, and minimize the sum of cross-entropy losses across heads.

Mel-RVQ is the defining tokenizer. At each frame, it emits an 8-tuple of token indices from \(N=8\) codebooks of size \(K=1024\). The residual cascade is reported as
\[
\begin{aligned}
r^{(0)} &= x,\\
z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\\
\tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\\
r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}.
\end{aligned}
\]
Its training objective combines codebook, commitment, and reconstruction terms, with \(\beta=0.25\) [2501.01108].

The pretraining regime was reported at two scales. Initial prototyping used Music4all with 0.9 K hours of open-source data, while the final scaled version used an in-house collection of approximately 160 K hours. Mel-RVQ pretraining took less than 1 h on a single GPU for a model of approximately 0.3 M parameters. MuQ pretraining used 32 \(\times\) A100 GPUs, batch size 192, 75 K steps on Music4all, then 200 K steps plus iterative RVQ refinement and 150 K more steps on the 160 K h setup, for a total of approximately two weeks wall-time [2501.01108].

## 2. Model family and reported architectural variants

Published work uses the name MuQ for several closely related pretrained music encoders rather than a single immutable architecture. The original paper describes a 12-layer Conformer with Mel-RVQ masked prediction on log-Mel inputs [2501.01108]. A later layer-wise investigation also describes MuQ as a 12-layer Conformer operating on 44.1 kHz audio converted to 128-band Mel-spectrograms, with hidden dimension \(d=1024\), 16 attention heads, and point-wise feed-forward dimension 4096, trained with a masked-prediction objective in the style of BEST-RQ on approximately 2 million unlabeled commercial music tracks [2505.16306]. MuQ-Eval instead uses “MuQ-310M” or “OpenMuQ/MuQ-large-msd-iter,” a 24-layer Wav2Vec2-Conformer model with 24 kHz raw-waveform input, 50 Hz frame rate, hidden dimension 1024, and self-supervised Mel-RVQ quantization plus reconstruction pretraining on \(\sim 1.6\times 10^5\) h of music [2603.22677]. MuQ-MuLan adds a text tower and contrastive alignment for joint music-text embedding [2501.01108].

| Variant | Reported configuration | Representative use |
|---|---|---|
| MuQ | 12-layer Conformer; Mel-RVQ tokenizer; masked prediction | MIR probing and linear-probe benchmarks |
| MuQ-MuLan | MuQ music tower + text tower; 512-d embeddings | Zero-shot tagging and perceptual similarity |
| MuQ-310M | 24-layer Wav2Vec2-Conformer; raw waveform at 24 kHz | Per-sample quality evaluation |

One additional downstream study describes MuQ as a 12-layer Transformer-style audio encoder pretrained on millions of hours of music audio via a query-based contrastive objective, taking raw waveform at 24 kHz, processing up to 10 s segments, and outputting 1024-dim frame embeddings at a 40 Hz frame rate [2601.19029]. This indicates that the literature uses “MuQ” both for the original Mel-RVQ masked-prediction model and for downstream checkpoints or interfaces whose architectural description is reported somewhat differently. A plausible implication is that “MuQ” should be read as a model family identifier unless a paper specifies the exact checkpoint and feature-extraction protocol.

## 3. Layer-wise organization and representation semantics

The most explicit layer analysis reports a smooth acoustic-to-semantic hierarchy across MuQ’s 12 Conformer layers. In probing experiments over 14 downstream tasks, singer identification is best at layer 1 with \(96.0\%\) accuracy, pitch classification at layer 2 with \(91.5\%\), instrument classification at layer 3 with \(79.7\%\), music tagging at layer 4 with ROC \(=91.5\%\), key detection at layer 5 with \(66.9\%\), genre classification and emotion prediction at layer 6 with \(85.9\%\), \(R^2=76.7\%\), and \(R^2=62.8\%\), singing technique detection and rap detection at layer 7 with \(81.5\%\) and \(94.5\%\), sound quality classification at layer 8 with \(82.0\%\), and structure analysis and music preference classification at layer 9 with \(77.0\%\) and \(78.0\%\). PWCCA similarity to input decreases from layer 1 to 12, and the final layers 10–12 are reported as not optimal for any tested task [2505.16306].

The original benchmark paper is consistent with strong transfer from layer-wise probing: MuQ or MuQ\(_{iter}\) improves over MusicFM and MERT on selected downstream tasks, including genre, music tagging, and instrument classification, while iterative refinement improves key detection and overall MARBLE average score [2501.01108]. The layer-wise study also argues that “layer scanning” often outperforms learned weighted sums, especially when downstream data is limited [2505.16306].

A piano-performance evaluation study demonstrates a different layer selection outcome. It extracts hidden states \(h_9,h_{10},h_{11},h_{12}\in\mathbb{R}^{T\times 1024}\), concatenates them framewise to obtain
\[
z_i=[h_{9,i}\,\|\,h_{10,i}\,\|\,h_{11,i}\,\|\,h_{12,i}] \in \mathbb{R}^{4096},
\]
and then mean-pools over time,
\[
\mathbf{z}=\frac1T\sum_{i=1}^{T}\mathbf{z}_i \in \mathbb{R}^{4096}.
\]
A two-layer MLP with ReLU and dropout(0.3) predicts 19 perceptual dimensions under joint MSE loss, and mean-pooling is reported to outperform attention, LSTM, or max-pooling, with \(R^2=0.405\) for the pooling ablation [2601.19029]. This suggests that layer utility is strongly task-dependent: layers 10–12 were not optimal in the 14-task probing study, whereas layers 9–12 were optimal for perceptual piano evaluation.

## 4. Empirical performance in music understanding and piano performance evaluation

On the MARBLE benchmark, the original MuQ paper reports the following selected results for the scaled models: genre classification on GTZAN reaches \(85.5 \pm 0.9\) for MuQ and \(85.6 \pm 0.7\) for MuQ\(_{iter}\); key detection on GiantSteps refined reaches \(63.5 \pm 0.7\) and \(65.0 \pm 0.8\); music tagging on MagnaTagATune reaches ROC-AUC \(91.4 \pm 0.0\) for both MuQ and MuQ\(_{iter}\), with AP \(40.3 \pm 0.1\) and \(40.1 \pm 0.1\); instrument classification on Nsynth reaches \(79.2 \pm 0.1\) and \(79.7 \pm 0.2\); pitch classification reaches \(92.3 \pm 0.2\) and \(91.3 \pm 0.1\); and the overall MARBLE average score reaches \(76.7\) for MuQ and \(77.0\) for MuQ\(_{iter}\) [2501.01108]. In that study, MuQ outperforms previous self-supervised music representation models with only 0.9 K hours of open-source pretraining data, and scaling to over 160 K hours plus iterative training consistently improves performance [2501.01108].

A controlled comparison between audio and symbolic piano-performance representations provides one of the clearest downstream validations of frozen MuQ features. Using synthesized audio from PercePiano MIDI files rendered via Pianoteq, the study compares symbolic MIDI/HAN against audio foundation models under conditions where both derive from identical source data. The reproduced symbolic baseline reaches \(R^2 = 0.347\) with \(95\%\) CI \([0.315, 0.375]\), MERT-95M reaches \(R^2 = 0.487\) with \(95\%\) CI \([0.460, 0.510]\), MuQ layers 9–12 without augmentation reaches \(R^2 = 0.533\) with \(95\%\) CI \([0.514, 0.560]\), and MuQ with Pianoteq ensemble augmentation over six soundfonts reaches \(R^2 = 0.537\) with \(95\%\) CI \([0.465, 0.575]\), described as a 55% improvement over the symbolic baseline. The study reports paired \(t=-10.71\), \(p=2.08\times10^{-25}\), Wilcoxon signed-rank \(p=2.16\times10^{-29}\), and Cohen’s \(d=0.31\), with audio outperforming symbolic on all 19 perceptual dimensions [2601.19029].

The same study adds three generalization analyses. Cross-soundfont leave-one-out evaluation over six Pianoteq presets yields \(R^2 = 0.534 \pm 0.075\). Difficulty correlation on PSyllabus (\(n=508\)) yields Spearman’s \(\rho=0.623\), with timing \(\rho=0.604\), valence \(\rho=0.604\), timbre\_depth \(\rho=0.580\), and pedal\_amount \(\rho=0.578\). Multi-performer consistency on ASAP over 206 pieces yields mean intra-piece std \(=0.020\), with highest performer sensitivity for dynamic\_range \((0.027)\), timing \((0.022)\), and articulation\_touch \((0.020)\), and lowest for mood\_energy \((0.011)\) and timbre\_brightness \((0.011)\) [2601.19029].

A common assumption is that symbolic and audio features should be complementary. The fusion experiment argues against that assumption in this setting: late fusion of MuQ and MIDI reaches at best \(R^2=0.524\), below audio alone at \(0.533\), and the Pearson correlation of absolute errors is \(r=0.738\), indicating that the two models err on the same difficult segments [2601.19029].

## 5. MuQ-MuLan, perceptual similarity, and recommendation

MuQ-MuLan is the joint music-text extension of MuQ. Its music tower uses pre-trained MuQ, average-pools over time, and applies a linear projection to obtain a 512-dim embedding \(e_m\); the text tower uses XLM-RoBERTa-base plus 8 Transformer layers, average-pools, and linearly projects to a 512-dim embedding \(e_t\). Training uses decoupled contrastive learning on approximately 130 K h of music paired with text descriptions, with 10 s audio clips and batch size 768 on 32 \(\times\) V100. On zero-shot tagging for MagnaTagATune using the prompt “A \<tag\> track,” MuQ-MuLan reaches ROC-AUC \(79.3\) and PR-AUC \(29.3\), reported as new SOTA in that paper [2501.01108].

Out-of-the-box MuQ-MuLan embeddings also support perceptual similarity. Using the frozen MuQ-MuLan audio encoder \(\Phi(\cdot)\in\mathbb{R}^{512}\) on 5 s Inst-Sim-ABX segments, the zero-shot baseline predicts similarity via cosine distance on full-mix embeddings. In the XAB configuration, mix-level accuracy is \(72.4\%\); in the XYC configuration it is \(96.6\%\). For individual instruments in XAB, the reported accuracies are \(79.3\%\) for drums, \(71.6\%\) for bass, \(66.7\%\) for guitar, \(66.3\%\) for piano, and \(65.3\%\) for residuals. The paper then separates each segment into stems with Demucs and learns a zero-intercept ridge-regression weighting over per-stem cosine-similarity differences,
\[
w^* = \arg\min_w \|F w-y\|_2^2 + \lambda\|w\|_2^2,
\]
which induces a weighted retrieval score
\[
S_w(X,V)=\sum_{k=1}^K w_k\, s_k(X,V).
\]
For MuQ-MuLan with 6 Demucs stems plus mix, the learned weights are Bass \(0.32\), Drums \(0.83\), Guitar \(0.61\), Piano \(0.27\), Vocals \(0.33\), Residuals \(1.27\), and Mix \(1.64\). The weighted 6-stem model reaches \(90.4 \pm 2.3\%\) ABX agreement, above the mix-only MuQ-MuLan baseline at \(86.8 \pm 2.7\%\) and the weighted CLAP baseline at \(83.2 \pm 3.3\%\) [2601.19109].

MuQ has also been evaluated as a frozen content backbone for music recommender systems. In that study, MuQ encodes 30-second music snippets into a 1 024-dimensional embedding, obtained by average-pooling final Conformer outputs over time, and these frozen embeddings are used in five downstream approaches: KNN, Shallow Neural Network, Contrastive Multi-Modal Projection, a Hybrid model, and BERT4Rec. For MuQ specifically, raw KNN yields hot-start HR@50 \(=0.0169\), Recall@50 \(=0.0008\), NDCG@50 \(=0.0005\), and cold-start HR@20 \(=0.1726\), Recall@20 \(=0.0487\), NDCG@20 \(=0.0191\). The Shallow Net yields hot HR@50 \(=0.3895 \pm 0.0089\) and cold HR@20 \(=0.5597 \pm 0.0166\). The Bimodal average-projection setup yields hot HR@50 \(=0.3794 \pm 0.0115\) and cold HR@20 \(=0.6517 \pm 0.0263\). The Hybrid model yields hot HR@50 \(=0.4239 \pm 0.0036\) and cold HR@20 \(=0.5396 \pm 0.0320\). BERT4Rec yields hot HR@50 \(=0.4410 \pm 0.0300\) and cold HR@20 \(=0.5117 \pm 0.0892\) [2604.23077]. The same paper reports that MuQ-MuLan slightly exceeds MuQ in best hot-start and cold-start HR, while MusiCNN edges out MuQ in hot-start HR@50.

## 6. Quality prediction, limitations, and nomenclature

MuQ has been adapted into an open-source per-sample quality metric for AI-generated music. MuQ-Eval uses frozen MuQ-310M frame-level features from the last Conformer layer, attention-pools them with a learnable vector \(w\), and passes the pooled representation through a two-layer MLP with hidden size 256 and GELU activation to predict mean opinion score. The simplest system, “A1 Frozen + MSE,” reaches system-level SRCC \(=0.957\) with \(95\%\) CI \([0.898,0.986]\) and utterance-level SRCC \(=0.838 \pm 0.016\); LoRA adaptation, ordinal classification, contrastive auxiliary loss, and uncertainty-weighted multi-task learning do not exceed the pre-registered \(\Delta \ge 0.02\) threshold, and replacing MuQ-310M with MERT-95M drops system-level SRCC to \(0.946\) [2603.22677]. The reported conclusion is that encoder choice dominates architectural and training decisions, and that frozen MuQ representations already capture quality-relevant information.

The same study identifies an important limitation. In controlled degradation analysis on top-quartile clips, MuQ-Eval is highly sensitive to MP3 compression and Gaussian noise, with concordance rates of \(0.97/0.84/0.52\) and \(0.99/0.89/0.39\) for severe/moderate/mild conditions, but near-chance on pitch shift and tempo stretch, with \(0.51/0.43/0.43\) and \(0.58/0.49/0.46\). Overall mean concordance is \(0.63\), below the pre-registered target of \(0.85\) [2603.22677]. This suggests that MuQ-based quality predictors are selectively aligned with signal-level artifacts but not with all musical-structural distortions.

A separate source of confusion is nomenclature. In the uncertainty quantification literature, MUQ refers not to a music encoder but to the MIT Uncertainty Quantification Library, a modular C++/Python framework for Bayesian inverse problems, graphical modeling through WorkGraph and ModPiece abstractions, and advanced MCMC methods including pCN, MALA, sMMALA, H-pCN, H-MALA, DILI, and parallel multilevel MCMC [2112.00713][2107.14552]. That framework includes a Root/Phonebook/Controller/Worker/Collector MPI layout for multilevel MCMC and is integrated with hIPPYlib for PDE-constrained Bayesian inference. The overlap in spelling is accidental; the music-model MuQ and the UQ software MUQ are unrelated research artifacts.

Source: https://www.emergentmind.com/topics/muq