MuQ: Self-Supervised Music Encoder

• MuQ is a self-supervised music representation model employing masked prediction with Mel-RVQ tokenization and a Conformer backbone to capture both acoustic details and semantic attributes.
• Its modular design spawns variants like MuQ-310M and MuQ-MuLan, which are tailored for tasks such as music evaluation, retrieval, and zero-shot tagging.
• Empirical analyses demonstrate that iterative refinement and layer-wise probing of MuQ enhance performance across diverse downstream music understanding and recommendation tasks.

Searching arXiv for papers on MuQ and related models. MuQ is a self-supervised music representation learning model and, in subsequent literature, a family of pretrained music encoders and derivatives for music understanding, retrieval, evaluation, and recommendation. In its original formulation, MuQ is distinguished by Mel Residual Vector Quantization (Mel-RVQ), which produces stable discrete targets for masked prediction with a Conformer backbone, with the stated goal of capturing both tonal/acoustic details such as melody, pitch, and timbre and high-level semantic attributes such as genre and mood (Zhu et al., 2 Jan 2025). Later work extends or reuses this family as MuQ$_{iter}$, MuQ-310M, and MuQ-MuLan, and applies frozen MuQ representations to piano performance evaluation, perceptual similarity, AI-music quality prediction, and music recommender systems (Dhiman, 26 Jan 2026, Vohra et al., 27 Jan 2026, Zhu et al., 24 Mar 2026, Tamm et al., 25 Apr 2026).

1. Foundational formulation

The original MuQ paper defines a masked-prediction SSL pipeline for 30 s music audio sampled at 24 kHz, converted to a 128-dim log-Mel spectrum at 25 Hz, encoded by a 12-layer Conformer of approximately 310 M parameters, and trained to predict discrete Mel-RVQ tokens on randomly masked frames, with mask probability $p=0.6$ (Zhu et al., 2 Jan 2025). The high-level recipe is: extract 128-dim Mel spectrogram frames, randomly mask 60% of frames, encode with a 12-layer Conformer, predict $N=8$ discrete tokens from Mel-RVQ via $N$ independent linear heads, and minimize the sum of cross-entropy losses across heads.

Mel-RVQ is the defining tokenizer. At each frame, it emits an 8-tuple of token indices from $N=8$ codebooks of size $K=1024$. The residual cascade is reported as

$$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$

Its training objective combines codebook, commitment, and reconstruction terms, with $\beta=0.25$ (Zhu et al., 2 Jan 2025).

The pretraining regime was reported at two scales. Initial prototyping used Music4all with 0.9 K hours of open-source data, while the final scaled version used an in-house collection of approximately 160 K hours. Mel-RVQ pretraining took less than 1 h on a single GPU for a model of approximately 0.3 M parameters. MuQ pretraining used 32 $\times$ A100 GPUs, batch size 192, 75 K steps on Music4all, then 200 K steps plus iterative RVQ refinement and 150 K more steps on the 160 K h setup, for a total of approximately two weeks wall-time (Zhu et al., 2 Jan 2025).

2. Model family and reported architectural variants

Published work uses the name MuQ for several closely related pretrained music encoders rather than a single immutable architecture. The original paper describes a 12-layer Conformer with Mel-RVQ masked prediction on log-Mel inputs (Zhu et al., 2 Jan 2025). A later layer-wise investigation also describes MuQ as a 12-layer Conformer operating on 44.1 kHz audio converted to 128-band Mel-spectrograms, with hidden dimension $d=1024$, 16 attention heads, and point-wise feed-forward dimension 4096, trained with a masked-prediction objective in the style of BEST-RQ on approximately 2 million unlabeled commercial music tracks (Zhou et al., 22 May 2025). MuQ-Eval instead uses “MuQ-310M” or “OpenMuQ/MuQ-large-msd-iter,” a 24-layer Wav2Vec2-Conformer model with 24 kHz raw-waveform input, 50 Hz frame rate, hidden dimension 1024, and self-supervised Mel-RVQ quantization plus reconstruction pretraining on $p=0.6$0 h of music (Zhu et al., 24 Mar 2026). MuQ-MuLan adds a text tower and contrastive alignment for joint music-text embedding (Zhu et al., 2 Jan 2025).

Variant	Reported configuration	Representative use
MuQ	12-layer Conformer; Mel-RVQ tokenizer; masked prediction	MIR probing and linear-probe benchmarks
MuQ-MuLan	MuQ music tower + text tower; 512-d embeddings	Zero-shot tagging and perceptual similarity
MuQ-310M	24-layer Wav2Vec2-Conformer; raw waveform at 24 kHz	Per-sample quality evaluation

One additional downstream study describes MuQ as a 12-layer Transformer-style audio encoder pretrained on millions of hours of music audio via a query-based contrastive objective, taking raw waveform at 24 kHz, processing up to 10 s segments, and outputting 1024-dim frame embeddings at a 40 Hz frame rate (Dhiman, 26 Jan 2026). This indicates that the literature uses “MuQ” both for the original Mel-RVQ masked-prediction model and for downstream checkpoints or interfaces whose architectural description is reported somewhat differently. A plausible implication is that “MuQ” should be read as a model family identifier unless a paper specifies the exact checkpoint and feature-extraction protocol.

3. Layer-wise organization and representation semantics

The most explicit layer analysis reports a smooth acoustic-to-semantic hierarchy across MuQ’s 12 Conformer layers. In probing experiments over 14 downstream tasks, singer identification is best at layer 1 with $p=0.6$1 accuracy, pitch classification at layer 2 with $p=0.6$2, instrument classification at layer 3 with $p=0.6$3, music tagging at layer 4 with ROC $p=0.6$4, key detection at layer 5 with $p=0.6$5, genre classification and emotion prediction at layer 6 with $p=0.6$6, $p=0.6$7, and $p=0.6$8, singing technique detection and rap detection at layer 7 with $p=0.6$9 and $N=8$0, sound quality classification at layer 8 with $N=8$1, and structure analysis and music preference classification at layer 9 with $N=8$2 and $N=8$3. PWCCA similarity to input decreases from layer 1 to 12, and the final layers 10–12 are reported as not optimal for any tested task (Zhou et al., 22 May 2025).

The original benchmark paper is consistent with strong transfer from layer-wise probing: MuQ or MuQ$N=8$4 improves over MusicFM and MERT on selected downstream tasks, including genre, music tagging, and instrument classification, while iterative refinement improves key detection and overall MARBLE average score (Zhu et al., 2 Jan 2025). The layer-wise study also argues that “layer scanning” often outperforms learned weighted sums, especially when downstream data is limited (Zhou et al., 22 May 2025).

A piano-performance evaluation study demonstrates a different layer selection outcome. It extracts hidden states $N=8$5, concatenates them framewise to obtain

$N=8$6

and then mean-pools over time,

$N=8$7

A two-layer MLP with ReLU and dropout(0.3) predicts 19 perceptual dimensions under joint MSE loss, and mean-pooling is reported to outperform attention, LSTM, or max-pooling, with $N=8$8 for the pooling ablation (Dhiman, 26 Jan 2026). This suggests that layer utility is strongly task-dependent: layers 10–12 were not optimal in the 14-task probing study, whereas layers 9–12 were optimal for perceptual piano evaluation.

4. Empirical performance in music understanding and piano performance evaluation

On the MARBLE benchmark, the original MuQ paper reports the following selected results for the scaled models: genre classification on GTZAN reaches $N=8$9 for MuQ and $N$0 for MuQ$N$1; key detection on GiantSteps refined reaches $N$2 and $N$3; music tagging on MagnaTagATune reaches ROC-AUC $N$4 for both MuQ and MuQ$N$5, with AP $N$6 and $N$7; instrument classification on Nsynth reaches $N$8 and $N$9; pitch classification reaches $N=8$0 and $N=8$1; and the overall MARBLE average score reaches $N=8$2 for MuQ and $N=8$3 for MuQ$N=8$4 (Zhu et al., 2 Jan 2025). In that study, MuQ outperforms previous self-supervised music representation models with only 0.9 K hours of open-source pretraining data, and scaling to over 160 K hours plus iterative training consistently improves performance (Zhu et al., 2 Jan 2025).

A controlled comparison between audio and symbolic piano-performance representations provides one of the clearest downstream validations of frozen MuQ features. Using synthesized audio from PercePiano MIDI files rendered via Pianoteq, the study compares symbolic MIDI/HAN against audio foundation models under conditions where both derive from identical source data. The reproduced symbolic baseline reaches $N=8$5 with $N=8$6 CI $N=8$7, MERT-95M reaches $N=8$8 with $N=8$9 CI $K=1024$0, MuQ layers 9–12 without augmentation reaches $K=1024$1 with $K=1024$2 CI $K=1024$3, and MuQ with Pianoteq ensemble augmentation over six soundfonts reaches $K=1024$4 with $K=1024$5 CI $K=1024$6, described as a 55% improvement over the symbolic baseline. The study reports paired $K=1024$7, $K=1024$8, Wilcoxon signed-rank $K=1024$9, and Cohen’s $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$0, with audio outperforming symbolic on all 19 perceptual dimensions (Dhiman, 26 Jan 2026).

The same study adds three generalization analyses. Cross-soundfont leave-one-out evaluation over six Pianoteq presets yields $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$1. Difficulty correlation on PSyllabus ($$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$2) yields Spearman’s $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$3, with timing $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$4, valence $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$5, timbre_depth $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$6, and pedal_amount $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$7. Multi-performer consistency on ASAP over 206 pieces yields mean intra-piece std $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$8, with highest performer sensitivity for dynamic_range $$\begin{aligned} r^{(0)} &= x,\ z^{(n)} &= M_P^{(n)}\,r^{(n-1)},\ \tau^{(n)}&=\arg\min_{1\le i\le K}\,\big\|\,\mathrm{norm}(z^{(n)})-\mathrm{norm}\big(Q^{(n)}_{i}\big)\big\|_2^2,\ r^{(n)} &= r^{(n-1)} \;-\; M_D^{(n)}\,Q^{(n)}_{\tau^{(n)}}. \end{aligned}$$9, timing $\beta=0.25$0, and articulation_touch $\beta=0.25$1, and lowest for mood_energy $\beta=0.25$2 and timbre_brightness $\beta=0.25$3 (Dhiman, 26 Jan 2026).

A common assumption is that symbolic and audio features should be complementary. The fusion experiment argues against that assumption in this setting: late fusion of MuQ and MIDI reaches at best $\beta=0.25$4, below audio alone at $\beta=0.25$5, and the Pearson correlation of absolute errors is $\beta=0.25$6, indicating that the two models err on the same difficult segments (Dhiman, 26 Jan 2026).

5. MuQ-MuLan, perceptual similarity, and recommendation

MuQ-MuLan is the joint music-text extension of MuQ. Its music tower uses pre-trained MuQ, average-pools over time, and applies a linear projection to obtain a 512-dim embedding $\beta=0.25$7; the text tower uses XLM-RoBERTa-base plus 8 Transformer layers, average-pools, and linearly projects to a 512-dim embedding $\beta=0.25$8. Training uses decoupled contrastive learning on approximately 130 K h of music paired with text descriptions, with 10 s audio clips and batch size 768 on 32 $\beta=0.25$9 V100. On zero-shot tagging for MagnaTagATune using the prompt “A <tag> track,” MuQ-MuLan reaches ROC-AUC $\times$0 and PR-AUC $\times$1, reported as new SOTA in that paper (Zhu et al., 2 Jan 2025).

Out-of-the-box MuQ-MuLan embeddings also support perceptual similarity. Using the frozen MuQ-MuLan audio encoder $\times$2 on 5 s Inst-Sim-ABX segments, the zero-shot baseline predicts similarity via cosine distance on full-mix embeddings. In the XAB configuration, mix-level accuracy is $\times$3; in the XYC configuration it is $\times$4. For individual instruments in XAB, the reported accuracies are $\times$5 for drums, $\times$6 for bass, $\times$7 for guitar, $\times$8 for piano, and $\times$9 for residuals. The paper then separates each segment into stems with Demucs and learns a zero-intercept ridge-regression weighting over per-stem cosine-similarity differences,

$d=1024$0

which induces a weighted retrieval score

$d=1024$1

For MuQ-MuLan with 6 Demucs stems plus mix, the learned weights are Bass $d=1024$2, Drums $d=1024$3, Guitar $d=1024$4, Piano $d=1024$5, Vocals $d=1024$6, Residuals $d=1024$7, and Mix $d=1024$8. The weighted 6-stem model reaches $d=1024$9 ABX agreement, above the mix-only MuQ-MuLan baseline at $p=0.6$00 and the weighted CLAP baseline at $p=0.6$01 (Vohra et al., 27 Jan 2026).

MuQ has also been evaluated as a frozen content backbone for music recommender systems. In that study, MuQ encodes 30-second music snippets into a 1 024-dimensional embedding, obtained by average-pooling final Conformer outputs over time, and these frozen embeddings are used in five downstream approaches: KNN, Shallow Neural Network, Contrastive Multi-Modal Projection, a Hybrid model, and BERT4Rec. For MuQ specifically, raw KNN yields hot-start HR@50 $p=0.6$02, Recall@50 $p=0.6$03, NDCG@50 $p=0.6$04, and cold-start HR@20 $p=0.6$05, Recall@20 $p=0.6$06, NDCG@20 $p=0.6$07. The Shallow Net yields hot HR@50 $p=0.6$08 and cold HR@20 $p=0.6$09. The Bimodal average-projection setup yields hot HR@50 $p=0.6$10 and cold HR@20 $p=0.6$11. The Hybrid model yields hot HR@50 $p=0.6$12 and cold HR@20 $p=0.6$13. BERT4Rec yields hot HR@50 $p=0.6$14 and cold HR@20 $p=0.6$15 (Tamm et al., 25 Apr 2026). The same paper reports that MuQ-MuLan slightly exceeds MuQ in best hot-start and cold-start HR, while MusiCNN edges out MuQ in hot-start HR@50.

6. Quality prediction, limitations, and nomenclature

MuQ has been adapted into an open-source per-sample quality metric for AI-generated music. MuQ-Eval uses frozen MuQ-310M frame-level features from the last Conformer layer, attention-pools them with a learnable vector $p=0.6$16, and passes the pooled representation through a two-layer MLP with hidden size 256 and GELU activation to predict mean opinion score. The simplest system, “A1 Frozen + MSE,” reaches system-level SRCC $p=0.6$17 with $p=0.6$18 CI $p=0.6$19 and utterance-level SRCC $p=0.6$20; LoRA adaptation, ordinal classification, contrastive auxiliary loss, and uncertainty-weighted multi-task learning do not exceed the pre-registered $p=0.6$21 threshold, and replacing MuQ-310M with MERT-95M drops system-level SRCC to $p=0.6$22 (Zhu et al., 24 Mar 2026). The reported conclusion is that encoder choice dominates architectural and training decisions, and that frozen MuQ representations already capture quality-relevant information.

The same study identifies an important limitation. In controlled degradation analysis on top-quartile clips, MuQ-Eval is highly sensitive to MP3 compression and Gaussian noise, with concordance rates of $p=0.6$23 and $p=0.6$24 for severe/moderate/mild conditions, but near-chance on pitch shift and tempo stretch, with $p=0.6$25 and $p=0.6$26. Overall mean concordance is $p=0.6$27, below the pre-registered target of $p=0.6$28 (Zhu et al., 24 Mar 2026). This suggests that MuQ-based quality predictors are selectively aligned with signal-level artifacts but not with all musical-structural distortions.

A separate source of confusion is nomenclature. In the uncertainty quantification literature, MUQ refers not to a music encoder but to the MIT Uncertainty Quantification Library, a modular C++/Python framework for Bayesian inverse problems, graphical modeling through WorkGraph and ModPiece abstractions, and advanced MCMC methods including pCN, MALA, sMMALA, H-pCN, H-MALA, DILI, and parallel multilevel MCMC (Kim et al., 2021, Seelinger et al., 2021). That framework includes a Root/Phonebook/Controller/Worker/Collector MPI layout for multilevel MCMC and is integrated with hIPPYlib for PDE-constrained Bayesian inference. The overlap in spelling is accidental; the music-model MuQ and the UQ software MUQ are unrelated research artifacts.