Adopting State-of-the-Art Pretrained Audio Representations for Music Recommender Systems

Abstract: Over the years, Music Information Retrieval (MIR) research community has released various models pretrained on large amounts of music data. Transfer learning showcases the proven effectiveness of pretrained backend models for a broad spectrum of downstream tasks, including auto-tagging and genre classification. However, MIR papers generally do not explore the efficiency of pretrained models for Music Recommender Systems (MRS). In addition, the Recommender Systems community tends to favour traditional end-to-end neural network training. Our research addresses this gap and evaluates the performance of nine pretrained backend models (MusicFM, Music2Vec, MERT, EncodecMAE, Jukebox, MusiCNN, MULE, MuQ and MuQ-MuLan) in the context of MRS. We assess them using five recommendation approaches: K-Nearest Neighbours (KNN), Shallow Neural Network, Contrastive Multi-Modal projection, a Hybrid model, and BERT4Rec both for the hot and cold-start scenarios. Our findings suggest that pretrained audio representations exhibit significant performance disparity between traditional MIR tasks and both hot and cold music recommendations, indicating that valuable aspects of musical information captured by backend models may differ depending on the task. This study establishes a foundation for further exploration of pretrained audio representations to enhance music recommendation systems.

• The paper compares nine frozen pretrained audio representations across various recommendation models, measuring their effectiveness in both hot and cold-start settings.
• The paper shows that supervised models like MusiCNN excel in hot recommendations, while generative and self-supervised models such as Jukebox and MusicFM are superior in cold-start scenarios.
• The paper highlights the need for task-specific evaluation of MIR embeddings, underscoring that performance in music recommendation systems diverges from traditional MIR benchmarks.

Comparative Evaluation of Pretrained Audio Representations in Music Recommender Systems

Introduction

This paper presents a comprehensive empirical analysis of state-of-the-art pretrained audio representations (PARs) applied to Music Recommender Systems (MRS). While Music Information Retrieval (MIR) research has produced numerous self-supervised and supervised audio embedding models, the majority of MRS research relies on traditional end-to-end training, typically integrating conventional audio features as inputs. The study evaluates nine pretrained backend models—including MusicFM, Music2Vec, MERT, EncodecMAE, Jukebox, MusiCNN, MULE, MuQ, and MuQ-MuLan—within five distinct recommendation models (KNN, shallow neural network, bimodal contrastive projection, a hybrid model, and BERT4Rec), across both hot and cold-start scenarios. The goal is to rigorously assess performance disparities and inform the selection of PARs for future hybrid music recommendation pipelines.

Evolution of Audio Representations in MRS

Historically, MRS systems adopted handcrafted low-level signal descriptors such as MFCCs or chroma vectors. From 2013 onward, the field shifted toward CNNs trained on Mel-spectrograms, driven by evidence of improved efficacy in capturing salient musical attributes for recommendation. Spotify-derived feature sets (e.g., danceability, acousticness) gained temporary popularity but faced reproducibility and coverage issues following Spotify API deprecation. More recently, the MIR community has proposed transfer learning approaches leveraging large-scale pretrained models, but their adoption in core MRS tasks has been limited.

Figure 1: Annual distribution of MRS papers employing different audio data representations, illustrating a transition from low-level handcrafted features toward spectrogram- and deep learning-based approaches.

Although MIR advances have greatly benefited tasks like genre classification or auto-tagging, the potential of PARs for real-world recommendation has not been systematically investigated. Only a minority of studies have directly evaluated frozen backend audio models for MRS, leaving a methodological gap this paper aims to address.

Dataset and Experimental Design

The experiments utilize the Music4All-Onion dataset, which provides rich audio previews and user listening histories across a variety of mainstream genres. A temporal train/validate/test split ensures realistic hot and cold-start evaluation: new items in user profiles form the hot test set, while genuinely novel tracks released in the final month constitute the cold test set. The study evaluates each recommendation model with frozen PARs as item representations, measuring performance with HitRate, Recall, and NDCG at standard cutoffs.

Figure 2: Genre distribution in the Music4All dataset, confirming broad genre coverage and similarity to training data distributions of most backend embedding models.

The nine evaluated PAR methods span a broad methodological spectrum:

• MFCC: Classic, hand-engineered baseline.
• MusiCNN: Supervised CNN trained for auto-tagging.
• Jukebox: Large VQ-VAE/Transformer for generative modeling.
• Music2Vec: Masked audio prediction via CNN and Transformers.
• EncodecMAE: Neural audio codec with masked autoencoding.
• MERT: Large-scale masked language modeling for acoustic/music features.
• MusicFM: Conformer encoder with random projection quantization.
• MuQ & MuQ-MuLan: Self-supervised, Conformer-based, with (MuLan) additional text-contrastive objective.
• MULE: Comparative model incorporating supervised/unsupervised pretraining.

Each representation is transformed into a fixed-length descriptor via temporal averaging, corresponding to the paradigm of “frozen” embeddings throughout the recommendation models.

Main Findings

Relative Performance of PARs Across Tasks

Hot-Start Recommendation

For hot recommendations, more expressive recommendation architectures (e.g., BERT4Rec, Hybrid) consistently outperform simpler ones (e.g., KNN, shallow net), regardless of the PAR used.

• MusiCNN, a supervised auto-tagging model, yields the highest HitRate@50 (0.4722) with BERT4Rec, a result not matched by more recent, self-supervised, or generative models.
• MuQ and MuQ-MuLan demonstrate robust performance across all recommenders, suggesting that their self-supervised, token-based approach captures features beneficial for matching content with collaborative signals.
• Unexpectedly, Jukebox, although powerful for audio generation and MIR tasks, performs poorly in hot-item recommendation, potentially due to representational dimensionality mismatches.
• All modern PARs outperform the MFCC baseline in the hot-start scenario, confirming the superiority of deep audio embeddings over traditional handcrafted features.

Cold-Start Recommendation

In the cold-start setting, where collaborative data is inherently limited, the ranking of PARs changes noticeably:

• Jukebox and MusicFM emerge as top performers, especially with the Bimodal and Hybrid recommenders, highlighting the value of content-centric information when little user history exists.
• MuQ-based models maintain high performance, but the advantage of other PARs such as MusiCNN diminishes.
• Bimodal contrastively-trained representations show a significant gain in the cold start, with PAR selection driving substantial differences in model effectiveness.

Comparison with MIR Task Performance

A detailed comparison with backend models’ published results on MIR benchmarks (auto-tagging, genre, key detection) reveals:

• There exist substantial discrepancies between which PARs excel at MRS versus MIR tasks. For example, MusiCNN leads for hot recommendations but is not state-of-the-art in general MIR.
• Models with the best MIR metrics (e.g., MusicFM in auto-tagging/key detection) do not necessarily lead in hot-item recommendation, although they are competitive in cold-start scenarios.
• The transferability of pretrained audio features to recommendation varies by downstream task, demanding careful selection and empirical validation of PARs for MRS.

Practical and Theoretical Implications

This work establishes that the performance hierarchy of pretrained audio embeddings for recommendation diverges from their established ranking on MIR tasks. It demonstrates the critical importance of task-specific evaluation when selecting or deploying foundation models as audio backends in hybrid MRS systems. In practical terms, content-based recommendations—especially for new or rare items—benefit substantially from state-of-the-art representations, while in hot-item recommendation, architectural choice may be as important as the PAR selected.

The study also highlights the ongoing necessity of hybrid (multi-modal) recommendation pipelines that can dynamically leverage both collaborative and content-derived signals, with model variants such as bimodal contrastive learning demonstrating clear advantages in cold-start contexts. Continued research may focus on improved combinatorial and fusion techniques, the reinterpretation of PARs for cold/warm transfer learning, and the incorporation of textual and metadata signals alongside audio features for maximal system flexibility and interpretability.

Future Directions

• Advanced Fusion Strategies: Exploring alternatives to frozen embedding initialization, such as fine-tuned adapters or parameter-efficient transfer approaches, could further close the performance gap between content-based and collaborative models.
• Beyond-Accuracy Metrics: Investigating diversity, novelty, and explainability in hybrid MRS utilizing PARs.
• Expanded Benchmarking: Broadening evaluation to multiple public MRS datasets, including non-mainstream genres and listener demographics, to ensure generalizability.

Conclusion

This study delivers a systematic comparison of frozen pretrained audio representations on both hot and cold item recommendation settings, demonstrating that cutting-edge MIR embeddings do not guarantee superior MRS performance. Supervised tag-based models (MusiCNN) are optimal for hot-item settings, while large-scale generative or token-based self-supervised models (Jukebox, MusicFM, MuQ-MuLan) are more effective under cold start. Such task-embedding interplay underscores the need for tailored selection of backend models, careful experimental design, and expanded hybrid architectures in the future development of music recommender systems.