Musical Word Embedding for Music Tagging and Retrieval (2404.13569v2)

Abstract: Word embedding has become an essential means for text-based information retrieval. Typically, word embeddings are learned from large quantities of general and unstructured text data. However, in the domain of music, the word embedding may have difficulty understanding musical contexts or recognizing music-related entities like artists and tracks. To address this issue, we propose a new approach called Musical Word Embedding (MWE), which involves learning from various types of texts, including both everyday and music-related vocabulary. We integrate MWE into an audio-word joint representation framework for tagging and retrieving music, using words like tag, artist, and track that have different levels of musical specificity. Our experiments show that using a more specific musical word like track results in better retrieval performance, while using a less specific term like tag leads to better tagging performance. To balance this compromise, we suggest multi-prototype training that uses words with different levels of musical specificity jointly. We evaluate both word embedding and audio-word joint embedding on four tasks (tag rank prediction, music tagging, query-by-tag, and query-by-track) across two datasets (Million Song Dataset and MTG-Jamendo). Our findings show that the suggested MWE is more efficient and robust than the conventional word embedding.

• The paper's main contribution is the development of MWE that combines general and music-specific texts using skip-gram modeling to capture nuanced musical contexts.
• It employs an audio-word joint embedding framework with metric learning and triplet networks, using tag, artist, and track ID supervision to improve tagging and retrieval.
• Evaluation on the MSD and MTG-Jamendo datasets shows improved nDCG, ROCAUC, and recall metrics, demonstrating MWE's effectiveness in zero-shot learning.

Musical Word Embedding: Enhancing Music Tagging and Retrieval through Domain-Specific Contextualization

Introduction

The proliferation of digital music platforms has benefitted from advancements in music tagging and retrieval, essential components of Music Information Retrieval (MIR). Traditional tagging methods relying on general word embeddings often falter in accurately interpreting domain-specific nuances. In response, this paper introduces a novel approach dubbed Musical Word Embedding (MWE), which distinctively leverages both general and music-specific text corpora to enhance music tagging and retrieval tasks across several benchmarks.

Methodology

Word Embedding Training

The proposed MWE paradigm addresses the contextual gap by incorporating texts varying in musical specificity—from general-purpose documents like Wikipedia entries to music-specific data such as review texts, tags, and artist/track IDs. This comprehensive corpus selection offers a nuanced embedding capable of understanding both broad and niche musical contexts.

For modeling word relationships, the authors employed the skip-gram model, a part of the Word2Vec suite, due to its efficacy in capturing associations between less frequently occurring words, which is beneficial for representing music-specific vocabulary.

1. General Corpus: Includes basic, non-music-specific words from extensive databases like Wikipedia.
2. Music-Specific Corpus: Integrates artist and track IDs, and tags from music datasets alongside music reviews, thereby embedding music-related vocabulary and concepts effectively.

The refinement of semantic connections among these words is achieved by maximizing the log probability of contextually related word pairs, a method that enhances the relevance of track and artist IDs when placed within musical discussions.

Audio-Word Joint Embedding

The dual modality embedding employs a metric learning framework, bridging audio and word embeddings by exploiting their contextual similarities. Music tracks and their associated text items (tags, artist IDs, etc.) form triplet networks, used to optimize a max-margin hinge loss function under various supervisory signals:

• Tag-based supervision
• Artist and track IDs for heightened musical specificity.

This structured embedding supports robust music tagging and query-by-track functionalities with the benefit of zero-shot learning capabilities, enabling the model to recognize and tag previously unseen music categories and tags.

Evaluation and Results

1. Datasets Used: The model's efficacy was explored using the Million Song Dataset (MSD) and the MTG-Jamendo dataset, focusing on tasks like tag rank prediction, query-by-tag, and query-by-track functionalities.
2. Performance Metrics:
   • For word embedding, metrics like normalized discounted cumulative gain (nDCG) and area under the ROC curve (ROCAUC) were calculated to assess tag-to-tag and tag-to-track retrieval accuracy.
   • For audio-word joint embedding, recall metrics at various levels (R@K) and ROCAUC were used to evaluate the retrieval quality and tagging accuracy.
3. Results:
   • The MWE model outperformed general word embeddings in contextually rich music tagging and retrieval tasks, demonstrating superior handling of genre-specific vocabularies.
   • Audio-word metric learning, especially when trained with artist and track ID supervisions, showed improved performance in predicting and retrieving over both seen and unseen data, harnessing zero-shot learning effectively.

Implications and Future Directions

The development of MWE suggests significant potential for applications in digital music platforms, potentially enhancing user experience through improved recommendation systems and search functionalities. Future work could extend the robustness of MWE to encompass multi-lingual data sets and further explore the integration of other forms of metadata to enrich embedding quality.

By aligning domain-specific text with audio data effectively, MWE sets the stage for more intuitive and context-aware systems in the music information retrieval field, promising exciting developments for both academic research and practical applications in digital music services.