Selective Auditory Attention Decoding for Symbolic Music

• Selective Auditory Attention Decoding is a computational method that models how listeners isolate target musical streams using both absolute and relative musical features like pitch and rhythm.
• Embedding strategies such as FME and PiRhDy, enhanced by attention mechanisms like RIPO, ensure translational invariance and context-awareness in decoding salient auditory cues.
• These approaches improve practical applications in melody completion, accompaniment assignment, and genre classification, as validated by superior predictive metrics and perceptual assessments.

Selective Auditory Attention Decoding refers to the computational modeling and neural decoding of how listeners segregate, focus on, and process a target auditory stream—such as a melody—within complex polyphonic or multi-instrumental soundscapes. Decoding this process is essential for symbolic music modeling, MIR, and cognitive neuroscience, as it illuminates how models and biological listeners attend to salient musical attributes (pitch, rhythm, dynamics) and contextually relevant streams (melody vs. harmony) for downstream tasks such as music generation, completion, or classification.

1. Theoretical Foundations: Selective Auditory Attention in Symbolic Music

Selective attention in auditory processing entails isolating specific features—such as a melodic line—from a mixture, based on both absolute parameters (e.g., pitch height) and relative cues (e.g., pitch intervals, rhythm patterns). Traditional deep learning approaches using word embeddings or one-hot tokenizations fundamentally lack structured mechanisms to encode and exploit the absolute and relative dimensions that underpin selective musical attention. These representations fail to guarantee properties critical for musical attention, such as translational invariance (equal intervals should be equidistant in embedding space) and explicit shift-vectors for musical transposition—both are vital for modeling attentional selection and invariance under transposition (Guo et al., 2022).

2. Embedding Strategies Enabling Attention Decoding

The problem of selective auditory attention decoding in symbolic music has motivated the development of embedding spaces that explicitly encode both absolute and relative musical features. Two notable embedding methodologies grounded in this principle are Fundamental Music Embedding (FME) and the PiRhDy framework.

Fundamental Music Embedding (FME)

FME utilizes a bias-adjusted sinusoidal encoding:

• Each fundamental music token $f$ (such as pitch, onset, or duration) is embedded as:

$$FME(f)=\left[P_0(f),P_1(f),...,P_{\frac d2-1}(f)\right]\in\mathbb{R}^d$$

with each sub-vector $$P_k(f) = [\sin(w_k f)+b_{\sin,k}, \cos(w_k f)+b_{\cos,k}]$$ for frequency $w_k$ and trainable biases $b_{\sin,k},b_{\cos,k}$.

• For relative shifts $\Delta f$, the relative embedding omits the biases:

$$FMS(\Delta f)=\left[A_0(\Delta f), A_1(\Delta f), ..., A_{\frac d2-1}(\Delta f)\right]\in\mathbb{R}^d,$$

where $$A_k(\Delta f) = [\sin(w_k \Delta f), \cos(w_k \Delta f)]$$.

This construction ensures that: - Euclidean distances in FME-space are only a function of $|f_1-f_2|$ (translational invariance), - An explicit, closed-form rotation transforms $FME(f)$ to $FME(f+\Delta f)$, supporting direct modeling of attention across transpositions (Guo et al., 2022).

PiRhDy and Multi-faceted Embeddings

PiRhDy provides an alternative: token-level embeddings $e_{\text{token}}$ are constructed by concatenating and nonlinearly transforming pitch, rhythm, and dynamics features, each derived as:

• Pitch: Fusion of chroma and octave encodings,
• Rhythm: Sinusoidal encoding of inter-onset interval plus note state,
• Dynamics: Embedded velocity.

Context modeling at multiple levels (local: melodic/harmonic windows; global: period/track encoding) produces embeddings explicitly attuned to target melodic or harmonic streams, providing machinery for selective attention decoding (Liang et al., 2020).

3. Attention Mechanisms: Relative and Context-aware Decoding

Decoding selective auditory attention is operationalized in neural architectures via specialized attention layers and context modeling frameworks:

• RIPO Attention: Extends standard transformer attention with three additive terms representing relative position, pitch, and onset:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\Bigl(\frac{QK^T+S_{rel}^i+S_{rel}^P+S_{rel}^O}{\sqrt{D_h}}\Bigr)V$$

where $S_{rel}^i$, $S_{rel}^P$, $S_{rel}^O$ encode, respectively, relative index, pitch, and onset relationships, computed using FME/FMS. This allows the model to natively modulate attention flow based on musically relevant relative properties.

• Hierarchical Context Modeling (PiRhDy): Attention is further focused via context windows and attention layers across melodic and harmonic axes, refining token embeddings to be preferentially attentive to either melody (FME_GM) or harmony (FME_GH), depending on downstream task requirements (Liang et al., 2020).

4. Empirical Evaluation: Decoding Efficacy and Musical Relevance

Table: Summary of Key Results on Embedding-based Selective Attention Decoding

Model/Embedding	Melody Completion (MAP)	Accompaniment Assignment (MAP)	Genre Classification (AUC, F1)
FME_GM (PiRhDy)	≈0.48	≈0.48	0.895, 0.668 (TOP-MAGD)
FME_GH (PiRhDy)	≈0.30	≈0.52	0.891, 0.663 (TOP-MAGD)
RIPO+FME Transformer	$CE_{sum}=2.367$	–	KL divergence, ISR, AR improved

Ablation studies confirm that each facet—relative pitch, temporal context, and feature integration—incrementally enhances attentional selectivity and decoding fidelity. Removal of any RIPO attention term or positional encoding worsens cross-entropy losses, indicating their necessity in modeling selective musical attention (Guo et al., 2022). In genre classification and accompaniment assignment, context-specific embeddings (melody-preferred or harmony-preferred) lead to marked accuracy improvements over baseline embeddings (Liang et al., 2020).

5. Practical Implications in Symbolic Music Modeling

Embeddings and attention mechanisms tailored for selective auditory attention decoding are deployed in a range of symbolic music MIR tasks:

• Melody Completion: Decoding the next salient note in a melodic context.
• Accompaniment Assignment: Selecting contextually appropriate harmony given a melodic fragment.
• Genre Classification: Exploiting attentional focus on discriminative musical dimensions.

Both FME via RIPO-attention and PiRhDy’s hierarchical modeling enable robust, pretrained plug-and-play embeddings for downstream music tasks, outperforming previous motif-based or Cartesian-product symbol encodings and providing efficient solutions for scalable symbolic music representation (Guo et al., 2022, Liang et al., 2020).

6. Evaluation Metrics and Musical Coherence

Objective and subjective assessments directly reflect attention decoding efficacy:

• Cross-entropy (CE): Evaluates next-token predictive likelihoods.
• Repetition metrics (seq-rep-4): Quantifies avoidance of degenerate looping in generated music.
• KL divergence to true distributions, ISR, AR: Assess musicality and coherence.
• Listening Tests: Subjective enjoyment and correctness evaluated via Likert ratings demonstrate perceptual gains for attention-aware models (Guo et al., 2022).

7. Limitations and Research Directions

While current frameworks enable explicit modeling and decoding of selective auditory attention in symbolic music, several practical aspects—such as exact hyperparameterization, optimizer details, and further extensions to polyphonic or multi-instrument scenarios—are not exhaustively specified. A plausible implication is that further research may refine these embeddings and context models for richer and more nuanced auditory attention modeling in broader musical contexts (Liang et al., 2020).

---

References:

• "A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling" (Guo et al., 2022)
• "PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music" (Liang et al., 2020)