---
title: 'PAMT: Perceptually-Aligned MERT Transformer'
url: https://www.emergentmind.com/topics/perceptually-aligned-mert-transformer-pamt
type: topic
---

# PAMT: Perceptually-Aligned MERT Transformer

Searching arXiv for the PAMT paper and directly related references.
Tool call: arxiv_search({"query":"arXiv:2509.04985 OR \"Training a Perceptual Model for Evaluating Auditory Similarity in Music Adversarial Attack\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"})
Searching for the MERT and contrastive-learning references.
Tool call: arxiv_search({"query":"MERT v0 Li et al 2023 music arXiv", "max_results":10, "sort_by":"relevance", "sort_order":"descending"})
Perceptually-Aligned MERT Transformer (PAMT) is a framework for learning music representations that are explicitly aligned with human auditory perception and robust to imperceptible adversarial perturbations. It is introduced in the context of Music Information Retrieval (MIR), where adversarial attacks can substantially degrade system performance despite remaining difficult for human listeners to detect. PAMT addresses the stated misalignment between model feature spaces and human auditory perception by combining a frozen MERT encoder with a psychoacoustically-conditioned sequential contrastive transformer projection head. In the reported experiments, the framework achieves a Spearman correlation coefficient of \(0.65\) with subjective scores and an average \(9.15\%\) improvement in robust accuracy on downstream MIR tasks under diverse perceptual adversarial attacks [2509.04985].

## 1. Problem formulation and design objective

The motivating problem is that MIR systems are described as highly vulnerable to adversarial attacks that are often imperceptible to humans, while existing defenses and perceptual metrics frequently fail to capture the relevant auditory nuances. The paper attributes this to a misalignment between learned feature spaces and human auditory perception, and supports that claim with initial listening tests showing low correlation between common metrics and human judgments [2509.04985].

Within that framing, PAMT is not merely a similarity metric in the narrow sense. It is presented as a representation-learning framework whose output space is intended to satisfy two conditions simultaneously: first, embeddings of original and perceptually indistinguishable perturbed music should remain close; second, those embeddings should retain downstream utility for MIR tasks under attack. The resulting representation is therefore used both for perceptual similarity assessment and for adversarially robust MIR evaluation [2509.04985].

A central premise is that perceptual invariance should be architecturally integrated rather than left to emerge implicitly. The paper identifies this as its core innovation, namely a psychoacoustically-conditioned sequential contrastive transformer built as a lightweight projection head on top of a frozen MERT encoder. This positioning distinguishes PAMT from approaches that either rely on generic perceptual metrics or use contrastive learning without psychoacoustic conditioning [2509.04985].

## 2. Architectural composition

PAMT consists of two principal components: a frozen MERT backbone and a Psychoacoustically-Conditioned Sequential Contrastive Transformer (PCSCT) projection head. The frozen encoder receives input audio as 10-second clips resampled to 24 kHz and produces a frame-wise embedding sequence
\[
E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},
\]
where \(T\) depends on audio length and MERT’s internal down-sampling. These embeddings are described as capturing broad musical semantics but not being optimized for adversarial robustness or perceptual alignment [2509.04985].

The PCSCT head refines \(E_{\mathrm{mert}}\) into a 128-dimensional sequence representation robust to imperceptible perturbations. It comprises a Perturbation Parameter Encoder (PPE), a 4-layer Transformer encoder stack, a final linear projection, and mean pooling. The PPE takes psychoacoustic perturbation parameters \(P_{\mathrm{params}}\)—examples given include Bark-band, \(L_2\) noise level, and pitch shift—and maps them through a 2-layer ReLU MLP into a 64-dimensional conditioning vector
\[
c_{\mathrm{perturb}} \in \mathbb{R}^{64}.
\]
The Transformer stack uses 4 attention heads per layer, model dimension \(d=256\), feed-forward inner dimension \(1024\), and Pre-LayerNorm with GELU activations. The incoming 768-dimensional MERT tokens are first linearly projected to 256 dimensions and then processed by standard self-attention and feed-forward sublayers, with FiLM-based conditioning injected at every layer [2509.04985].

After the Transformer, a final linear projection maps each 256-dimensional token to a 128-dimensional output token, yielding
\[
Z_{\mathrm{pamt}} \in \mathbb{R}^{T \times 128}.
\]
For downstream tasks and contrastive similarity, the sequence is mean-pooled:
\[
\bar Z_{\mathrm{pamt}} = \frac{1}{T}\sum_{t=1}^{T} Z_{\mathrm{pamt}}[t] \in \mathbb{R}^{128}.
\]

| Component | Specification | Output |
|---|---|---|
| Frozen MERT encoder | 10-second clips; 24 kHz; frozen MERT-v0 | \(E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768}\) |
| Perturbation Parameter Encoder | 2-layer ReLU MLP | \(c_{\mathrm{perturb}} \in \mathbb{R}^{64}\) |
| PCSCT Transformer | 4 layers; 4 heads; \(d=256\); FFN \(=1024\); Pre-LN; GELU | Conditioned token sequence |
| Final projection + pooling | Linear \(256 \rightarrow 128\); mean-pooling | \(\bar Z_{\mathrm{pamt}} \in \mathbb{R}^{128}\) |

One noteworthy technical detail is that the architecture description specifies 24 kHz input to the frozen MERT encoder, whereas the training data preparation section states that audio is resampled to 16 kHz and cut into 10-second segments. The paper presents both statements explicitly [2509.04985]. A plausible implication is that implementation details of the preprocessing interface are important for reproduction.

## 3. Sequential contrastive learning objective

PAMT is trained with an InfoNCE-style sequential contrastive objective defined over mean-pooled representations of original and perturbed audio pairs. The training objective pulls together embeddings of \((\text{original}, \text{perturbed})\) pairs and pushes apart other samples within the same batch. Using
\[
z_i^{\mathrm{orig}} = \bar Z_{\mathrm{pamt}}(A_i^{\mathrm{orig}}), \qquad
z_i^{\mathrm{pert}} = \bar Z_{\mathrm{pamt}}(A_i^{\mathrm{pert}}),
\]
and cosine similarity
\[
\mathrm{sim}(u,v)=\frac{u \cdot v}{\|u\|\,\|v\|},
\]
with temperature \(\tau = 0.1\), the loss for pair \(i\) is
\[
L_i = - \log
\frac{
\exp(\mathrm{sim}(z_i^{\mathrm{orig}}, z_i^{\mathrm{pert}})/\tau)
}{
\exp(\mathrm{sim}(z_i^{\mathrm{orig}}, z_i^{\mathrm{pert}})/\tau) + \sum_{j \ne i}\exp(\mathrm{sim}(z_i^{\mathrm{orig}}, z_j^{\mathrm{pert}})/\tau)
}.
\]
The batch-level objective is
\[
L_{\mathrm{PCSCT}} = \frac{1}{N}\sum_{i=1}^{N} L_i,
\]
where \(N\) is the number of original–perturbed pairs in the batch [2509.04985].

The paper characterizes this objective as “sequential contrastive learning” because it operates on representations derived from temporal embedding sequences rather than only on fixed handcrafted descriptors. Although the loss is applied to mean-pooled vectors, the encoder preceding pooling is explicitly sequence-based. The stated role of this objective is to enforce collapse between embeddings of original and perceptually indistinguishable perturbed versions while preserving separation from other musical items [2509.04985].

The training section further states that the contrastive loss implicitly encourages uniform embedding distribution, following Wang and Isola (2020), but the PAMT contribution lies in coupling that generic contrastive pressure to a psychoacoustically-conditioned sequence model rather than using contrastive learning alone [2509.04985].

## 4. Psychoacoustic conditioning and FiLM modulation

The principal mechanism for perceptual alignment is explicit psychoacoustic conditioning via Feature-wise Linear Modulation (FiLM). For Transformer layer \(l\), if \(h_l \in \mathbb{R}^{T \times 256}\) denotes the FFN sublayer output, the model computes layer-specific modulation vectors \((\gamma_l,\beta_l)\in\mathbb{R}^{256}\) from the conditioning vector \(c_{\mathrm{perturb}}\):
\[
\gamma_l = W_l^\gamma c_{\mathrm{perturb}} + b_l^\gamma,
\qquad
\beta_l = W_l^\beta c_{\mathrm{perturb}} + b_l^\beta.
\]
These are then applied as
\[
\mathrm{FiLM}(h_l, c_{\mathrm{perturb}}) = \gamma_l \odot h_l + \beta_l.
\]
The paper states that this allows the Transformer to alter its attention and feature transforms based on the psychoacoustic profile of the perturbation [2509.04985].

The intended effect is to teach the model to ignore distortions that fall below human masking thresholds, with Bark-band masking given as an example. In the paper’s account, this is what makes the representation perceptually aligned in a specifically psychoacoustic sense rather than merely invariant under arbitrary augmentation. The conditioning signal is therefore not an auxiliary label in the usual sense; it modulates internal computation at each layer [2509.04985].

This design also underlies the paper’s explanation of why PAMT is more perceptually aligned than prior models. The authors identify three interacting factors: psychoacoustic conditioning through FiLM, sequential contrastive learning that enforces temporal consistency, and the use of a frozen large-scale music encoder whose broad musical semantics are then specialized by the lightweight projection head. The empirical evidence cited for this alignment is the increase in Spearman \(\rho\) from baseline levels to \(0.65\), together with qualitative listening-test observations that nearest neighbors in PAMT’s embedding space more closely match human-judged similar excerpts [2509.04985].

A common misconception would be to interpret PAMT as simply a new distance function layered on top of MERT. The description in the paper is more specific: the framework learns a new representation space \(Z_{\mathrm{pamt}}\) through psychoacoustically-conditioned transformation of frozen MERT embeddings, and the distance function is then defined in that learned space [2509.04985].

## 5. Data, perturbations, and optimization protocol

The training corpus is assembled from FMA, GTZAN, and MTG-Jamendo, with 1,000 unique tracks from each dataset. The preprocessing section states that audio is resampled to 16 kHz and segmented into 10-second excerpts, producing 18,000 original–perturbed pairs. The split is 80% train and 20% test on unique tracks [2509.04985].

Six perturbation types are used: \(L_2\) noise, \(L_\infty\) noise, Bark-band noise, pitch shift, speed change, and dynamics compression. Their parameter ranges are randomized, with the paper referring to Table 1 for specifics. These perturbations define the positive pairs for contrastive learning and also form the basis of the perceptual adversaries used in downstream robustness evaluation [2509.04985].

Optimization uses AdamW with initial learning rate \(1\times 10^{-4}\), weight decay \(1\times 10^{-5}\), and batch size 32 original–perturbed pairs. The schedule is cosine-annealing learning rate with 10% warm-up. The contrastive temperature is fixed at \(\tau=0.1\), maximum training runs for 100 epochs, and early stopping is triggered after 10 epochs without validation-set Spearman improvement [2509.04985].

The paper’s regularization narrative emphasizes three points. First, the frozen MERT backbone is said to prevent catastrophic forgetting. Second, FiLM conditioning is said to focus learning on psychoacoustic invariances. Third, the contrastive loss is said to encourage a more uniform embedding distribution. Taken together, these design choices frame PAMT as a lightweight adaptation of a large pre-trained encoder rather than a fully end-to-end re-training procedure [2509.04985].

## 6. Evaluation methodology, results, and significance

Alignment with human judgments is measured through Spearman’s rank correlation coefficient between model-predicted distances and human 2AFC scores. For \(n\) test samples with ranks \(R_{x,i}\) and \(R_{y,i}\), the paper gives
\[
\rho = 1 - \frac{6\sum_{i=1}^{n} d_i^2}{n(n^2-1)},
\qquad
d_i = R_{x,i} - R_{y,i}.
\]
Reported correlation results are: best baseline, FAD with raw MERT embeddings, \(\rho=0.44\); MERT + MLP contrastive, \(\rho=0.55\); and PAMT (PCSCT), \(\rho=0.65\) [2509.04985].

For downstream adversarial evaluation, the paper considers two MIR tasks: Cover Song Identification (CSI), measured by mean Average Precision (mAP), and Music Genre Classification (MGC), measured by classification accuracy. Attacks are constrained by
\[
d_{\mathrm{pamt}}(A',A) = \left\| \mathrm{mean\_pool}(Z_{\mathrm{pamt}}(A')) - \mathrm{mean\_pool}(Z_{\mathrm{pamt}}(A)) \right\|_2 \le \epsilon.
\]
This definition makes the learned PAMT space itself the perceptual constraint for adversarial generation [2509.04985].

Under the union of attacks over all six perturbation types, the paper reports worst-case robust performance as follows.

| Method | CSI Clean / Robust | MGC Clean / Robust |
|---|---:|---:|
| No Defense | 0.845 / 0.021 | 0.828 / 0.035 |
| Standard AT (\(L_p\)) | 0.820 / 0.315 | 0.805 / 0.330 |
| MERT+MLP AT | 0.831 / 0.415 | 0.812 / 0.402 |
| PAMT AT (Ours) | 0.835 / 0.535 | 0.818 / 0.465 |

The average robust-accuracy gain of PAMT over the best baseline is reported as \(9.15\%\). The paper also highlights task-specific gains relative to the best baseline, namely \(+0.120\) mAP in CSI and \(+0.063\) accuracy in MGC [2509.04985].

The significance assigned to these results is twofold. First, PAMT improves agreement with subjective auditory similarity judgments compared with existing metrics. Second, embeddings learned under this criterion transfer to adversarially robust MIR tasks. The paper therefore frames PAMT as evidence that architecturally integrated psychoacoustic conditioning can improve both perceptual fidelity and robustness. This suggests a broader methodological implication: in music-domain adversarial settings, robustness may depend not only on perturbation budgets and optimization procedures, but also on whether the representation space is explicitly aligned with human auditory masking phenomena [2509.04985].

Source: https://www.emergentmind.com/topics/perceptually-aligned-mert-transformer-pamt