PAMT: Perceptually-Aligned MERT Transformer

• The paper introduces PAMT, a framework that fuses a frozen MERT encoder with a psychoacoustically-conditioned sequential contrastive transformer to align musical embeddings with human auditory perception.
• It employs an InfoNCE-style sequential contrastive learning objective and FiLM modulation to maintain consistency between original and perceptually perturbed audio pairs while ensuring robustness for downstream MIR tasks.
• Experiments demonstrate a Spearman correlation of 0.65 with subjective scores and a 9.15% robust accuracy gain in tasks like cover song identification and genre classification under adversarial attacks.

Searching arXiv for the PAMT paper and directly related references. Tool call: arxiv_search({"query":"(Liu et al., 5 Sep 2025) OR \"Training a Perceptual Model for Evaluating Auditory Similarity in Music Adversarial Attack\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) Searching for the MERT and contrastive-learning references. Tool call: arxiv_search({"query":"MERT v0 Li et al 2023 music arXiv", "max_results":10, "sort_by":"relevance", "sort_order":"descending"}) Perceptually-Aligned MERT Transformer (PAMT) is a framework for learning music representations that are explicitly aligned with human auditory perception and robust to imperceptible adversarial perturbations. It is introduced in the context of Music Information Retrieval (MIR), where adversarial attacks can substantially degrade system performance despite remaining difficult for human listeners to detect. PAMT addresses the stated misalignment between model feature spaces and human auditory perception by combining a frozen MERT encoder with a psychoacoustically-conditioned sequential contrastive transformer projection head. In the reported experiments, the framework achieves a Spearman correlation coefficient of $0.65$ with subjective scores and an average $9.15\%$ improvement in robust accuracy on downstream MIR tasks under diverse perceptual adversarial attacks (Liu et al., 5 Sep 2025).

1. Problem formulation and design objective

The motivating problem is that MIR systems are described as highly vulnerable to adversarial attacks that are often imperceptible to humans, while existing defenses and perceptual metrics frequently fail to capture the relevant auditory nuances. The paper attributes this to a misalignment between learned feature spaces and human auditory perception, and supports that claim with initial listening tests showing low correlation between common metrics and human judgments (Liu et al., 5 Sep 2025).

Within that framing, PAMT is not merely a similarity metric in the narrow sense. It is presented as a representation-learning framework whose output space is intended to satisfy two conditions simultaneously: first, embeddings of original and perceptually indistinguishable perturbed music should remain close; second, those embeddings should retain downstream utility for MIR tasks under attack. The resulting representation is therefore used both for perceptual similarity assessment and for adversarially robust MIR evaluation (Liu et al., 5 Sep 2025).

A central premise is that perceptual invariance should be architecturally integrated rather than left to emerge implicitly. The paper identifies this as its core innovation, namely a psychoacoustically-conditioned sequential contrastive transformer built as a lightweight projection head on top of a frozen MERT encoder. This positioning distinguishes PAMT from approaches that either rely on generic perceptual metrics or use contrastive learning without psychoacoustic conditioning (Liu et al., 5 Sep 2025).

2. Architectural composition

PAMT consists of two principal components: a frozen MERT backbone and a Psychoacoustically-Conditioned Sequential Contrastive Transformer (PCSCT) projection head. The frozen encoder receives input audio as 10-second clips resampled to 24 kHz and produces a frame-wise embedding sequence

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$

where $T$ depends on audio length and MERT’s internal down-sampling. These embeddings are described as capturing broad musical semantics but not being optimized for adversarial robustness or perceptual alignment (Liu et al., 5 Sep 2025).

The PCSCT head refines $E_{\mathrm{mert}}$ into a 128-dimensional sequence representation robust to imperceptible perturbations. It comprises a Perturbation Parameter Encoder (PPE), a 4-layer Transformer encoder stack, a final linear projection, and mean pooling. The PPE takes psychoacoustic perturbation parameters $P_{\mathrm{params}}$—examples given include Bark-band, $L_2$ noise level, and pitch shift—and maps them through a 2-layer ReLU MLP into a 64-dimensional conditioning vector

$c_{\mathrm{perturb}} \in \mathbb{R}^{64}.$

The Transformer stack uses 4 attention heads per layer, model dimension $d=256$, feed-forward inner dimension $1024$, and Pre-LayerNorm with GELU activations. The incoming 768-dimensional MERT tokens are first linearly projected to 256 dimensions and then processed by standard self-attention and feed-forward sublayers, with FiLM-based conditioning injected at every layer (Liu et al., 5 Sep 2025).

After the Transformer, a final linear projection maps each 256-dimensional token to a 128-dimensional output token, yielding

$9.15\%$0

For downstream tasks and contrastive similarity, the sequence is mean-pooled: $9.15\%$1

Component	Specification	Output
Frozen MERT encoder	10-second clips; 24 kHz; frozen MERT-v0	$9.15\%$2
Perturbation Parameter Encoder	2-layer ReLU MLP	$9.15\%$3
PCSCT Transformer	4 layers; 4 heads; $9.15\%$4; FFN $9.15\%$5; Pre-LN; GELU	Conditioned token sequence
Final projection + pooling	Linear $9.15\%$6; mean-pooling	$9.15\%$7

One noteworthy technical detail is that the architecture description specifies 24 kHz input to the frozen MERT encoder, whereas the training data preparation section states that audio is resampled to 16 kHz and cut into 10-second segments. The paper presents both statements explicitly (Liu et al., 5 Sep 2025). A plausible implication is that implementation details of the preprocessing interface are important for reproduction.

3. Sequential contrastive learning objective

PAMT is trained with an InfoNCE-style sequential contrastive objective defined over mean-pooled representations of original and perturbed audio pairs. The training objective pulls together embeddings of $9.15\%$8 pairs and pushes apart other samples within the same batch. Using

$9.15\%$9

and cosine similarity

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$0

with temperature $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$1, the loss for pair $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$2 is

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$3

The batch-level objective is

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$4

where $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$5 is the number of original–perturbed pairs in the batch (Liu et al., 5 Sep 2025).

The paper characterizes this objective as “sequential contrastive learning” because it operates on representations derived from temporal embedding sequences rather than only on fixed handcrafted descriptors. Although the loss is applied to mean-pooled vectors, the encoder preceding pooling is explicitly sequence-based. The stated role of this objective is to enforce collapse between embeddings of original and perceptually indistinguishable perturbed versions while preserving separation from other musical items (Liu et al., 5 Sep 2025).

The training section further states that the contrastive loss implicitly encourages uniform embedding distribution, following Wang and Isola (2020), but the PAMT contribution lies in coupling that generic contrastive pressure to a psychoacoustically-conditioned sequence model rather than using contrastive learning alone (Liu et al., 5 Sep 2025).

4. Psychoacoustic conditioning and FiLM modulation

The principal mechanism for perceptual alignment is explicit psychoacoustic conditioning via Feature-wise Linear Modulation (FiLM). For Transformer layer $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$6, if $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$7 denotes the FFN sublayer output, the model computes layer-specific modulation vectors $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$8 from the conditioning vector $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$9: $T$0 These are then applied as

$T$1

The paper states that this allows the Transformer to alter its attention and feature transforms based on the psychoacoustic profile of the perturbation (Liu et al., 5 Sep 2025).

The intended effect is to teach the model to ignore distortions that fall below human masking thresholds, with Bark-band masking given as an example. In the paper’s account, this is what makes the representation perceptually aligned in a specifically psychoacoustic sense rather than merely invariant under arbitrary augmentation. The conditioning signal is therefore not an auxiliary label in the usual sense; it modulates internal computation at each layer (Liu et al., 5 Sep 2025).

This design also underlies the paper’s explanation of why PAMT is more perceptually aligned than prior models. The authors identify three interacting factors: psychoacoustic conditioning through FiLM, sequential contrastive learning that enforces temporal consistency, and the use of a frozen large-scale music encoder whose broad musical semantics are then specialized by the lightweight projection head. The empirical evidence cited for this alignment is the increase in Spearman $T$2 from baseline levels to $T$3, together with qualitative listening-test observations that nearest neighbors in PAMT’s embedding space more closely match human-judged similar excerpts (Liu et al., 5 Sep 2025).

A common misconception would be to interpret PAMT as simply a new distance function layered on top of MERT. The description in the paper is more specific: the framework learns a new representation space $T$4 through psychoacoustically-conditioned transformation of frozen MERT embeddings, and the distance function is then defined in that learned space (Liu et al., 5 Sep 2025).

5. Data, perturbations, and optimization protocol

The training corpus is assembled from FMA, GTZAN, and MTG-Jamendo, with 1,000 unique tracks from each dataset. The preprocessing section states that audio is resampled to 16 kHz and segmented into 10-second excerpts, producing 18,000 original–perturbed pairs. The split is 80% train and 20% test on unique tracks (Liu et al., 5 Sep 2025).

Six perturbation types are used: $T$5 noise, $T$6 noise, Bark-band noise, pitch shift, speed change, and dynamics compression. Their parameter ranges are randomized, with the paper referring to Table 1 for specifics. These perturbations define the positive pairs for contrastive learning and also form the basis of the perceptual adversaries used in downstream robustness evaluation (Liu et al., 5 Sep 2025).

Optimization uses AdamW with initial learning rate $T$7, weight decay $T$8, and batch size 32 original–perturbed pairs. The schedule is cosine-annealing learning rate with 10% warm-up. The contrastive temperature is fixed at $T$9, maximum training runs for 100 epochs, and early stopping is triggered after 10 epochs without validation-set Spearman improvement (Liu et al., 5 Sep 2025).

The paper’s regularization narrative emphasizes three points. First, the frozen MERT backbone is said to prevent catastrophic forgetting. Second, FiLM conditioning is said to focus learning on psychoacoustic invariances. Third, the contrastive loss is said to encourage a more uniform embedding distribution. Taken together, these design choices frame PAMT as a lightweight adaptation of a large pre-trained encoder rather than a fully end-to-end re-training procedure (Liu et al., 5 Sep 2025).

6. Evaluation methodology, results, and significance

Alignment with human judgments is measured through Spearman’s rank correlation coefficient between model-predicted distances and human 2AFC scores. For $E_{\mathrm{mert}}$0 test samples with ranks $E_{\mathrm{mert}}$1 and $E_{\mathrm{mert}}$2, the paper gives

$E_{\mathrm{mert}}$3

Reported correlation results are: best baseline, FAD with raw MERT embeddings, $E_{\mathrm{mert}}$4; MERT + MLP contrastive, $E_{\mathrm{mert}}$5; and PAMT (PCSCT), $E_{\mathrm{mert}}$6 (Liu et al., 5 Sep 2025).

For downstream adversarial evaluation, the paper considers two MIR tasks: Cover Song Identification (CSI), measured by mean Average Precision (mAP), and Music Genre Classification (MGC), measured by classification accuracy. Attacks are constrained by

$E_{\mathrm{mert}}$7

This definition makes the learned PAMT space itself the perceptual constraint for adversarial generation (Liu et al., 5 Sep 2025).

Under the union of attacks over all six perturbation types, the paper reports worst-case robust performance as follows.

Method	CSI Clean / Robust	MGC Clean / Robust
No Defense	0.845 / 0.021	0.828 / 0.035
Standard AT ($E_{\mathrm{mert}}$8)	0.820 / 0.315	0.805 / 0.330
MERT+MLP AT	0.831 / 0.415	0.812 / 0.402
PAMT AT (Ours)	0.835 / 0.535	0.818 / 0.465

The average robust-accuracy gain of PAMT over the best baseline is reported as $E_{\mathrm{mert}}$9. The paper also highlights task-specific gains relative to the best baseline, namely $P_{\mathrm{params}}$0 mAP in CSI and $P_{\mathrm{params}}$1 accuracy in MGC (Liu et al., 5 Sep 2025).

The significance assigned to these results is twofold. First, PAMT improves agreement with subjective auditory similarity judgments compared with existing metrics. Second, embeddings learned under this criterion transfer to adversarially robust MIR tasks. The paper therefore frames PAMT as evidence that architecturally integrated psychoacoustic conditioning can improve both perceptual fidelity and robustness. This suggests a broader methodological implication: in music-domain adversarial settings, robustness may depend not only on perturbation budgets and optimization procedures, but also on whether the representation space is explicitly aligned with human auditory masking phenomena (Liu et al., 5 Sep 2025).