PAMT: Perceptually-Aligned MERT Transformer

• The paper introduces PAMT, a framework that fuses a frozen MERT encoder with a psychoacoustically-conditioned sequential contrastive transformer to align musical embeddings with human auditory perception.
• It employs an InfoNCE-style sequential contrastive learning objective and FiLM modulation to maintain consistency between original and perceptually perturbed audio pairs while ensuring robustness for downstream MIR tasks.
• Experiments demonstrate a Spearman correlation of 0.65 with subjective scores and a 9.15% robust accuracy gain in tasks like cover song identification and genre classification under adversarial attacks.

Searching arXiv for the PAMT paper and directly related references. Tool call: arxiv_search({"^^^^^^^^2query2 OR \2^^^^^^^^"Training a Perceptual Model for Evaluating Auditory Similarity in Music Adversarial Attack\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) Searching for the MERT and contrastive-learning references. Tool call: arxiv_search({"^^^^^^^^2query2 v^^^^2query2^^^^ Li et al 2^^^^2query22^^^^3 music arXiv", "max_results":^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^, "sort_by":"relevance", "sort_order":"descending"}) Perceptually-Aligned MERT Transformer (PAMT) is a framework for learning music representations that are explicitly aligned with human auditory perception and robust to imperceptible adversarial perturbations. It is introduced in the context of Music Information Retrieval (MIR), where adversarial attacks can substantially degrade system performance despite remaining difficult for human listeners to detect. PAMT addresses the stated misalignment between model feature spaces and human auditory perception by combining a frozen MERT encoder with a psychoacoustically-conditioned sequential contrastive transformer projection head. In the reported experiments, the framework achieves a Spearman correlation coefficient of PRESERVED_PLACEHOLDER_^^^^2query2^^^^ with subjective scores and an average PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^ improvement in robust accuracy on downstream MIR tasks under diverse perceptual adversarial attacks (&&&^^^^2query2^^^^&&&).

^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^. Problem formulation and design objective

The motivating problem is that MIR systems are described as highly vulnerable to adversarial attacks that are often imperceptible to humans, while existing defenses and perceptual metrics frequently fail to capture the relevant auditory nuances. The paper attributes this to a misalignment between learned feature spaces and human auditory perception, and supports that claim with initial listening tests showing low correlation between common metrics and human judgments (&&&^^^^2query2^^^^&&&).

Within that framing, PAMT is not merely a similarity metric in the narrow sense. It is presented as a representation-learning framework whose output space is intended to satisfy two conditions simultaneously: first, embeddings of original and perceptually indistinguishable perturbed music should remain close; second, those embeddings should retain downstream utility for MIR tasks under attack. The resulting representation is therefore used both for perceptual similarity assessment and for adversarially robust MIR evaluation (&&&^^^^2query2^^^^&&&).

A central premise is that perceptual invariance should be architecturally integrated rather than left to emerge implicitly. The paper identifies this as its core innovation, namely a psychoacoustically-conditioned sequential contrastive transformer built as a lightweight projection head on top of a frozen MERT encoder. This positioning distinguishes PAMT from approaches that either rely on generic perceptual metrics or use contrastive learning without psychoacoustic conditioning (&&&^^^^2query2^^^^&&&).

2. Architectural composition

PAMT consists of two principal components: a frozen MERT backbone and a Psychoacoustically-Conditioned Sequential Contrastive Transformer (PCSCT) projection head. The frozen encoder receives input audio as ^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^-second clips resampled to 24 kHz and produces a frame-wise embedding sequence

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$

where $T$ depends on audio length and MERT’s internal down-sampling. These embeddings are described as capturing broad musical semantics but not being optimized for adversarial robustness or perceptual alignment (&&&^^^^2query2^^^^&&&).

The PCSCT head refines $E_{\mathrm{mert}}$ into a ^^^^2(Liu et al., 5 Sep 2025) OR \22^^^^8-dimensional sequence representation robust to imperceptible perturbations. It comprises a Perturbation Parameter Encoder (PPE), a 4-layer Transformer encoder stack, a final linear projection, and mean pooling. The PPE takes psychoacoustic perturbation parameters $P_{\mathrm{params}}$—examples given include Bark-band, $L_2$ noise level, and pitch shift—and maps them through a 2-layer ReLU MLP into a 64-dimensional conditioning vector

$c_{\mathrm{perturb}} \in \mathbb{R}^{64}.$

The Transformer stack uses 4 attention heads per layer, model dimension $d=256$, feed-forward inner dimension $1024$, and Pre-LayerNorm with GELU activations. The incoming 768-dimensional MERT tokens are first linearly projected to 256 dimensions and then processed by standard self-attention and feed-forward sublayers, with FiLM-based conditioning injected at every layer (&&&^^^^2query2^^^^&&&).

After the Transformer, a final linear projection maps each 256-dimensional token to a ^^^^2(Liu et al., 5 Sep 2025) OR \22^^^^8-dimensional output token, yielding

PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^

For downstream tasks and contrastive similarity, the sequence is mean-pooled: PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2(Liu et al., 5 Sep 2025) OR \2^^^^

Component	Specification	Output
Frozen MERT encoder	^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^-second clips; 24 kHz; frozen MERT-v^^^^2query2^^^^	PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \22^^^^
Perturbation Parameter Encoder	2-layer ReLU MLP	PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^3
PCSCT Transformer	4 layers; 4 heads; PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^4; FFN PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^5; Pre-LN; GELU	Conditioned token sequence
Final projection + pooling	Linear PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^6; mean-pooling	PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^7

One noteworthy technical detail is that the architecture description specifies 24 kHz input to the frozen MERT encoder, whereas the training data preparation section states that audio is resampled to ^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^6 kHz and cut into ^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^-second segments. The paper presents both statements explicitly (&&&^^^^2query2^^^^&&&). A plausible implication is that implementation details of the preprocessing interface are important for reproduction.

3. Sequential contrastive learning objective

PAMT is trained with an InfoNCE-style sequential contrastive objective defined over mean-pooled representations of original and perturbed audio pairs. The training objective pulls together embeddings of PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^8 pairs and pushes apart other samples within the same batch. Using

PRESERVED_PLACEHOLDER_^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^9

and cosine similarity

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$^^^^2query2^^^^

with temperature $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^, the loss for pair $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$2 is

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$3

The batch-level objective is

$E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$4

where $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$5 is the number of original–perturbed pairs in the batch (&&&^^^^2query2^^^^&&&).

The paper characterizes this objective as “sequential contrastive learning” because it operates on representations derived from temporal embedding sequences rather than only on fixed handcrafted descriptors. Although the loss is applied to mean-pooled vectors, the encoder preceding pooling is explicitly sequence-based. The stated role of this objective is to enforce collapse between embeddings of original and perceptually indistinguishable perturbed versions while preserving separation from other musical items (&&&^^^^2query2^^^^&&&).

The training section further states that the contrastive loss implicitly encourages uniform embedding distribution, following Wang and Isola (2^^^^2query22query2^^^^), but the PAMT contribution lies in coupling that generic contrastive pressure to a psychoacoustically-conditioned sequence model rather than using contrastive learning alone (&&&^^^^2query2^^^^&&&).

4. Psychoacoustic conditioning and FiLM modulation

The principal mechanism for perceptual alignment is explicit psychoacoustic conditioning via Feature-wise Linear Modulation (FiLM). For Transformer layer $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$6, if $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$7 denotes the FFN sublayer output, the model computes layer-specific modulation vectors $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$8 from the conditioning vector $E_{\mathrm{mert}} \in \mathbb{R}^{T \times 768},$9: $T$^^^^2query2^^^^ These are then applied as

$T$^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^

The paper states that this allows the Transformer to alter its attention and feature transforms based on the psychoacoustic profile of the perturbation (&&&^^^^2query2^^^^&&&).

The intended effect is to teach the model to ignore distortions that fall below human masking thresholds, with Bark-band masking given as an example. In the paper’s account, this is what makes the representation perceptually aligned in a specifically psychoacoustic sense rather than merely invariant under arbitrary augmentation. The conditioning signal is therefore not an auxiliary label in the usual sense; it modulates internal computation at each layer (&&&^^^^2query2^^^^&&&).

This design also underlies the paper’s explanation of why PAMT is more perceptually aligned than prior models. The authors identify three interacting factors: psychoacoustic conditioning through FiLM, sequential contrastive learning that enforces temporal consistency, and the use of a frozen large-scale music encoder whose broad musical semantics are then specialized by the lightweight projection head. The empirical evidence cited for this alignment is the increase in Spearman $T$2 from baseline levels to $T$3, together with qualitative listening-test observations that nearest neighbors in PAMT’s embedding space more closely match human-judged similar excerpts (&&&^^^^2query2^^^^&&&).

A common misconception would be to interpret PAMT as simply a new distance function layered on top of MERT. The description in the paper is more specific: the framework learns a new representation space $T$4 through psychoacoustically-conditioned transformation of frozen MERT embeddings, and the distance function is then defined in that learned space (&&&^^^^2query2^^^^&&&).

5. Data, perturbations, and optimization protocol

The training corpus is assembled from FMA, GTZAN, and MTG-Jamendo, with ^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^,^^^^2query2query2query2^^^^ unique tracks from each dataset. The preprocessing section states that audio is resampled to ^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^6 kHz and segmented into ^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^-second excerpts, producing ^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^8,^^^^2query2query2query2^^^^ original–perturbed pairs. The split is 8^^^^2query2^^^^% train and 2^^^^2query2^^^^% test on unique tracks (&&&^^^^2query2^^^^&&&).

Six perturbation types are used: $T$5 noise, $T$6 noise, Bark-band noise, pitch shift, speed change, and dynamics compression. Their parameter ranges are randomized, with the paper referring to Table ^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^ for specifics. These perturbations define the positive pairs for contrastive learning and also form the basis of the perceptual adversaries used in downstream robustness evaluation (&&&^^^^2query2^^^^&&&).

Optimization uses AdamW with initial learning rate $T$7, weight decay $T$8, and batch size 32 original–perturbed pairs. The schedule is cosine-annealing learning rate with ^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^% warm-up. The contrastive temperature is fixed at $T$9, maximum training runs for ^^^^2(Liu et al., 5 Sep 2025) OR \2query2query2^^^^ epochs, and early stopping is triggered after ^^^^2(Liu et al., 5 Sep 2025) OR \2query2^^^^ epochs without validation-set Spearman improvement (&&&^^^^2query2^^^^&&&).

The paper’s regularization narrative emphasizes three points. First, the frozen MERT backbone is said to prevent catastrophic forgetting. Second, FiLM conditioning is said to focus learning on psychoacoustic invariances. Third, the contrastive loss is said to encourage a more uniform embedding distribution. Taken together, these design choices frame PAMT as a lightweight adaptation of a large pre-trained encoder rather than a fully end-to-end re-training procedure (&&&^^^^2query2^^^^&&&).

6. Evaluation methodology, results, and significance

Alignment with human judgments is measured through Spearman’s rank correlation coefficient between model-predicted distances and human 2AFC scores. For $E_{\mathrm{mert}}$^^^^2query2^^^^ test samples with ranks $E_{\mathrm{mert}}$^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^ and $E_{\mathrm{mert}}$2, the paper gives

$E_{\mathrm{mert}}$3

Reported correlation results are: best baseline, FAD with raw MERT embeddings, $E_{\mathrm{mert}}$4; MERT + MLP contrastive, $E_{\mathrm{mert}}$5; and PAMT (PCSCT), $E_{\mathrm{mert}}$6 (&&&^^^^2query2^^^^&&&).

For downstream adversarial evaluation, the paper considers two MIR tasks: Cover Song Identification (CSI), measured by mean Average Precision (mAP), and Music Genre Classification (MGC), measured by classification accuracy. Attacks are constrained by

$E_{\mathrm{mert}}$7

This definition makes the learned PAMT space itself the perceptual constraint for adversarial generation (&&&^^^^2query2^^^^&&&).

Under the union of attacks over all six perturbation types, the paper reports worst-case robust performance as follows.

Method	CSI Clean / Robust	MGC Clean / Robust
No Defense	^^^^2query2^^^^.845 / ^^^^2query2^^^^.^^^^2query22(Liu et al., 5 Sep 2025) OR \2^^^^	^^^^2query2^^^^.828 / ^^^^2query2^^^^.^^^^2query2
Standard AT ($E_{\mathrm{mert}}$8)	^^^^2query2^^^^.82^^^^2query2 / ^^^^2query2^^^^.3^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^5	^^^^2query2^^^^.8^^^^2query2 / ^^^^2query2^^^^.33^^^^2query2
MERT+MLP AT	^^^^2query2^^^^.83^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^ / ^^^^2query2^^^^.4^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^5	^^^^2query2^^^^.8^^^^2(Liu et al., 5 Sep 2025) OR \22^^^^ / ^^^^2query2^^^^.4^^^^2query22
PAMT AT (Ours)	^^^^2query2^^^^.835 / ^^^^2query2^^^^.535	^^^^2query2^^^^.8^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^8 / ^^^^2query2^^^^.465

The average robust-accuracy gain of PAMT over the best baseline is reported as $E_{\mathrm{mert}}$9. The paper also highlights task-specific gains relative to the best baseline, namely $P_{\mathrm{params}}$^^^^2query2^^^^ mAP in CSI and $P_{\mathrm{params}}$^^^^2(Liu et al., 5 Sep 2025) OR \2^^^^ accuracy in MGC (&&&^^^^2query2^^^^&&&).

The significance assigned to these results is twofold. First, PAMT improves agreement with subjective auditory similarity judgments compared with existing metrics. Second, embeddings learned under this criterion transfer to adversarially robust MIR tasks. The paper therefore frames PAMT as evidence that architecturally integrated psychoacoustic conditioning can improve both perceptual fidelity and robustness. This suggests a broader methodological implication: in music-domain adversarial settings, robustness may depend not only on perturbation budgets and optimization procedures, but also on whether the representation space is explicitly aligned with human auditory masking phenomena (&&&^^^^2query2^^^^&&&).