Self-Supervised Multi-Class-Token Multitask (MT²)

• The paper introduces MT², a self-supervised framework that uses dual class tokens to separately optimize contrastive and equivariant tasks on audio spectrogram inputs.
• MT² leverages a shared ViT-1D backbone that jointly minimizes NT-Xent and CPSD losses, balancing timbre-sensitive tagging with pitch/transposition awareness.
• The model achieves competitive results across tagging, key estimation, instrument recognition, and pitch tasks, outperforming baselines with 18× fewer parameters.

Searching arXiv for the specified paper to ground the article in the published record. Self-Supervised Multi-Class-Token Multitask, abbreviated MT², is a multitask self-supervised learning framework for music information retrieval (MIR) in which a single Vision Transformer with 1-D spectrogram patches (ViT-1D) is trained jointly with two specialized class tokens, each aligned to a different pretext objective. It was introduced in "Multi-Class-Token Transformer for Multitask Self-supervised Music Information Retrieval" (Kong et al., 17 Jul 2025) to address a specific MIR SSL trade-off: contrastive learning is described as more effective on tagging tasks such as instrument recognition but less effective on structured prediction such as tonality estimation, whereas equivariant learning can match supervised methods on the task for which it is designed but does not generalize well across tasks. MT² combines both objectives in one shared backbone, with the stated aim of unifying timbre-sensitive and harmony-sensitive representations in a compact model.

1. Conceptual basis and problem setting

MT² is defined by the joint use of two independent learnable class tokens prepended to a common patch-token sequence and processed by the same Transformer layers. One token is specialized for contrastive learning through normalized temperature-scaled cross-entropy (NT-Xent), while the other is specialized for equivariant learning through a cross-power spectral density (CPSD) loss over the circle of fifths. The model therefore uses multiple class tokens not as redundant summary vectors, but as objective-specific latent readouts sharing a single encoder (Kong et al., 17 Jul 2025).

The central design premise is that MIR contains heterogeneous downstream targets. Tagging and instrument recognition reward representations that preserve timbral information, whereas key, pitch, and related structured tasks benefit from representations constrained by musically meaningful transformations such as pitch transposition. MT² is presented as a best-of-both-worlds construction: the contrastive token targets invariance useful for semantic similarity, and the equivariant token targets transposition-aware structure. A plausible implication is that the architecture treats self-supervision not as a single universal objective, but as a coordinated family of objectives with partially distinct sufficient statistics.

A common misunderstanding is to read MT² as a multi-encoder system. It is not. Both class tokens share the same ViT-1D weights and are optimized through the same model, so multitask behavior is induced by token specialization and head-specific losses rather than by separate backbones.

2. ViT-1D architecture and input representations

The backbone operates on 4-second audio clips at 16 kHz. Inputs are converted either to a 128-bin mel-spectrogram with 31.5 Hz frame rate and $T=126$ frames, or to a constant-Q transform (CQT) with $Q=12$ bins per octave over $J=8$ octaves, yielding 84 bins and again $T=126$. Each time frame is treated as one 1-D patch whose size equals the number of frequency bins. A learnable linear layer projects each patch $x_t \in \mathbb{R}^F$ to an embedding of dimension $d_e=192$. Positional information is injected through 2-D time-by-frequency learnable or fixed embeddings added to each token. The Transformer body follows a standard ViT with 12 Transformer blocks, 3 attention heads per block, model dimension 192, and MLPs corresponding to the “small” ViT configuration (Kong et al., 17 Jul 2025).

Component	Specification	Role
Audio input	4-second clips at 16 kHz	SSL pretraining input
Spectral views	128-bin mel or 84-bin CQT, $T=126$	Objective-specific representations
Patching	One time frame per 1-D patch	Tokenization
Embedding	Linear projection to $d_e=192$	Patch-token formation
Transformer	12 blocks, 3 heads, dimension 192	Shared encoder
Output length	$T$ plus prepended class tokens	Joint processing of patches and summaries

Two independent class tokens are prepended to the patch sequence. The contrastive token $c_{\text{in}}^{\text{cont}} \in \mathbb{R}^{192}$ is initialized as a learnable vector plus the average of the mel-spectrogram patch embeddings. After the Transformer, its output $Q=12$0 is linearly projected to $Q=12$1. The equivariant token $Q=12$2 is initialized as a learnable vector plus the average of the CQT patch embeddings. After the Transformer, $Q=12$3 is added residually to the average CQT embedding, projected to dimension 84, octave-pooled to dimension 12, and softmaxed to obtain $Q=12$4, a 12-class probability vector.

The total parameter count is approximately 5.3 million. This compactness is central to the later comparison with MERT.

3. Self-supervised objectives

The contrastive branch uses NT-Xent on paired segments from the same track. For two segments $Q=12$5 from the same track and negatives $Q=12$6 in the batch, the loss is

$Q=12$7

where $Q=12$8 is cosine similarity and $Q=12$9 (Kong et al., 17 Jul 2025).

The equivariant branch uses a CPSD loss defined on the 12-point discrete Fourier transform of the 12-way output vector. Writing $J=8$0, for a pitch shift $J=8$1 and fixed frequency index $J=8$2, the pairwise discrepancy is

$J=8$3

Given two unshifted segments $J=8$4 and a third segment $J=8$5 artificially transposed by $J=8$6 semitones, the equivariant loss is

$J=8$7

This objective encodes equivariance over the circle of fifths. The use of $J=8$8 and a 12-point spectrum places the output space in direct correspondence with pitch-class structure, rather than with generic augmentation invariance.

The multitask objective is the unweighted sum of the two losses:

$J=8$9

Backpropagation updates all shared Transformer weights and both class-token heads simultaneously. One important consequence is that MT² does not rely on explicit loss-balancing coefficients; the paper states that no extra weighting is needed.

4. Training protocol, probing, and inference

Pretraining uses 100 k commercial tracks for 600 epochs with batch size 128 pairs and a learning rate schedule of $T=126$0 under cosine decay. For the contrastive objective, two disjoint 4-second segments $T=126$1 are sampled from the same track. For the equivariant objective, a segment $T=126$2 is generated by random frequency transposition of $T=126$3 by $T=126$4 semitones. These augmentations are paired with the objective-specific mel and CQT views and processed in the same forward pass (Kong et al., 17 Jul 2025).

After SSL pretraining, either class token can be probed separately. More commonly, the two 192-dimensional outputs are averaged:

$T=126$5

A single linear layer is then trained per downstream task while the backbone remains frozen. For local tasks, the probing uses sequence tokens rather than only class tokens. The reported empirical motivation for token averaging is that it yields better or at least as good performance on several tasks and reflects the complementary nature of the two learned representations.

The implementation notes specify the use of 4-second mel-spectrograms with 128 bins at 31.5 Hz and CQTs with 12 bins per octave over 8 octaves, class-token initialization as a learnable vector plus the mean of the relevant patch embeddings, a “small” ViT with 12 layers and 3 heads at $T=126$6, simple loss summation without weighting, and frozen-backbone linear probing. Code and weights are reported as available at https://github.com/deezer/mt2.

5. Benchmark design and reported results

The downstream benchmark separates global, time-invariant tasks from local, frame-level tasks. Global tasks are tagging on MagnaTagATune with 50 tags, evaluated by macro mAP and ROC-AUC; instrument recognition on TinySOL with 14 classes, evaluated by top-1 accuracy; key estimation from FMAKv2 to GiantSteps with 24 pitch classes, evaluated by MIREX score; and pitch estimation on TinySOL with 82 semitones, evaluated by top-1 accuracy. Local tasks are beat tracking from Ballroom to GTZAN Rhythm, evaluated by $T=126$7 with a $T=126$8 ms tolerance, and chord estimation on RWC-POP+SWD with 25 classes, evaluated by frame accuracy (Kong et al., 17 Jul 2025).

The selected results below are those explicitly reported for class-token probing or, for beat tracking, sequence-token probing:

Task	MT² result	Comparison
Tagging	MT²-cont $T=126$9; MT²-avg $x_t \in \mathbb{R}^F$0	ViT-Mel $x_t \in \mathbb{R}^F$1; MERT $x_t \in \mathbb{R}^F$2
Key estimation	MT²-equiv $x_t \in \mathbb{R}^F$3; MT²-avg $x_t \in \mathbb{R}^F$4	ViT-CQT $x_t \in \mathbb{R}^F$5; MERT $x_t \in \mathbb{R}^F$6
Instrument recognition	MT²-cont $x_t \in \mathbb{R}^F$7; MT²-avg $x_t \in \mathbb{R}^F$8	ViT-Mel $x_t \in \mathbb{R}^F$9; MERT $d_e=192$0
Pitch estimation	MT²-equiv $d_e=192$1; MT²-avg $d_e=192$2	ViT-CQT $d_e=192$3; MERT $d_e=192$4
Beat tracking	MT² $d_e=192$5	ViT-Mel $d_e=192$6; ViT-CQT $d_e=192$7; MERT $d_e=192$8
Chord estimation	MT² $d_e=192$9	ViT-Mel $T=126$0; ViT-CQT $T=126$1; MERT $T=126$2

Several patterns are explicit in these numbers. The contrastive token is particularly strong on tagging and instrument recognition. The equivariant token is strong on pitch-related tasks. Averaging the two class tokens improves or preserves performance on several tasks, most notably pitch estimation, where MT²-avg reaches 0.990. Beat tracking is the primary task on which MERT remains ahead, with 0.786 versus MT² at 0.698.

The paper’s overall summary states that MT² outperforms both single-task ViT-1D baselines on all tasks and outperforms MERT on all tasks except beat tracking, while using 18× fewer parameters. At the same time, the selected per-task results show that performance depends on the probing choice and comparison point, especially for key and chord estimation. This suggests that the strongest interpretation is not uniform dominance of every MT² readout on every metric, but the availability of a single compact backbone supporting multiple useful readouts.

6. Interpretation, scope, and limitations

MT² is notable for treating class tokens as task-specialized representational interfaces rather than as interchangeable summary embeddings. The contrastive token is associated with timbre-specific details, while the equivariant token is associated with timbre-invariant harmony; their average is reported to exploit the complementary nature of these representations (Kong et al., 17 Jul 2025). This suggests that multi-class-token design can be used to preserve objective-specific inductive biases inside a single encoder, instead of forcing all SSL behavior into a single pooled representation.

The framework also sharpens the distinction between invariance and equivariance in MIR. Contrastive learning favors agreement between different excerpts of the same track and is therefore effective for semantic tagging. CPSD equivariance enforces predictable transformation under pitch transposition and is therefore aligned with key and pitch structure. MT² does not collapse these requirements into one compromise objective; it allocates separate output tokens while keeping the encoder shared.

Several practical boundaries are equally clear. The model is benchmarked with frozen-feature linear probing rather than end-to-end finetuning. The inference recipe privileges token averaging for many tasks, but local frame-level tasks use sequence-token probes. Beat tracking remains a case where MERT performs better. In addition, the selected results for key and chord estimation show that averaging or multitask coupling does not automatically surpass every single-task baseline in every reported configuration.

Within those boundaries, MT² establishes a specific design pattern for self-supervised MIR: a compact ViT-1D with two jointly trained class tokens, an unweighted sum of NT-Xent and CPSD losses, and downstream probing that can target either token separately or their average. The reported outcome is a versatile SSL model that, with approximately 5.3 million parameters, exceeds MERT on all reported tasks except beat tracking while using 18× fewer parameters.