ModAn-MulSupCon: Metadata Supervised Contrastive Learning

• The paper introduces ModAn-MulSupCon, which harnesses readily available modality and anatomy metadata to guide contrastive pretraining, addressing expert label scarcity in medical imaging.
• It employs a Jaccard-weighted multi-label supervised contrastive objective to encode graded semantic similarity, thereby improving fine-tuning performance on downstream clinical classification tasks.
• Practical insights include its suitability for scenarios with limited expert annotations, while noting that its frozen encoder performance lags behind SimCLR and ImageNet baselines.

Searching arXiv for the specified paper and key related work to ground the article in current literature. arXiv search query: (Takaya et al., 26 Aug 2025) ModAn-MulSupCon is a medical-image representation-learning method that uses readily available metadata—specifically imaging modality and anatomical region—as supervision for contrastive pretraining. Introduced in "ModAn-MulSupCon: Modality-and Anatomy-Aware Multi-Label Supervised Contrastive Pretraining for Medical Imaging" (Takaya et al., 26 Aug 2025), it addresses the mismatch between the scarcity of expert disease annotations and the ubiquity of metadata in clinical archives. Its central idea is to encode modality and anatomy as a multi-hot target and to optimize a Jaccard-weighted multi-label supervised contrastive objective, with the aim of learning representations that transfer effectively to downstream clinical classification tasks when fine-tuning is permitted.

1. Problem formulation and conceptual position

The method is motivated by a persistent bottleneck in medical imaging: large-scale supervised pretraining is difficult because expert labels are scarce and expensive, whereas metadata such as modality and anatomical region are often already available in PACS archives or can be derived from study context without lesion-level or diagnosis-level annotation (Takaya et al., 26 Aug 2025). In this formulation, modality captures image-formation physics and appearance characteristics, while anatomical region captures organ- or body-part semantics and structural context.

Within the paper’s framing, this places ModAn-MulSupCon between three established pretraining paradigms. Relative to ImageNet pretraining, it aims to inject medical-imaging-specific priors absent from natural-image supervision. Relative to SimCLR, it uses available metadata instead of relying solely on instance discrimination. Relative to standard supervised contrastive learning, it is designed for the case where each image has multiple semantic attributes rather than a single class label.

A key conceptual claim is that semantic similarity in medical imaging is graded rather than binary. Two images may share both modality and anatomy, only one of them, or neither. This structure is not naturally represented by single-class supervised contrastive learning, which treats all same-class positives uniformly. ModAn-MulSupCon therefore uses metadata overlap as a continuous notion of similarity rather than collapsing each image into one rigid class identity.

2. Metadata encoding and contrastive objective

Each image’s modality and anatomy are represented as independent one-hot vectors and then concatenated into a multi-hot vector

$\mathbf y_i \in \{0,1\}^k.$

Because one modality bit and one anatomy bit are active, the resulting target is multi-hot rather than single-class (Takaya et al., 26 Aug 2025). The paper gives example categories such as CT, US, and MR for modality, and lung, abdomen, thyroid, brain, and knee for anatomy, but it does not report the exact total number of categories used in miniRIN.

The loss construction proceeds from the standard contrastive progression. For InfoNCE, an anchor has one positive view and all other in-batch views act as negatives. In standard supervised contrastive learning, the positive set is defined by exact class equality: $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$ ModAn-MulSupCon replaces this with multi-label overlap computed by Jaccard similarity: $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$

Positives are then defined by thresholding: $P_{\tau}(a)=\{\,p \in A(a)\mid w_{ap}\ge \tau\,\},$ and the loss becomes

$$\mathcal{L}_{\mathrm{MulSupCon}} = -\frac{1}{\lvert P_{\tau}(a)\rvert} \sum_{p\in P_{\tau}(a)} w_{ap}\, \log \frac{\exp\!\bigl(\mathbf z_a^{\top}\mathbf z_p/T\bigr)} {\displaystyle\sum_{k\in A(a)} \exp\!\bigl(\mathbf z_a^{\top}\mathbf z_k/T\bigr)}.$$

Two points are structurally important. First, positive eligibility is thresholded, not unconditional. Second, contributions within the positive set are continuously weighted by metadata overlap. The method therefore differs from standard supervised contrastive learning in both positive-set definition and positive weighting. It also differs from SimCLR because it uses cross-sample semantic supervision rather than only paired augmentations of the same instance.

A common misconception is that ModAn-MulSupCon is simply supervised contrastive learning with two labels. The paper’s formulation is stricter than that description: it uses a thresholded multi-label positive set together with Jaccard-based soft weighting, which is intended to represent partial semantic overlap rather than exact label identity.

3. Architecture, pretraining corpus, and optimization pipeline

The encoder is a ResNet-18 whose first convolution is modified for single-channel input, using a $3\times 3$ kernel and stride $1$; the remaining layers are unchanged (Takaya et al., 26 Aug 2025). The paper states that a shared encoder maps images to embeddings, but it does not specify the projection-head architecture or dimensionality.

Pretraining is performed on a subset of RadImageNet. RadImageNet contains about 1.3 million medical images annotated with 165 disease and normal classes, along with modality and anatomical metadata. Because full training would require “tens of GPU-days per run,” the paper constructs miniRIN by randomly sampling up to 100 images per class, using all images for classes with fewer than 100 samples, yielding 16,222 images.

Contrastive pretraining uses a SimCLR-style TwoCropTransform with the following augmentations: RandomResizedCrop with scale $[0.2,1.0]$, RandomHorizontalFlip, ColorJitter with probability $0.8$, and RandomGrayscale with probability $0.2$. For ModAn-MulSupCon, the pretraining hyperparameters are temperature $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$0, threshold $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$1, SGD, learning rate $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$2, momentum $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$3, weight decay $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$4, 1000 epochs, and batch size 256. The miniRINSimCLR baseline uses NT-Xent with temperature $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$5, SGD, learning rate $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$6, momentum $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$7, weight decay $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$8, 1000 epochs, and batch size 256. The miniRIN165 supervised baseline uses cross-entropy with Adam default settings, 1000 epochs, and batch size 256. ImageNet uses off-the-shelf pretrained weights, and Scratch uses random initialization.

For downstream transfer, the same recipe is used for both full fine-tuning and linear probing with frozen encoder: 10 epochs, batch size 32, Adam, learning rate $P(a)=\{\,p \in A(a)\mid y_p=y_a\,\}.$9, and StepLR with decay at epoch 5 and factor $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$0. The reported compute environment is PyTorch 2.1.2 on a single NVIDIA Quadro RTX 8000 (48 GB), with wall-clock times of about 10 min/epoch for pretraining, 0.7 sec/epoch for fine-tuning, and 0.1 sec/epoch for linear probing.

Several implementation details remain unspecified in the paper excerpt. In particular, it does not report the exact modality/anatomy category counts used in miniRIN, image resizing resolution, normalization parameters, or modality-specific intensity preprocessing.

4. Downstream evaluation design

Transfer performance is evaluated on three public binary classification tasks (Takaya et al., 26 Aug 2025).

Dataset	Modality and task	Size
MRNet-ACL	Knee MRI; ACL tear vs. intact ligament	1021
Breast US	Ultrasound; benign vs. malignant lesion	779
Thyroid US	Ultrasound; benign vs. malignant nodule	349

The reported class distributions are 569/452 for MRNet-ACL, 569/210 for Breast US, and 288/61 for Thyroid US. The primary metric is AUC. All methods are evaluated with the same fixed 5-fold partition for each downstream dataset, repeated 10 times to characterize training stochasticity. For each repeat, fold-wise AUCs are averaged into one summary AUC, so each method yields 10 summary AUCs. Statistical testing uses a paired Wilcoxon signed-rank test, two-sided, with significance level $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$1.

The comparison covers five initializations: ModAn-MulSupCon, miniRINSimCLR, miniRIN165, ImageNet, and Scratch. Two transfer regimes are studied. In full fine-tuning, all encoder layers are trainable, so the experiment measures the value of the initialization when adaptation is allowed. In frozen-encoder linear probing, only a linear classifier is trained, so the experiment instead measures the linear separability of the pretrained representation.

5. Empirical results and representation behavior

Under full fine-tuning, ModAn-MulSupCon achieves the best mean AUC on MRNet-ACL and Thyroid, and ranks second on Breast US (Takaya et al., 26 Aug 2025). On ACL, it obtains 0.964, compared with 0.951 for miniRINSimCLR, 0.630 for miniRIN165, 0.876 for ImageNet, and 0.907 for Scratch; the paper states that it surpasses all baselines with $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$2. On Thyroid, it obtains 0.763, compared with 0.685, 0.558, 0.696, and 0.677 respectively, again surpassing all baselines with $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$3. On Breast US, it reaches 0.926, while miniRINSimCLR attains 0.940; however, the reported $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$4-value for that comparison is 1.000, so the numerical difference is not statistically significant.

This establishes the paper’s main empirical claim: the method is strongest as a fine-tuning initialization. The corresponding conclusion is not that the representation is universally superior, but that it is especially effective when downstream task adaptation is feasible.

The ranking changes substantially under linear probing. On ACL, ModAn-MulSupCon yields 0.681, while miniRINSimCLR reaches 0.844 and ImageNet 0.824. On Breast US, ModAn-MulSupCon yields 0.717, while miniRINSimCLR reaches 0.805 and ImageNet 0.781. On Thyroid, ModAn-MulSupCon yields 0.639, while miniRINSimCLR reaches 0.657; the paper notes that on Thyroid it is significantly better than Scratch only. The authors therefore conclude that SimCLR performs best on all three tasks when the encoder is frozen, with ImageNet also strong on ACL and Breast.

The representation analysis is consistent with this pattern. UMAP visualizations of penultimate-layer features indicate that both ModAn-MulSupCon and SimCLR produce clusters roughly organized by modality and anatomy, but SimCLR shows higher linear separability when frozen, whereas ModAn-MulSupCon shows more entangled clusters. The paper interprets this as evidence that metadata-driven pairwise weighting creates “soft positives” that encode hierarchical proximity, producing features that are less linearly separable in frozen form but more adaptable during end-to-end fine-tuning.

A further methodological point is that the result tables report only mean AUC and Wilcoxon $$w_{ap} = \mathrm{Jaccard}(\mathbf y_a,\mathbf y_p) = \frac{\lvert S(a)\cap S(p)\rvert}{\lvert S(a)\cup S(p)\rvert}, \qquad S(i)=\{\,m\mid (\mathbf y_i)_m=1\,\}.$$5-values; confidence intervals and standard deviations are not reported.

6. Practical use, assumptions, and limitations

The paper’s practical recommendation is explicit (Takaya et al., 26 Aug 2025). ModAn-MulSupCon should be preferred when full fine-tuning is feasible, expert disease labels for pretraining are unavailable, modality/anatomy metadata are available and reasonably consistent, and downstream clinical datasets are label-scarce. By contrast, SimCLR or ImageNet are preferable when the encoder must remain frozen for lightweight deployment or linear-probe-style transfer.

The method’s scalability claim is tied to the nature of the supervision signal rather than to full-scale experimental validation. Modality and anatomy metadata are described as broadly available, cheap to collect, semantically relevant, and potentially scalable across institutions and datasets. A plausible implication is that the method addresses a label-efficiency problem more than a raw-capacity problem: it substitutes weak but ubiquitous supervision for scarce disease labels. However, the paper validates this claim only on miniRIN, not on full RadImageNet scale.

Several limitations are stated directly. Scalability assessment is limited to miniRIN. The downstream evaluation is restricted in modality diversity, consisting of two ultrasound tasks and one knee MRI task. Only a ResNet-18 backbone is evaluated; no Vision Transformers or 3D backbones are studied. Frozen-encoder performance is weak relative to SimCLR and ImageNet. Metadata granularity may be too coarse, particularly for breast ultrasound, where device variability, speckle noise, acquisition protocols, and subtle peri-lesional textures may matter. The paper also notes that ultrasound may benefit from richer metadata such as device type, probe type, acquisition view, and laterality.

Several frequently expected analyses are absent. The excerpt does not report modality-only versus anatomy-only ablations, nor systematic sensitivity studies over Jaccard weighting, temperature, metadata quality, dataset size, projection head, or augmentation variants. These omissions are significant for readers interested in isolating which component drives the reported gains.

The future directions proposed in the paper follow directly from these limitations: scaling beyond miniRIN, enriching metadata with device/probe/view/laterality, combining metadata supervision with instance-level contrastive objectives, evaluating ViTs and 3D encoders, testing robustness across institutions and modalities, and expanding to broader clinical classification settings. Collectively, these directions suggest that ModAn-MulSupCon is best understood not as a final architecture, but as a metadata-supervised contrastive pretraining principle whose effectiveness depends on downstream adaptation and on the fidelity of the metadata used to define semantic overlap.