Patch-Wise Cross-Modality Contrastive Learning

• PCCL is a representation-learning paradigm that aligns local units (patches, regions, etc.) across modalities using objectives like InfoNCE.
• It enables fine-grained cross-modal correspondence through explicit or implicit contrastive objectives, adaptable to various architectures and tasks.
• Empirical studies demonstrate PCCL’s effectiveness in improving classification, segmentation, and registration performance by leveraging local structural alignment.

Searching arXiv for the primary paper and closely related patch-wise cross-modality contrastive learning work. arxiv_search.search(query="(Liu et al., 2021) CMA-CLIP Cross-Modality Attention CLIP for Image-Text Classification", max_results=5) arxiv_search.search(query="Patch-wise cross-modality contrastive learning arXiv image text patch-wise contrastive", max_results=10) arxiv_search.search(query="(Zhang et al., 3 Dec 2025) Cross-Stain Contrastive Learning (Liu et al., 2021) Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning (Dey et al., 2022) ContraReg (Shi et al., 2021) Dense Contrastive Visual-Linguistic Pretraining", max_results=10) Patch-wise Cross-modality Contrastive Learning (PCCL) denotes a family of representation-learning methods that align local units across modalities rather than relying solely on global image- or sample-level similarity. The aligned unit may be an image patch, a detector region, a local 3D neighborhood, a spatial location in an intermediate feature map, or even a temporally localized window. Across applications, PCCL is used to learn embeddings that are simultaneously correspondence-sensitive at fine granularity and transferable across modality gaps, with instantiations in semantic segmentation, visual-language pretraining, deformable registration, multimodal classification, computational pathology, image translation, visible-infrared person re-identification, and neural-stimulus alignment (Liu et al., 2021, Liu et al., 2021, Dey et al., 2022, Zhang et al., 3 Dec 2025).

1. Conceptual scope and definition

PCCL is unified by a local alignment premise: matched cross-modal units should be close in an embedding space, while mismatched units should be separated. In the canonical form, this is implemented with a patch-level contrastive objective, typically InfoNCE, over paired local features. The “cross-modality” component is broad. It may refer to image and text, H&E and IHC stains, T1- and T2-weighted MRI, visible and infrared imagery, source and target domains in domain adaptation, or neural activity and natural-movie features treated as distinct streams (Shi et al., 2021, Zhang et al., 3 Dec 2025, Dey et al., 2022, Qian et al., 2023, Wang et al., 2023).

The “patch-wise” component is likewise modality- and architecture-dependent. In CMA-CLIP, the local units are ViT image patches and text tokens, and the method realizes fine-grained patch–token interactions through self-attention over a concatenated multimodal sequence rather than through an explicit patch-level contrastive loss (Liu et al., 2021). In computational pathology, Cross-Stain Patch-wise Alignment (CPA) uses non-overlapping $256\times256$ patches extracted at $10\times$ magnification from slide-level co-registered whole-slide images (Zhang et al., 3 Dec 2025). In semantic segmentation domain adaptation, local units are $64\times32$ image patches whose semantic similarity is measured by a spatial pyramid of label histograms (Liu et al., 2021). In ContraReg, they are multi-scale local 3D patches induced by convolutional receptive fields and indexed within feature maps under the current deformable warp (Dey et al., 2022). In DCVLP, they are bottom-up detector regions rather than grid patches (Shi et al., 2021). In the representational-drift study, the effective “patch” is a single $33$ ms temporal window, with no spatial partitioning reported (Wang et al., 2023).

A central distinction within the literature is between explicit and implicit PCCL. Explicit PCCL introduces a patch-level contrastive term directly into optimization, as in CPA for H&E↔IHC alignment, label-structured patch matching for segmentation adaptation, or multi-scale local InfoNCE for deformable registration (Zhang et al., 3 Dec 2025, Liu et al., 2021, Dey et al., 2022). Implicit PCCL uses patch-wise or region-wise cross-modal interaction without a dedicated patch-level contrastive objective. CMA-CLIP exemplifies this latter case: it refines CLIP’s coarse alignment into patch–token granularity through sequence-wise self-attention, but its downstream training uses supervised classification losses rather than an explicit patch-level InfoNCE term (Liu et al., 2021).

2. Core objectives and correspondence mechanisms

The standard PCCL objective is a local InfoNCE loss over matched cross-modal units. In CSCL, for an H&E patch $z$, a corresponding patch $z_c^+$ from IHC stain $c$, negatives $z^{-}$, similarity $\operatorname{sim}(\cdot,\cdot)$, and temperature $\tau$, the patch-level term is

$10\times$0

CSCL further weights this term with an adaptive schedule that transitions from uniform weighting to similarity-aware weighting in order to mitigate residual misalignment after slide-level registration (Zhang et al., 3 Dec 2025).

How positive pairs are defined varies substantially by task. In CSCL, positives are same-location cross-stain patches induced by identical tiling grids on co-registered slides (Zhang et al., 3 Dec 2025). In ContraReg, the positive for a local feature at spatial index $10\times$1 in modality $10\times$2 is the feature at the same index $10\times$3 in modality $10\times$4 after warping, yielding a symmetric registration similarity

$10\times$5

with negatives sampled from other spatial indices in the moved feature map (Dey et al., 2022). In semantic segmentation adaptation, positives are not same-location pairs but structurally similar label patches across domains: patches with semantic disparity $10\times$6 are positives, those with $10\times$7 are negatives, and intermediate pairs are ignored; the reported best thresholds are $10\times$8 and $10\times$9 with patch size $64\times32$0 and $64\times32$1 (Liu et al., 2021).

Several papers replace exact geometric matching with feature-space alignment under preserved spatial correspondence. In image-to-image translation, the source patch embedding $64\times32$2 and translated patch embedding $64\times32$3 at the same spatial location form the positive pair. The paper introduces Decoupled Contrastive Loss (DCE),

$64\times32$4

which removes the positive term from the denominator to avoid negative-positive coupling, and combines it with Semantic Relation Consistency (SRC), defined via Jensen–Shannon divergence between intra-image similarity distributions $64\times32$5 and $64\times32$6 (Jung et al., 2022).

Other PCCL variants weaken or alter the notion of contrastive supervision. DCVLP builds positives from the same RoI index across QueryNet and KeyNet, where each region embedding is already contextualized by cross-attention to text; negatives are both intra-image other regions and in-batch regions from other images (Shi et al., 2021). The representational-drift framework uses co-occurrence within a single-frame window at $64\times32$7 Hz as weak supervision, treating neural responses and stimulus-derived features at the same frame index $64\times32$8 as positives and other frames $64\times32$9 as negatives (Wang et al., 2023). By contrast, PMCM aligns visible and infrared learning signals through KL divergence, center-to-center loss, sample-to-sample matching, and triplet loss rather than an explicit patch-level InfoNCE; this is patch-wise cross-modality learning, but not a pure InfoNCE-style PCCL formulation (Qian et al., 2023).

3. Architectural realizations

PCCL is not tied to a single backbone family. Its implementations range from transformer fusion modules to CNN-based projection heads, MIL pipelines, and registration networks.

Instantiation	Local unit	Cross-modal mechanism
CMA-CLIP (Liu et al., 2021)	ViT image patches and text tokens	Sequence-wise self-attention over concatenated multimodal sequence; modality-wise task gate
CSCL (Zhang et al., 3 Dec 2025)	Co-registered WSI patches	CPA patch-wise InfoNCE, CAF along stain dimension, CGA on slide embeddings
Domain adaptation (Liu et al., 2021)	Image patches from intermediate feature maps	Multi-level projection heads with structurally defined positives and negatives
ContraReg (Dey et al., 2022)	Multi-scale local 3D patches	Symmetric cross-modality InfoNCE under current deformable warp
DCVLP (Shi et al., 2021)	Detector RoIs	Region-level contrastive learning on cross-attended visual embeddings
PMCM (Qian et al., 2023)	Input-space visible/IR square patches	Patch-mixed image construction with KL, triplet, and center losses

In CMA-CLIP, CLIP’s ViT-B/32 image encoder yields $33$0 patches for a $33$1 input, while the text encoder outputs full token sequences. A $33$2-layer, $33$3-width, $33$4-head transformer encoder is applied to the concatenated image-patch and text-token sequence, so cross-modality interaction is realized by self-attention itself. Refined image and text $33$5 tokens, $33$6 and $33$7, are then fused by a task-specific keyless modality gate

$33$8

before task-specific MLP classification (Liu et al., 2021).

CSCL uses a different decomposition. Stage 1 freezes a CONCH-pretrained ViT that outputs $33$9-dimensional patch embeddings for all stains and trains only a lightweight adapter on the H&E path, aligning H&E patches to IHC patches with CPA. Stage 2 freezes encoder and adapter, applies Cross-Stain Attention Fusion (CAF) along the stain dimension at each spatial location, and then aggregates stain-specific patch features with ABMIL. Slide embeddings are further aligned with Cross-Stain Global Alignment (CGA), another InfoNCE-style objective at the slide level (Zhang et al., 3 Dec 2025).

In domain adaptation for segmentation, DeepLabV2 with a ResNet-101 encoder is augmented with multi-level average pooling and two-layer MLP projection heads attached to intermediate stages. The method exploits semantic structure in label space by comparing spatial-pyramid histograms over patches, thereby defining positives and negatives without relying on photometric similarity (Liu et al., 2021). ContraReg instead uses modality-specific autoencoders to extract multi-scale features, per-scale $z$0-layer $z$1-wide ReLU MLP projectors, and a VoxelMorph-style 3D U-Net that predicts a stationary velocity field $z$2 for diffeomorphic registration (Dey et al., 2022). DCVLP relies on Faster R-CNN region proposals projected into a multimodal transformer embedding space, then contrasts contextualized region embeddings rather than raw RoI features (Shi et al., 2021).

This architectural diversity suggests that PCCL is best understood as a training principle rather than a fixed model class. A plausible implication is that the decisive design choices are the definition of local correspondence, the hardness and quality of negatives, and the way local alignment is connected to downstream aggregation.

4. Supervision regimes and training protocols

PCCL spans supervised, weakly supervised, and self-supervised settings. In semantic segmentation adaptation, the full objective combines source cross-entropy, target cross-entropy when available, Charbonnier-weighted entropy minimization on unlabeled target data, pseudo-label self-training, and the patch-wise contrastive loss: $z$3 with $z$4, $z$5, and $z$6 for ground-truth target labels versus $z$7 for pseudo target labels (Liu et al., 2021). The procedure first trains with $z$8 to produce pseudo labels, then reinitializes and retrains with $z$9.

CSCL adopts a staged pretraining regime: adapter-only training with $z_c^+$0, followed by frozen encoder+adapter training of CAF and MIL with $z_c^+$1, optionally combined with supervised downstream task loss during fine-tuning. A notable property is that multi-stain inputs are required only during pretraining; inference uses H&E only, with frozen ViT plus trained adapter and ABMIL aggregation (Zhang et al., 3 Dec 2025). CMA-CLIP also uses staged optimization, first warming up with CLIP encoders frozen, then fine-tuning end-to-end, and finally freezing encoders plus sequence-wise attention while tuning task-specific modality gates and MLP heads for multi-task classification (Liu et al., 2021).

In ContraReg, supervision is entirely unsupervised at registration time. The autoencoders are first pretrained with an $z_c^+$2 plus local NCC reconstruction loss on $z_c^+$3 crops, then frozen. Registration training jointly optimizes the registration U-Net and projector MLPs with the symmetric patch-wise contrastive similarity and the velocity regularizer $z_c^+$4, using Adam with learning rate $z_c^+$5, temperature $z_c^+$6, $z_c^+$7 negatives per anchor, and $z_c^+$8 iterations (Dey et al., 2022).

The representational-drift work uses a two-phase scheme of single-modality pretraining followed by cross-modality alignment. Neural activity is first contrasted across pseudomice at the same frame $z_c^+$9, stimulus streams are contrasted across scene and optic flow at the same frame, and then all modality pairs are contrasted using frame-wise co-occurrence as the positive relation. The pretrained model is frozen for cross-session drift analysis (Wang et al., 2023). DCVLP augments dense region contrastive learning with two perturbation strategies—Mask Perturbation and Intra-/Inter-Adversarial Perturbation—and optimizes the unweighted sum $c$0 (Shi et al., 2021).

A common misconception is that PCCL necessarily implies fully self-supervised training. The literature shows instead that patch-wise contrastive alignment is often one term in a composite objective that may include cross-entropy, triplet loss, entropy minimization, adversarial loss, registration regularization, or MIL aggregation (Liu et al., 2021, Jung et al., 2022, Qian et al., 2023, Dey et al., 2022).

5. Empirical behavior across application areas

The reported empirical gains indicate that local cross-modal alignment is beneficial when global alignment is too coarse, when modalities have unequal reliability, or when fine structural correspondence is decisive.

Setting	Reported result	Source
MRWPA multi-task classification	CMA-CLIP average recall at 90% precision $c$1 vs raw CLIP $c$2 and fine-tuned CLIP $c$3; abstract reports average improvement of $c$4 recall at fixed precision	(Liu et al., 2021)
Fashion-Gen	CMA-CLIP $c$5 accuracy vs KaleidoBERT $c$6	(Liu et al., 2021)
Food101	CMA-CLIP $c$7 accuracy; higher than ViT $c$8, BERT $c$9, CLIP $z^{-}$0, MMBT $z^{-}$1	(Liu et al., 2021)
GTA5→Cityscapes SSDA	PCCL reaches $z^{-}$2 mIoU with $z^{-}$3 labels vs FDA $z^{-}$4, ASS $z^{-}$5, Universal $z^{-}$6	(Liu et al., 2021)
SYNTHIA→Cityscapes SSDA	PCCL reaches $z^{-}$7 mIoU with $z^{-}$8 labels vs FDA $z^{-}$9, ASS $\operatorname{sim}(\cdot,\cdot)$0, Universal $\operatorname{sim}(\cdot,\cdot)$1	(Liu et al., 2021)
Neonatal T1–T2 MRI registration	mCR Dice $\operatorname{sim}(\cdot,\cdot)$2 vs MI $\operatorname{sim}(\cdot,\cdot)$3, LocalMI $\operatorname{sim}(\cdot,\cdot)$4, MIND $\operatorname{sim}(\cdot,\cdot)$5 for $\operatorname{sim}(\cdot,\cdot)$6	(Dey et al., 2022)
VI-ReID on SYSU-MM01	PMCM all-search single-shot Rank-1 $\operatorname{sim}(\cdot,\cdot)$7, mAP $\operatorname{sim}(\cdot,\cdot)$8	(Qian et al., 2023)
VQA with UNITER backbone	DCVLP_MASK $\operatorname{sim}(\cdot,\cdot)$9 on test-dev vs UNITER-Base $\tau$0	(Shi et al., 2021)

Ablation studies provide much of the mechanistic evidence. On MRWPA, removing modality-wise attention drops average recall at $\tau$1 precision from $\tau$2 to $\tau$3, and removing both modality-wise attention and sequence-wise attention drops it further to $\tau$4, supporting the claim that robustness to noisy modalities and patch–token alignment are complementary (Liu et al., 2021). In semantic segmentation adaptation, pyramid label disparity outperforms exact Hamming label matching, and the best patch size is $\tau$5, which indicates that spatial composition and coarse layout matter in defining local positives (Liu et al., 2021). In CSCL, removing CPA, CAF, or CGA degrades performance across TransMIL, DSMIL, and ABMIL backbones; for example, on BCNB IHC $\tau$6 with TransMIL, performance rises from $\tau$7 at baseline to $\tau$8 with $\tau$9CPA$10\times$00CAF$10\times$01CGA (Zhang et al., 3 Dec 2025).

The empirical record also clarifies that PCCL is not only about classification accuracy. ContraReg evaluates both alignment accuracy and deformation quality, reporting Dice, Dice30, percentage of folding voxels, and $10\times$02, thereby framing local contrastive alignment as an inter-domain similarity metric for optimization rather than merely a representation regularizer (Dey et al., 2022). The representational-drift work uses cross-modality contrastive embeddings to study local geometry: session-transfer decoding shows up to approximately $10\times$03 accuracy drop, while local $10\times$04-NN structure deteriorates more strongly than global smoothness, especially for fast optic-flow features whose autocorrelation scale is $10\times$05–$10\times$06 ms (Wang et al., 2023).

6. Limitations, misconceptions, and open directions

A persistent limitation is correspondence quality. CSCL depends on paired multi-stain slides that are only slide-level aligned; identical tiling induces patch correspondences, but residual deformations remain, motivating adaptive weighting and prompting explicit future directions such as neighborhood-based positive expansion, deformation-invariant matching, unpaired cross-stain learning, and weakly supervised registration (Zhang et al., 3 Dec 2025). ContraReg avoids fixed pre-aligned pairs by coupling patch correspondence to the current warp, but low regularization can yield irregular deformations and external negatives were reported to degrade performance (Dey et al., 2022). In segmentation adaptation, pseudo-label noise can create false positives and false negatives, especially under large modality gaps, which is why the method lowers $10\times$07 for pseudo labels and recommends confidence-aware weighting and modality-aware encoders as practical extensions (Liu et al., 2021).

Another recurring issue is that explicit patch-wise contrast is not always present even when the method exhibits PCCL-like behavior. CMA-CLIP is explicitly positioned as embodying the core PCCL principle—fine-grained patch-wise alignment—while not adding a patch-level InfoNCE objective in the reported experiments (Liu et al., 2021). PMCM similarly realizes patch-wise cross-modality learning through patch-mixed images plus KL, center, and triplet losses rather than via a dedicated patch-level contrastive term (Qian et al., 2023). This suggests that PCCL is best treated as a spectrum of local cross-modal alignment strategies, from strict InfoNCE formulations to transformer-mediated or distribution-alignment variants.

Computational cost is also intrinsic to local alignment. CMA-CLIP’s sequence-wise attention over $10\times$08 tokens has quadratic cost $10\times$09, which becomes more demanding for long text or higher image resolution; the paper therefore recommends text truncation, moderate hidden dimension, mixed precision, checkpointing, or sparse and hierarchical attention if scaling further (Liu et al., 2021). Dense region-wise or multi-scale patch-wise contrast likewise increases memory and negative-sampling cost in DCVLP and ContraReg (Shi et al., 2021, Dey et al., 2022).

Several papers explicitly point to richer future PCCL variants. CMA-CLIP notes that a PCCL-style extension could add an auxiliary patch–token contrastive loss over matched patch–token pairs against in-batch negatives, thereby making its local alignment objective explicit (Liu et al., 2021). The image-translation literature argues for semantically informed hard negative mining and semantic relation preservation, rather than treating all non-matching patches as equally informative negatives (Jung et al., 2022). The semantic-segmentation paper proposes class-aware queues, structure-aware matching without labels, modality-specific normalization, and prototype-based alignment for severe modality gaps (Liu et al., 2021). A plausible implication is that future PCCL systems will increasingly combine three ingredients: local correspondence definitions that tolerate imperfect registration, negative selection strategies that reflect semantic hardness, and downstream aggregation modules that preserve gains from local alignment at the sample or slide level.

In aggregate, the literature portrays PCCL as a general local-alignment paradigm rather than a single algorithm. Its core claim is not merely that modalities should be globally aligned, but that transferable multimodal representations often depend on whether the correct local units—patches, regions, spatial indices, or temporal windows—are brought into correspondence under an objective that is robust to noise, ambiguity, and partial misalignment (Zhang et al., 3 Dec 2025, Liu et al., 2021, Dey et al., 2022).