Cross-Modal Prototype Alignment

• The paper demonstrates that cross-modal prototype alignment leverages varied semantic representations—such as Gaussian prototypes and discrete codebooks—to reduce modality gaps.
• It utilizes techniques like attention-based translation, optimal transport, and index-level matching to align heterogeneous features under weak supervision and missing modalities.
• Empirical evaluations in survival analysis, federated learning, and medical imaging show improved performance and robust cross-modal consistency with optimized prototype guidance.

Cross-modal prototype alignment is a family of multimodal learning strategies in which prototypes act as semantic, temporal, or class-conditioned anchors for relating heterogeneous representations. Across recent work, these anchors have been instantiated as interval-aware prototype banks for histopathology and genomics, shared disease prototypes for medical image–report learning, complete class prototypes in federated multimodal learning, Gaussian prototype dictionaries for decoupled multimodal representations, discrete codebooks whose indices are aligned across modalities, and even recycled classifier weights used as semantic prototypes in vision–language alignment (Liu et al., 13 Mar 2025, Wang et al., 2022, Le et al., 2024, Qian et al., 14 Mar 2025, Sen et al., 12 May 2026, Méndez et al., 21 May 2026). The central purpose is consistent: to reduce the semantic gap between modalities while preserving task-relevant structure, especially under missing modalities, weak supervision, or distribution shift.

1. Conceptual basis and scope

In this literature, a prototype is not restricted to a single centroid. It may be a learnable vector, a bank indexed by time interval, a class-wise aggregate, a Gaussian component with mean and covariance, a vector-quantized codeword, or a supervised classifier weight reused as a semantic anchor (Liu et al., 13 Mar 2025, Qian et al., 14 Mar 2025, Sen et al., 12 May 2026, Méndez et al., 21 May 2026). What unifies these formulations is that alignment is mediated through a compact set of reference elements that are more stable than raw instance-to-instance matching.

A recurrent design choice is whether prototypes are shared across modalities or modality-specific. ProSurv keeps two modality-specific banks, $B_p$ and $B_g$, each structured by survival intervals, and achieves alignment through instance-level translation and consistency rather than explicit prototype-to-prototype matching (Liu et al., 13 Mar 2025). MGCA instead uses a shared trainable prototype set $C=\{c_1,\dots,c_K\}$ and enforces cross-modal cluster assignment consistency at the disease level (Wang et al., 2022). CoDAAR takes a third route: each modality has its own codebook, but identical indices are forced to denote the same latent concept through Discrete Temporal Alignment and Cascading Semantic Alignment, explicitly avoiding “prototype competition” from a single shared codebook (Sen et al., 12 May 2026).

This diversity corrects a common misconception: cross-modal prototype alignment is not synonymous with one-to-one prototype matching. Several systems do not directly couple prototype $k$ in one modality to prototype $k$ in another. In ProSurv, alignment is mediated by attention-based translation and $L_2$ consistency on paired instances (Liu et al., 13 Mar 2025). In ProAlign, patch–text similarity allocates patches to LLM-generated textual prototypes, and the alignment is operational rather than loss-driven (Chen et al., 26 Mar 2025). In PICO, per-dimension semantic probabilities suppress style-dominated interactions, and prototypes are constructed over style information rather than over semantic classes directly (Ma et al., 13 Oct 2025).

2. Prototype parameterizations

Prototype parameterization varies with the structure of the task. In ProSurv, each modality $m\in\{p,g\}$ maintains a bank

$B_m\in\mathbb{R}^{K\times n\times d},$

with $K$ survival intervals and $n$ prototypes per interval. The bank is explicitly time-structured: bin $B_g$0 summarizes modality-specific, risk-relevant patterns associated with that interval (Liu et al., 13 Mar 2025).

In DecAlign, prototypes are Gaussian rather than point embeddings. For each modality,

$B_g$1

where $B_g$2 is a mean, $B_g$3 is an SPD covariance, and $B_g$4 is a mixture weight. This parameterization is designed to preserve modality-unique structure while aligning distributions through multi-marginal optimal transport (Qian et al., 14 Mar 2025).

In multimodal federated learning, MFCPL constructs class-wise local prototypes $B_g$5, $B_g$6, and $B_g$7 in a shared projection space, then aggregates them into a complete prototype

$B_g$8

These complete prototypes serve as cross-modal carriers for clients with missing modalities (Le et al., 2024).

Medical image–report learning introduces still another formulation. MGCA uses a shared prototype set with $B_g$9 trainable vectors and obtains soft assignments via Sinkhorn-Knopp over cosine similarities to prototypes (Wang et al., 2022). XPRONET uses a shared cross-modal prototype matrix

$C=\{c_1,\dots,c_K\}$0

with $C=\{c_1,\dots,c_K\}$1 clinical categories and $C=\{c_1,\dots,c_K\}$2 prototypes per category, initialized from K-Means over concatenated image and text global features (Wang et al., 2022).

Vision–language few-shot work broadens the notion further. In "Supervised Classification Heads as Semantic Prototypes" (Méndez et al., 21 May 2026), the rows of a supervised classifier head are recycled as semantic prototypes, $C=\{c_1,\dots,c_K\}$3. In "Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification" (Goswami et al., 25 Mar 2026), text prototypes $C=\{c_1,\dots,c_K\}$4 and image prototypes $C=\{c_1,\dots,c_K\}$5 are mixed, and aligned variants are obtained by projecting image prototypes onto the principal directions of the text embedding space. In ProAlign, global textual prototypes are LLM-generated descriptions of WSI prototype regions, and an initial visual prototype is formed by

$C=\{c_1,\dots,c_K\}$6

where $C=\{c_1,\dots,c_K\}$7 is the most similar patch embedding to textual prototype $C=\{c_1,\dots,c_K\}$8 in a global training patch pool (Chen et al., 26 Mar 2025).

Taken together, these formulations show that “prototype” is an architectural role rather than a fixed mathematical object.

3. Alignment operators

The alignment mechanism determines how prototypes mediate cross-modal correspondence. ProSurv uses scaled dot-product cross-attention to translate a source-modality feature into a mixture over target-modality prototypes:

$C=\{c_1,\dots,c_K\}$9

When both modalities are observed, the translated feature is aligned to the true feature by

$k$0

The paper explicitly states that there is no explicit one-to-one distance term between individual prototypes across banks (Liu et al., 13 Mar 2025).

DecAlign instead aligns prototype distributions through multi-marginal OT. Its heterogeneous alignment combines a global transport term

$k$1

with local sample-to-prototype calibration

$k$2

where the cost tensor is constructed from pairwise Gaussian $k$3-Wasserstein/Bures distances (Qian et al., 14 Mar 2025). This formulation couples global distribution matching with local semantic consistency.

CoDAAR aligns discrete representations at the index level. Each modality has a codebook $k$4 and nearest-neighbor quantization

$k$5

DTA updates codewords using self-modal and time-aligned cross-modal features, while CSA moves same-index centroids toward a multimodal anchor through fixed convex updates. The paper’s stated objective is semantic consensus across separate codebooks without erasing modality-specific structure (Sen et al., 12 May 2026).

MGCA operationalizes disease-level prototype alignment through cross-modal cluster assignment consistency. Prototype prediction distributions are

$k$6

and the loss uses Sinkhorn-derived soft targets from the opposite modality (Wang et al., 2022).

In PICO, alignment is column-wise but prototype-informed. Patch–word interaction is reweighted by semantic probabilities,

$k$7

where the probabilities are derived from style prototypes constructed by weighted clustering and iteratively aggregated using performance feedback (Ma et al., 13 Oct 2025). This formulation treats style suppression as a prerequisite for reliable semantic alignment.

4. Objectives, supervision, and inference regimes

Training objectives are typically composite. ProSurv combines discrete-time survival loss, intra-modal prototype similarity loss, and paired cross-modal alignment:

$k$8

for multimodal inputs, and removes $k$9 for unimodal inputs. The framework supports unpaired training because $k$0 is defined per available modality, and at inference it can generate a missing modality feature from the observed modality’s prototype-guided translation (Liu et al., 13 Mar 2025).

MFCPL combines supervised classification with dual-level prototype alignment and explicit feature matching:

$k$1

Here, $k$2 aligns shared fused representations to complete class prototypes, $k$3 is a prototype-based contrastive term over modality-specific features, and $k$4 directly aligns projected modality-specific embeddings (Le et al., 2024). This objective is tailored to federated settings with zero-filled missing modalities and server-broadcast complete prototypes.

XPRONET combines report-generation cross-entropy with an improved multi-label contrastive loss:

$k$5

Its positive pairs are samples that share at least one non-zero label, and the tolerance for positive dissimilarity is modulated by the ratio $k$6 of differing to total labels (Wang et al., 2022). The prototypes are therefore not only cross-modal anchors but also multi-label regularizers.

Not all methods require an explicit alignment loss. ProAlign states that “patch–text contrast” does not mean InfoNCE training. Instead, similarity matrices

$k$7

are used to initialize prototypes and allocate patches to prototype regions, and refinement is performed by parameter-free attention over assigned patches (Chen et al., 26 Mar 2025). This is a deterministic alignment pipeline rather than an optimized contrastive objective.

Few-shot methods expose another regime. CMM learns a residual linear mapping

$k$8

to align image features to the text feature space, and then optimizes a triplet loss against text prototypes with hard negative mining (Yang et al., 2024). The training-free method in (Goswami et al., 25 Mar 2026) projects image prototypes onto the principal directions of the semantic text embedding space and mixes them with text prototypes; a separate image-space LDA classifier models anisotropy when CLIP’s cross-modal alignment is weak. These methods suggest that prototype alignment can be performed either by optimizing mappings or by analytically manipulating prototype geometry.

5. Applications and empirical behavior

The empirical record covers survival analysis, federated learning, general multimodal representation learning, grounding, retrieval, and medical report generation. In ProSurv, evaluation on four TCGA cohorts—BRCA $k$9, BLCA $L_2$0, STAD $L_2$1, and CRAD $L_2$2—used 5-fold cross-validation and test-fold C-index. Under multimodal inference, ProSurv achieved an overall C-index of $L_2$3, exceeding MCAT $L_2$4, MOTCAT $L_2$5, CMTA $L_2$6, and SurvPath $L_2$7. Under H-only inference after multimodal training, it achieved $L_2$8 overall versus G-HANet $L_2$9 and MCAT $m\in\{p,g\}$0, and removing prototypes reduced overall C-index by $m\in\{p,g\}$1 across settings (Liu et al., 13 Mar 2025).

In multimodal federated learning, MFCPL maintained gains under missing rates $m\in\{p,g\}$2. On UCI-HAR, it outperformed the best baseline by $m\in\{p,g\}$3 at $m\in\{p,g\}$4 and still led at $m\in\{p,g\}$5 with $m\in\{p,g\}$6 versus $m\in\{p,g\}$7. On Hateful Memes, it reached AUC values of $m\in\{p,g\}$8, $m\in\{p,g\}$9, and $B_m\in\mathbb{R}^{K\times n\times d},$0 for $B_m\in\mathbb{R}^{K\times n\times d},$1, $B_m\in\mathbb{R}^{K\times n\times d},$2, and $B_m\in\mathbb{R}^{K\times n\times d},$3, respectively, and on MELD it reached UAR values of $B_m\in\mathbb{R}^{K\times n\times d},$4, $B_m\in\mathbb{R}^{K\times n\times d},$5, and $B_m\in\mathbb{R}^{K\times n\times d},$6 (Le et al., 2024). These results are directly tied to complete-prototype guidance and modality-specific alignment.

DecAlign reports consistent gains across CMU-MOSI, CMU-MOSEI, CH-SIMS, and IEMOCAP. On MOSI it reported MAE $B_m\in\mathbb{R}^{K\times n\times d},$7, Corr $B_m\in\mathbb{R}^{K\times n\times d},$8, Acc-2 $B_m\in\mathbb{R}^{K\times n\times d},$9, Acc-7 $K$0, and F1 $K$1; on MOSEI it reported MAE $K$2, Corr $K$3, Acc-2 $K$4, Acc-7 $K$5, and F1 $K$6 (Qian et al., 14 Mar 2025). The ablation finding that removing heterogeneous alignment causes larger drops than removing homogeneous alignment indicates that prototype-guided OT is carrying a substantial part of the cross-modal benefit.

CoDAAR positions index-level alignment as a route to cross-modal-domain generalization. With AV pretraining, it reported $K$7 versus DCID $K$8 and MICU $K$9 on the AVE classification and AVVP localization average. With AVT pretraining, it reported $n$0 versus DCID $n$1 and MICU $n$2, and on zero-shot retrieval across MSCOCO and Clotho it reported average $n$3 of $n$4 versus DCID $n$5 and MICU $n$6 (Sen et al., 12 May 2026). These results suggest that index-level prototype consensus can support both cross-modal and cross-domain transfer.

Open-vocabulary grounding supplies a different kind of evidence. PAML combines ALBEF-aligned features, a visual discriminative feature encoder, a prototype discovering and inheriting mechanism with a bank $n$7, and a multi-stage decoder. On standard-scene benchmarks it reported RefCOCO val/testA/testB of $n$8, RefCOCO+ val/testA/testB of $n$9, RefCOCOg val-g of $B_g$00, ReferIt test of $B_g$01, and Flickr30K test of $B_g$02; under cross-dataset open-vocabulary evaluation it also reported state-of-the-art results on several train–test transfers (Xie et al., 8 Sep 2025).

Medical report generation furnishes a sequence-generation view of prototype alignment. XPRONET reported on IU-Xray BLEU-1/2/3/4 of $B_g$03, ROUGE-L $B_g$04, METEOR $B_g$05, and CIDEr $B_g$06, while on MIMIC-CXR it reported BLEU-1/2/3/4 of $B_g$07, ROUGE-L $B_g$08, and METEOR $B_g$09 (Wang et al., 2022). The ablation removing the prototype network produced the largest degradation, indicating that the shared prototype matrix is not merely auxiliary.

6. Limitations, misconceptions, and research directions

A first limitation is alignment ambiguity. ProSurv explicitly notes that it lacks explicit cross-modal prototype coupling, so banks may drift and alignment remains indirect (Liu et al., 13 Mar 2025). PAML similarly does not construct a text-side prototype bank; its visual prototypes are discovered in a linguistically steered visual space, and the paper identifies fixed prototype bank size as a limitation (Xie et al., 8 Sep 2025). This suggests that stronger prototype-to-prototype coupling remains an open design axis.

A second limitation is the trade-off between consensus and modality specificity. CoDAAR argues that a single shared codebook can cause high-variance modalities to dominate prototype positioning, harming modality-specific fidelity through “prototype competition” (Sen et al., 12 May 2026). DecAlign warns that overly large alignment weights can suppress modality-unique features, and recommends small values such as $B_g$10 to avoid over-alignment (Qian et al., 14 Mar 2025). The image-tabular DFPL paper makes a related point: global token-averaged alignment captures only coarse consistency and overlooks fine-grained distributional and semantic misalignment, motivating shared-specific prototype modeling and prototype-guided fine-grained alignment (Zhou et al., 3 Jun 2026).

A third limitation is dependence on external semantic sources. ProAlign depends on LLM-generated textual prototype descriptions, so prototype quality is tied to the pathology vocabulary supplied by the prompt (Chen et al., 26 Mar 2025). XPRONET depends on CheXbert pseudo labels for prototype initialization and query restriction (Wang et al., 2022). PICO depends on rSum feedback to weight iteratively constructed style prototypes, so noisy performance estimates can affect prototype aggregation (Ma et al., 13 Oct 2025).

The literature also converges on several future directions. ProSurv proposes dynamic prototype budgeting, explicit cross-modal prototype matching through OT or symmetric InfoNCE, cohort-specific banks with shared anchors, prototype-to-instance attributions, and uncertainty-aware censoring (Liu et al., 13 Mar 2025). DFPL identifies exploitation of unpaired samples as future work in image-tabular settings (Zhou et al., 3 Jun 2026). The social-media popularity framework proposes adaptive hierarchical taxonomies, dynamic prototypes that evolve over time, and automated prompt discovery (Zhou et al., 22 Aug 2025). PAML suggests dynamic prototype management and explicit text-side prototype contrast (Xie et al., 8 Sep 2025).

A plausible implication is that cross-modal prototype alignment is becoming less about static centroids and more about structured correspondence operators: attention-based translation, transport plans, balanced assignment consistency, codebook index consensus, and task-aware fusion. The common denominator is not the prototype form itself, but the use of a compact semantic scaffold to stabilize multimodal learning under heterogeneity.