Prototype-Aware Multimodal Learning (PAML)

• Prototype-Aware Multimodal Learning (PAML) is a framework where prototypes serve as primary computational structures to organize and align cross-modal information.
• It employs diverse construction strategies—from centroid estimation to learnable query-derived anchors—to optimize fusion, balance modalities, and substitute missing inputs.
• PAML techniques integrate various loss functions and dynamic adaptations, yielding improved performance in applications such as few-shot recognition, cancer survival analysis, and open-vocabulary grounding.

Prototype-Aware Multimodal Learning (PAML) denotes a family of multimodal methods in which prototypes become the primary computational object for organizing cross-modal information. In the works considered here, a prototype may be a class centroid in an embedding space, a learnable query-derived latent summary, a case-aware class anchor, or an evolving memory over test samples. Rather than treating multimodality as simple early or late fusion, PAML uses prototypes to align modalities, stabilize representation learning, compress redundant tokens, substitute missing inputs, and structure downstream prediction. This formulation appears in few-shot recognition, multimodal classification, cancer survival analysis, molecular property prediction, federated learning, test-time adaptation of vision-LLMs, and open-vocabulary visual grounding (Pahde et al., 2020, Liu et al., 6 Mar 2025, Xie et al., 8 Sep 2025).

1. Conceptual core and formal basis

The foundational formalism comes from prototypical metric learning. In a standard prototypical network, the prototype for class $k$ is the mean of support embeddings,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$

and the posterior over labels is obtained by a softmax over negative distances to prototypes,

$$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$

"Multimodal Prototypical Networks for Few-shot Learning" keeps this decision rule but enriches prototype formation through an auxiliary modality: text is mapped into the visual feature space during training so that prototype estimation is informed by more than a few support images, while inference remains image-only (Pahde et al., 2020).

A second core formulation appears in "PMR: Prototypical Modal Rebalance for Multimodal Learning", where each modality has its own class prototype, defined as a centroid in the corresponding representation space,

$$c_k^0=\frac{1}{N_k}\sum_{i=1}^{N_k} z_{k_i}^0,\qquad c_k^1=\frac{1}{N_k}\sum_{i=1}^{N_k} z_{k_i}^1.$$

These prototypes support non-parametric unimodal classifiers based on Euclidean distance, allowing each modality to be evaluated independently of the fusion head. This is a distinct PAML move: prototypes are not only class summaries for prediction, but also instruments for diagnosing modality imbalance and providing modality-specific supervision (Fan et al., 2022).

Taken together, these two strands already delimit the field. First, prototypes are typically anchored in an embedding space shared across instances of a class. Second, multimodality enters by changing how those prototypes are constructed, aligned, or used. Third, the prototype may govern not only classification but also optimization, regularization, and transfer. This suggests that PAML is best understood as prototype-centric multimodal representation learning rather than as conventional fusion with an auxiliary clustering step.

2. Prototype construction strategies

A major axis of variation in PAML concerns how prototypes are instantiated. The simplest strategy is centroid construction from observed embeddings, but recent systems expand this substantially.

In "Multimodal Prototypical Networks for Few-shot Learning", the key design is a text-to-visual generator $g_\phi(\cdot)$ that maps text descriptions into the visual feature space,

$z_v=f_\theta(x_v),\qquad z_t=g_\phi(x_t).$

The generated text features behave like image embeddings, so prototypes can be formed from both actual image samples and description-derived features. Multimodality is therefore injected at the level of prototype formation rather than by requiring fused test-time inputs (Pahde et al., 2020).

"Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition" adopts a different construction. Visual prototypes are produced by a baseline prototype constructor such as TRX or STRM, while label texts are encoded by a frozen CLIP text encoder, refined by a semantic-enhanced module, and then inflated to match the tuple-wise visual prototype tensor. The final multimodal prototype is formed by the Multimodal Prototype-Enhanced module, with the default weighted average

$P^M=(1-\lambda)P^V+\lambda P^T,$

using $\lambda=0.5$ by default. Here the prototype is explicitly the fusion locus between video evidence and label semantics (Ni et al., 2022).

"Adaptive Prototype Learning for Multimodal Cancer Survival Analysis" replaces centroids with learnable query-induced prototypes. Histology patches and genomic pathway embeddings are compressed by cross-attention from learnable query sets $Q^{(H)}$ and $Q^{(G)}$, producing compact prototype sequences $$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$0 and $$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$1. These are then concatenated and fused by multimodal mixed self-attention. The prototype is thus a task-driven latent summary discovered end-to-end rather than a predefined semantic bin or class mean (Liu et al., 6 Mar 2025).

"ProtoMol: Enhancing Molecular Property Prediction via Prototype-Guided Multimodal Learning" introduces a shared prototype bank with learnable class-specific anchors,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$2

Graph and text representations are projected into this shared space, compared to all prototypes by squared Euclidean distance, transformed into similarities, and sparsified by top-$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$3 selection before normalization into prototype distributions. In this design, prototypes are neither direct averages nor latent attention outputs alone; they are explicit semantic anchors that both modalities must activate coherently (Wang et al., 19 Oct 2025).

"Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification" further decomposes prototypes into shared and modality-specific sets extracted from token-level features by learnable prototype queries. It adds a prototype diversity loss to prevent collapse and prototype-level disentanglement to separate shared and specific semantics. This makes prototype construction simultaneously compact, diverse, and branch-specific (Zhou et al., 3 Jun 2026).

A plausible implication is that PAML has moved from static centroid estimation toward learned prototype systems in which prototypes function as bottlenecks, semantic anchors, or structured latent variables. The choice among centroids, query-derived summaries, and learnable anchor banks typically reflects the task’s sparsity, redundancy, or heterogeneity profile.

3. Objective functions and training regimes

PAML methods differ sharply in the losses used to shape prototype geometry. The few-shot line remains closest to metric learning. "Multimodal Prototypical Networks for Few-shot Learning" uses episodic negative log-likelihood over query points and retains a nearest-prototype test rule,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$4

while adding training signals that make text-generated features close to real visual features and useful for prototype construction (Pahde et al., 2020).

"PMR: Prototypical Modal Rebalance for Multimodal Learning" adds the Prototypical Cross-Entropy loss for each modality,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$5

to accelerate the slow-learning modality by clustering its features toward class prototypes. It then introduces Prototypical Entropy Regularization during early epochs to prevent premature convergence of the dominant modality. The total loss adaptively activates prototype supervision for the weaker branch while slowing the stronger one, making prototype learning part of the optimization schedule rather than just the classifier (Fan et al., 2022).

"MORN" introduces PRIDE, the Prototype Similarity Difference,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$6

which measures whether a prototype is closer to its own class’s “real prototype” than to other classes. PRIDE is also converted into an InfoNCE-style training loss. This is unusual within PAML because prototype quality is itself operationalized and optimized, not merely assumed (Ni et al., 2022).

"Adaptive Prototype Learning for Multimodal Cancer Survival Analysis" supervises the entire prototype extractor and fusion stack with negative log-likelihood for survival prediction and, in the provided text, introduces no auxiliary prototype-specific losses. The prototypes are learned indirectly through end-to-end survival supervision, which contrasts with methods that explicitly regularize prototype relations (Liu et al., 6 Mar 2025).

"Decoupled Multimodal Prototypes for Visual Recognition with Missing Modalities" trains case-aware class prototypes with ArcFace and supervised contrastive loss,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$7

Because logits are prototype similarities, the margin-based loss directly shapes class-wise prototype geometry under complete, image-missing, and text-missing conditions (Lu et al., 13 May 2025).

"ProtoMol" combines prediction loss, prototype-distribution alignment, and prototype contrastive loss,

$$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$8

Alignment is enforced by KL divergence between graph and text prototype distributions, while prototype contrastive learning keeps same-class prototypes close and different-class prototypes separated. "DFPL" similarly combines classification, coarse shared-specific disentanglement, diversity regularization, and prototype-guided fine-grained alignment based on optimal transport and prototype-to-class semantic consistency (Wang et al., 19 Oct 2025, Zhou et al., 3 Jun 2026).

These objective designs show that PAML does not prescribe a single loss family. Prototypes may be optimized through episodic NLL, prototype-classification terms, contrastive alignment, entropy regularization, optimal transport, or margin-based metric learning. What remains invariant is that the training signal is routed through prototype structure.

4. Representative domains and empirical profiles

PAML is not confined to one modality pair or one downstream problem. It appears in recognition, survival modeling, intent understanding, property prediction, popularity prediction, and grounding.

Setting	Representative method	Prototype role
Few-shot image recognition	"Multimodal Prototypical Networks for Few-shot Learning" (Pahde et al., 2020)	Text-derived visual features improve image prototypes
Few-shot action recognition	"Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition" (Ni et al., 2022)	Label-text prototypes are fused with visual prototypes
Cancer survival analysis	"Adaptive Prototype Learning for Multimodal Cancer Survival Analysis" (Liu et al., 6 Mar 2025)	Query-learned histology and genomic prototypes compress redundant tokens
Multimodal intent recognition	"MVCL-DAF++" (Huang et al., 22 Sep 2025)	Batch-computed class prototypes anchor instance embeddings
Molecular property prediction	"ProtoMol" (Wang et al., 19 Oct 2025)	Shared class-specific anchors align graph and text
Social media popularity prediction	"Cross-Modal Prototype Augmentation and Dual-Grained Prompt Learning..." (Zhou et al., 22 Aug 2025)	Hierarchical visual and textual category prototypes regularize alignment
Open-vocabulary visual grounding	"Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding" (Xie et al., 8 Sep 2025)	Multi-neighbor semantic prototypes support open-vocabulary transfer

The empirical record in these papers is correspondingly diverse. In fine-grained few-shot image recognition, "Multimodal Prototypical Networks for Few-shot Learning" reports that the method outperforms state-of-the-art single-modal and multimodal few-shot learning methods on CUB-200 and Oxford-102, with the consistent trend that auxiliary text improves low-data generalization (Pahde et al., 2020). In few-shot action recognition, "MORN" reports best accuracies of 87.1% on HMDB51, 97.7% on UCF101, 94.6% on Kinetics, and 73.6% on SSv2, while also showing large PRIDE gains over CLIP-initialized baselines (Ni et al., 2022).

In multimodal cancer survival analysis, "APL" evaluates on BRCA, BLCA, HNSC, COADREAD, and STAD, reporting C-index values of $$c_k=\frac{1}{|S_k|}\sum_{(x_i,y_i)\in S_k} f_\theta(x_i),$$9, $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$0, $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$1, $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$2, and $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$3, with an average of $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$4. The paper states that this surpasses all unimodal and multimodal baselines and improves over CCL’s average of 0.690 by 3.4 percentage points (Liu et al., 6 Mar 2025). In multimodal intent recognition, "MVCL-DAF++" reports 76.18 ACC and 75.66 WF1 on MIntRec, and 60.40 ACC and 59.23 WF1 on MIntRec2.0, with rare-class WF1 gains of +1.05% and +4.18%, respectively (Huang et al., 22 Sep 2025).

PAML also supports interpretability claims when the prototype mechanism is explicit enough. "APL" visualizes histology queries attending to different whole-slide regions and genomic queries corresponding to different pathway sets, while "ProtoMol" reports prototype activations associated with chemically meaningful motifs such as benzene rings, amide linkages, cyclohexylamide, naphthalene systems, biphenyl scaffolds, and carbonyl bridges (Liu et al., 6 Mar 2025, Wang et al., 19 Oct 2025). This suggests that prototype structure is especially valuable when the downstream community requires latent summaries with domain semantics rather than only improved accuracy.

5. Imbalance, missing modalities, and federated heterogeneity

A recurrent misconception is that PAML requires all modalities to be present at inference. Several systems explicitly contradict this. "Multimodal Prototypical Networks for Few-shot Learning" uses text only during training and performs standard prototype-based image classification at test time (Pahde et al., 2020). Other works go further by making prototypes the mechanism for missing-modality robustness.

"PMR" addresses modality imbalance in two-modality classification by constructing modality-specific prototypes, using them to evaluate unimodal performance independently of the fusion head, and then selectively accelerating the slower branch through Prototypical Cross-Entropy while applying prototype-based entropy regularization to the dominant branch during early training. The central claim is that imbalance is not only a matter of gradient magnitude but also of gradient-direction interference, so prototype-space supervision provides a more intrinsic signal for the weak modality (Fan et al., 2022).

In federated learning, "Adaptive Prototype Knowledge Transfer for Federated Learning with Mixed Modalities and Heterogeneous Tasks" uses label-guided prototypes on labeled unimodal clients, clustering-based prototype construction on unlabeled multimodal clients, and server-side semantic completion to turn unimodal prototypes into paired multimodal prototypes before clustering them into global prototypes. Knowledge transfer is then mediated by Jensen–Shannon consistency between image and text prototype assignments. The paper also states that only mapping modules are aggregated, and that a client relationship graph dynamically adjusts aggregation weights (Gai et al., 6 Feb 2025).

"Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast" introduces a server prototype library containing image, text, and fusion prototypes. These prototypes play three roles simultaneously: masks replacing missing modalities during local training, contrastive anchors reducing client drift, and retrieval keys for model-agnostic unimodal inference. The paper reports a 3.7% improvement in inference accuracy with 50% modality missing during training and a 23.8% improvement during uni-modality inference relative to baselines (Bao et al., 2023).

"Decoupled Multimodal Prototypes for Visual Recognition with Missing Modalities" moves the prototype mechanism to the output head. For each class, it learns complete-case, image-missing, and text-missing prototypes, each decomposed into modality-specific components. Logits are cosine similarities against the appropriate case-specific prototype, and the paper reports consistent gains across MM-IMDb, UPMC Food-101, and Hateful Memes under 50%, 70%, and 90% missing rates (Lu et al., 13 May 2025). "DFPL" addresses incomplete image-tabular classification with shared and modality-specific prototypes, prototype-level disentanglement, optimal-transport alignment, and class-aware multi-scale aggregation, reporting superiority on MIMIC, ADNI, and DVM under multiple missing-modality settings (Zhou et al., 3 Jun 2026).

These works collectively establish a more precise view: in PAML, prototypes can substitute for missing evidence, calibrate case-specific decision boundaries, and carry semantic structure across clients or modalities even when raw data are absent or incomparable.

6. Dynamic prototypes, online adaptation, and open-vocabulary transfer

Another major development is the transformation of prototypes from static class summaries into dynamic memories that evolve during deployment.

"Dual Prototype Evolving for Test-Time Generalization of Vision-LLMs" maintains textual prototypes built from prompt ensembles and visual prototypes stored in class-specific priority queues of low-entropy test embeddings. For each incoming unlabeled test image, DPE optimizes residuals on both prototype sets using self-entropy minimization and a bidirectional prototype alignment loss, then updates textual prototypes by cumulative averaging and visual prototypes by queue replacement rules. The paper reports average improvements of 3.55% over TPT with ResNet-50 and 3.49% over TPT with ViT-B/16 across ImageNet and its out-of-distribution variants, while also being about 5× faster than TPT and over 10× faster than DiffTPT (Zhang et al., 2024).

"Dynamic Multimodal Prototype Learning in Vision-LLMs" extends this idea with a training-free optimal-transport formulation. Each class prototype is modeled as a discrete distribution over multiple textual descriptions and dynamically updated visual particles,

$$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$5

while each test image is represented as a distribution over augmented views. Class prediction is based on an OT distance between the image distribution and the multimodal prototype distribution, and only confident samples update the visual particle cache. The paper reports a 1.03% average accuracy improvement over state-of-the-art methods on ImageNet and its variant datasets (Zhu et al., 4 Jul 2025).

Open-vocabulary grounding pushes PAML into dense localization. "Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Grounding" combines ALBEF for initial alignment, a Visual Discriminative Feature Encoder, a prototype discovering and inheriting mechanism with $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$6-nearest semantic prototypes, and a Multi-stage Decoder for iterative fusion and box regression. The prototype bank contains 2048 embeddings, multi-neighbor inheritance uses soft assignment over the $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$7-nearest prototypes, and the paper finds the best result at $$p(y=k\mid z)=\frac{\exp(-d(z,c_k))}{\sum_{k'}\exp(-d(z,c_{k'}))}.$$8. It reports 75.65% on ReferIt in standard scene and state-of-the-art or near-state-of-the-art open-vocabulary results across ReferIt, Flickr30K, RefCOCO, RefCOCO+, and RefCOCOg (Xie et al., 8 Sep 2025).

This line of work changes the interpretation of prototypes. They are no longer only class centroids learned offline; they become adaptive memories, transport supports, and semantic neighborhoods that evolve with the test stream or enable transfer to unseen categories.

7. Limitations, misconceptions, and open directions

The literature also makes clear that prototype awareness does not remove fundamental design tradeoffs. "Adaptive Prototype Learning for Multimodal Cancer Survival Analysis" explicitly notes that the number of learnable queries is fixed to 300 for histology and 128 for genomics across all datasets, and identifies dynamic determination of the number of queries as future work (Liu et al., 6 Mar 2025). "ProtoMM" depends on confidence filtering, meaningful image and text augmentations, a fixed number of visual particles, and a streaming test-time protocol; the paper notes that too low a threshold admits noisy samples and too high a threshold yields insufficient cache growth (Zhu et al., 4 Jul 2025).

In federated settings, some aggregation mechanisms remain only partially formalized in the available description. "AproMFL" states that the semantic-completion weighting and client relationship graph aggregation are conceptually described, but the excerpt does not provide a fully explicit mathematical rule; its experiments are also limited to two image clients, two text clients, and two multimodal clients (Gai et al., 6 Feb 2025). "PmcmFL" evaluates on a tiny VQA benchmark derived from VQAv2 with 30 non-IID clients, so its scalability to broader multimodal federated regimes is not fully established (Bao et al., 2023).

Several papers also correct a second misconception: prototypes are not necessarily fixed global class means. They may be case-specific, as in DPL; shared and modality-specific, as in DFPL; or learned latent summaries, as in APL and ProtoMM (Lu et al., 13 May 2025, Zhou et al., 3 Jun 2026). A third misconception is that nearest-prototype inheritance is always sufficient. The open-vocabulary grounding literature argues the opposite: reliance on a single nearest prototype can be brittle for novel categories, which motivates multi-neighbor inheritance (Xie et al., 8 Sep 2025).

The remaining research agenda, as these papers jointly indicate, concerns adaptive prototype cardinality, robustness under noisy pseudo-labels and confidence filtering, principled handling of heterogeneous clients and tasks, and tighter coupling between interpretability and performance. A plausible implication is that future PAML systems will be judged less by whether they use prototypes at all than by how they control prototype granularity, evolution, sparsity, and semantic faithfulness across modalities.