Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

Abstract: Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities. In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation. Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features, regardless of whether the input modalities are complete or partially absent. However, such input-conditioned strategies are hindered by the escalating unreliability of instance-level features; as higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance. To address this limitation, we hypothesize that learnable latent prompts themselves encapsulate stable, modality-intrinsic priors that are decoupled from corrupted inputs. Consequently, we propose a novel paradigm: Learning from Reliable Latent Prompts. Unlike prior methods, we model input-agnostic learnable prompts as stable latent anchors that enable robust guidance and effective cross-modal knowledge compensation, even under extreme missing rates (e.g., 90%). Empirical results across three benchmark datasets demonstrate that our "learn-from-latent-prompts" approach achieves state-of-the-art performance across a wide range of missing-modality scenarios. Extensive experiments further confirm the effectiveness of this paradigm in providing a robust solution to the missing-modality problem.

• The paper introduces Reliable Latent Prompts (LLP), which uses input-agnostic modality anchors and dual cross-attention prompts to preserve cross-modal alignment when image or text inputs are missing.
• LLP outperforms six competing methods in all 27 tested dataset, missing-modality, and missing-rate combinations, with notable gains including 2.6 AUROC points on Hateful Memes under missing-image conditions.
• Ablations and robustness tests show that latent anchors improve stability as missing rates rise, although the method remains limited to bimodal CLIP-style models and requires validation on larger multimodal architectures.

Overview and Motivation

The paper "Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities" (2606.30597) addresses a persistent weakness in large-scale multimodal models (LMMs): their reliance on modality-complete inputs. In deployment, privacy constraints, sensor failures, and heterogeneous collection environments routinely leave one modality absent, and models trained on complete paired data degrade sharply. The authors focus on the dominant prompt-learning line of work, in which a frozen CLIP backbone (Shen, 2022) is adapted via learnable tokens. They identify a structural flaw in recent input-conditioned prompting methods such as DCP and SyP: these methods generate dynamic prompts from instance-level features, so as the missing rate $\eta$ grows, the conditioning signal is drawn increasingly from incomplete, modality-biased observations. The paper's central hypothesis is that learnable latent prompts, optimized independently of inputs, encode stable modality-intrinsic priors and therefore constitute a more reliable basis for prompt generation than corrupted instance features. This is a strong claim: it asserts that input-agnostic prompting is not merely a simplification but is intrinsically more reliable under missingness, and the paper supports it with a gradient-decomposition argument and consistent empirical gains.

Method: Learning from Reliable Latent Prompts (LLP)

LLP operates on a bimodal (image–text) CLIP backbone with a ViT-B/16 image encoder. For each missing-modality type $m \in \{c, m_1, m_2\}$, prompts are prepended to the token sequences of both encoders, with only the prompts and a final fully connected layer trained. The prompt design has three components.

Modality-specific latent anchors. Input-agnostic learnable prompts $P_{I,A}$ and $P_{T,A}$ are transformed through a lightweight FC–GELU–FC–LayerNorm block into latent anchors $P_{I,S}$ and $P_{T,S}$. These anchors, rather than instance features, serve as the semantic foundation for prompt formation.

Dual anchor-induced prompts (DAIP). A dual-path multi-head cross-attention mechanism lets the anchors of the two modalities exchange information bidirectionally: text anchors attend to image anchors and vice versa. The resulting attended representations are passed through a bottleneck MLP with LayerNorm to produce $P_{I,D}$ and $P_{T,D}$. The final per-encoder prompt is the concatenation $[P_{I,S}, P_{I,D}]$, so the anchors act as semantic foundations rather than static offsets.

Layer-wise prompt refinement. Following prior work, prompts are injected into each of $M=6$ encoder layers and progressively refined through bottleneck MLP transformations, maintaining coherent information flow across depth.

The paper also provides a theoretical analysis: for input-conditioned prompting $m \in \{c, m_1, m_2\}$0, the expected gradient decomposes as $m \in \{c, m_1, m_2\}$1, so as $m \in \{c, m_1, m_2\}$2 optimization is dominated by unimodal gradients, biasing prompts toward unimodal semantics and disrupting cross-modal alignment. Because LLP's prompts $m \in \{c, m_1, m_2\}$3 do not depend on the input or missing mask, this bias is removed by construction. The argument is straightforward and plausible, though it is an idealized analysis that assumes the anchors' gradients are unaffected by the distribution of missing patterns in practice.

Empirical Results

The evaluation covers MM-IMDb (F1-Macro), UPMC Food-101 (top-1 accuracy), and Hateful Memes (AUROC), under missing-text, missing-image, and missing-both settings at missing rates $m \in \{c, m_1, m_2\}$4, against CoOp, MMP, MaPLe, DePT, DCP, and SyP. LLP achieves the best result in all 27 dataset–rate–scenario combinations. Representative results:

Dataset / Metric	Setting ($m \in \{c, m_1, m_2\}$5)	SyP	LLP
MM-IMDb (F1-Macro)	missing text	50.21	50.47
MM-IMDb (F1-Macro)	missing both	49.63	51.25
Food-101 (Accuracy)	missing text	76.33	78.10
Food-101 (Accuracy)	missing both	81.03	81.69
Hateful Memes (AUROC)	missing image	69.70	70.90
Hateful Memes (AUROC)	missing both	68.93	69.17

Two observations stand out. First, the margins widen as the missing rate increases, which is precisely the regime where input-conditioned prompts should be least reliable; the gains are largest on Hateful Memes, the dataset that explicitly requires joint cross-modal reasoning (e.g., +2.6 AUROC over SyP at $m \in \{c, m_1, m_2\}$6 missing-image). Second, the paper documents asymmetric modality sensitivity across datasets — MM-IMDb and Food-101 degrade more when text is missing, Hateful Memes when images are missing — and LLP yields its largest gains in exactly these harder cases, suggesting the anchors provide modality-appropriate compensation rather than a generic boost.

Ablations isolate the contributions. Latent anchors alone (variant "Ours (A)") already substantially outperform a feature-dropping baseline, and adding DAIP yields further consistent gains (e.g., on MM-IMDb missing-both at $m \in \{c, m_1, m_2\}$7: 51.40% → 53.28%; Hateful Memes: 66.81% → 69.62%). The anchor-to-full-model gap grows with the missing rate, consistent with the theoretical claim. Generalization experiments on MM-IMDb show that models trained under the missing-both setting perform stably across all evaluation scenarios and across the full 10%–90% missing-rate range. Robustness analysis on Hateful Memes reports the smallest degradation of all methods, with slight performance improvements in some cases at $m \in \{c, m_1, m_2\}$8 — an unusual result that the authors attribute to reduced reliance on corrupted inputs. Feature visualizations under $m \in \{c, m_1, m_2\}$9 show that SyP produces fragmented, modality-split class clusters while LLP maintains compact intra-class and separated inter-class structure, corroborating the gradient analysis empirically. A hyperparameter study finds a prompt depth of 6 and length of 16 optimal.

Limitations and Open Questions

The paper is candid about scope. LLP is developed and evaluated exclusively in bimodal settings; extension to three or more modalities (audio, video) is untested, and the stated proportional-sampling formulation for $P_{I,A}$0 modalities is unvalidated. The method is evaluated only on two-stream CLIP-style architectures; its behavior on unified or large-scale foundation models remains an open question. The theoretical analysis, while directionally informative, rests on a decomposition of expected gradients that does not model the learned anchors' own optimization dynamics. Additionally, the reported improvements at extreme missing rates over the strongest baseline (SyP) are sometimes modest in absolute terms (e.g., 0.24–0.49 points on several MM-IMDb and Hateful Memes settings), so the practical significance of the largest-missing-rate gains warrants scrutiny. Whether latent anchors can capture instance-specific semantics at all — as opposed to dataset-level priors — and how they scale to fine-grained or open-vocabulary tasks are questions the paper leaves unaddressed.

Conclusion

This paper reorients missing-modality prompt learning away from input-conditioned dynamic prompts toward input-agnostic latent anchors, arguing and showing that prompts grounded in stable modality-level priors degrade more gracefully than prompts conditioned on incomplete observations. LLP attains state-of-the-art results across all tested settings on three benchmarks, with the largest advantages at high missing rates, and its component ablations, generalization tests, robustness curves, and gradient analysis jointly support the latent-prior hypothesis. The main caveats are the bimodal scope, the CLIP-only backbone, and margins over the strongest baseline that narrow in some extreme-missing configurations.