Asymmetric Orthogonal Prompting

• Asymmetric Orthogonal Prompting is a multimodal design that indexes prompts by modality, ensuring robust learning when one or more modalities are missing.
• The method employs modality-specific learnable prompt tensors that are combined via element-wise summation for complete inputs, enhancing task performance.
• An orthogonality loss minimizes cosine similarity between the prompts, promoting diversity and improving generalization across seen and unseen missing-modality cases.

Asymmetric Orthogonal Prompting is an Editor’s term for a multimodal prompt design in which prompts are indexed by modality rather than by missingness pattern, and approximate orthogonality between those prompts is enforced to encourage distinct modality-specific information under missing modalities. In "Towards Robust Multimodal Prompting with Missing Modalities" (Jang et al., 2023), this design is instantiated in a two-modality image–text setting: an image-specific prompt and a text-specific prompt are activated according to which modalities are present, combined by element-wise summation for modality-complete inputs, and regularized by minimizing their absolute cosine similarity. The stated objective is to enhance generalization in cases where modalities are missing, including seen and unseen missing-modality patterns between training and inference.

1. Problem formulation and motivation

The formulation is given for two modalities, $m_1$ and $m_2$, corresponding to text and image. The modality-complete dataset is

$D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$

while the single-modality datasets are

$$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$

To preserve a fixed two-modality input structure for the multimodal transformer, missing inputs are replaced by placeholders, denoted $\tilde{x}_{m_1}$ for text and $\tilde{x}_{m_2}$ for image. The augmented single-modality datasets are therefore

$$\tilde{D}_{m_1}=\{(x_{m_1}^j,\tilde{x}_{m_2}^j,y^j)\}, \qquad \tilde{D}_{m_2}=\{(\tilde{x}_{m_1}^k,x_{m_2}^k,y^k)\},$$

and the full training dataset is

$\tilde{D}=\{D_c,\tilde{D}_{m_1},\tilde{D}_{m_2}\}.$

The immediate background is prior multimodal prompting based on missing-aware prompts, which are case-specific. In the two-modality setting, those prompts are $P_i$ for image-only inputs, $P_t$ for text-only inputs, and $m_2$0 for modality-complete inputs. For a general $m_2$1-modality task, the number of nonempty missingness cases is $m_2$2, so the requisite number of missing-aware prompts is also $m_2$3. The paper identifies two critical issues with that formulation: exponential prompt growth as the number of modalities increases, and lack of robustness when training and inference differ in their missing-modality settings (Jang et al., 2023).

The robustness issue is illustrated directly in the source material. If a training set consists of 100% image-modality and 30% text-modality samples, the modality-complete prompt $m_2$4 and image-only prompt $m_2$5 can be trained, while the text-only prompt $m_2$6 remains unlearned. A text-only test sample then depends on a prompt with little or no learned information. This establishes the central motivation for replacing missingness-indexed prompting with modality-indexed prompting.

2. Modality-specific prompting and the meaning of asymmetry

The core design change is the replacement of case-specific prompts with modality-specific prompts. For the two-modality image–text case, the method introduces an image-specific prompt $m_2$7 and a text-specific prompt $m_2$8. These are global learnable prompt tensors assigned to each modality, and they are described as learning knowledge from the pretrained backbone specific to the corresponding modality (Jang et al., 2023).

Prompt injection is performed at the input level. The prompts are attached to the input of the multi-head self-attention layer, following the standard prompt-tuning pattern in which learnable prompt tokens are concatenated to the input token sequence of the multimodal transformer. The paper does not describe layer-wise prompt insertion; it uses a single set of input-level prompt tokens at the multimodal transformer input.

The usage rule is modality-dependent rather than missingness-dependent. For an image-only sample, the model receives the real image, a text placeholder, and only $m_2$9. For a text-only sample, it receives the real text, an image placeholder, and only $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$0. For a modality-complete sample, there is no separate full-case prompt; instead, the effective prompt is formed by combining the two modality-specific prompts,

$D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$1

where $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$2 is implemented as element-wise summation. The paper states that this element-wise summation updates both $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$3 and $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$4 when the input is modality-complete.

The asymmetry lies in the parameterization. Each modality has its own prompt tensor with independent parameters, and prompt activation depends on which modality is present. The combination rule for the full case is symmetric in form, but it still combines distinct modality-specific components. The missing modality’s prompt is not used, even though placeholder inputs are retained to preserve architectural input format. In this sense, the method is asymmetric across modalities while remaining structurally simple.

3. Orthogonality as a diversity constraint

The orthogonality component is implemented as a cosine-similarity minimization between the flattened image-specific and text-specific prompts. Let $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$5 denote flattening of a prompt tensor to a vector. The orthogonality loss is

$D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$6

This is the absolute cosine similarity between $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$7 and $D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$8. Minimizing it encourages

$D_c=\{(x_{m_1}^i,x_{m_2}^i,y^i)\},$9

The paper refers to this as orthogonality, while also making clear that it is enforced through a soft penalty rather than an exact hard constraint. Its stated intuition is to reduce similarity between $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$0 and $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$1, maximize the gap between them, and ensure diversity of information learned from image and text (Jang et al., 2023).

In the reported two-modality setting, orthogonality is enforced only between the image-specific and text-specific prompts. There is no within-modality orthogonality because there are not multiple prompts per modality. The conceptual significance is that the full-case prompt $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$2 combines linearly independent prompt directions to a greater extent than would be expected without explicit regularization. This suggests a diversity-promoting role rather than a claim of semantic disentanglement in a strict sense.

4. Architecture, freezing strategy, and optimization

The implementation uses ViLT as the multimodal backbone, with a text encoder, an image encoder, and a multimodal transformer that fuses serialized visual and textual tokens. The text encoder, image encoder, and multimodal transformer are frozen. Training is restricted to the modality-specific prompts $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$3 and $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$4, together with task-specific layers consisting of the pooler and classifier (Jang et al., 2023).

Missingness is simulated with a missing rate of 70%, defined as the total proportion of modality instances removed across the dataset. Three train/test scenarios are considered: image 100% present with text 30% present, image 30% present with text 100% present, and image 65% present with text 65% present. Missing inputs are replaced by placeholders as described in the formal setup. A crucial consequence of modality-specific prompting is that a prompt is updated whenever its modality is present, regardless of the missingness status of the other modality. This makes the training procedure missing-modality-case-agnostic.

The total training objective is

$$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$5

where $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$6 is the task classification loss and $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$7 is the orthogonality loss. For MM-IMDb, $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$8 is binary cross-entropy and $$D_{m_1}=\{(x_{m_1}^j,y^j)\}, \qquad D_{m_2}=\{(x_{m_2}^k,y^k)\}.$$9. For UPMC Food-101, $\tilde{x}_{m_1}$0 is cross-entropy and $\tilde{x}_{m_1}$1. Optimization uses AdamW with learning rate $\tilde{x}_{m_1}$2, weight decay $\tilde{x}_{m_1}$3, batch size 6, and a single RTX 3090 GPU. No auxiliary contrastive or related losses are reported.

This training regime places the method squarely within parameter-efficient prompt tuning. The source notes that prompt-based finetuning in general requires less than 1% of the total parameters relative to full backbone finetuning, and the modality-specific design is intended to retain that benefit while avoiding combinatorial prompt proliferation.

5. Empirical behavior under matched and mismatched missingness

Experiments are reported on two image–text classification datasets. MM-IMDb is a movie genre multi-label classification benchmark with movie poster and plot or description as modalities, evaluated by F1-Macro. UPMC Food-101 is a food recipe recognition benchmark with image and text, evaluated by accuracy. The baselines are ViLT without prompts, missing-aware prompts, and modality-specific prompts with and without orthogonality (Jang et al., 2023).

When train and test missingness patterns match, missing-aware prompts often outperform prompt-free ViLT, but modality-specific prompts match or exceed them. On MM-IMDb with train and test both at image 30%, text 100%, ViLT records 37.63 F1-Macro, missing-aware prompts 47.18, and modality-specific prompts with $\tilde{x}_{m_1}$4 47.45. Under balanced missingness on MM-IMDb, with train and test both at 65%–65%, ViLT records 36.54, missing-aware prompts 40.84, and modality-specific prompts with $\tilde{x}_{m_1}$5 42.03. On UPMC Food-101 with train and test both at 65%–65%, the corresponding values are 68.60, 78.49, and 78.89.

The more consequential pattern appears under train–test mismatch. On MM-IMDb, training with image 100% and text 30% and testing with image 30% and text 100% yields 27.25 for ViLT, 20.17 for missing-aware prompts, and 34.11 for modality-specific prompts with orthogonality. On UPMC Food-101 under the same mismatch, ViLT records 43.67, missing-aware prompts 27.46, and modality-specific prompts with orthogonality 57.22. The reported interpretation is that modality-specific prompts deliver robustly uniform performance improvement across various missing-modality scenarios, degrading more gracefully when inference missingness differs from training.

The ablation study on MM-IMDb isolates the contribution of modality specificity and orthogonality. For train image 100%, text 30% and test image 30%, text 100%, performance progresses from 27.25 for ViLT to 32.11 for ViLT plus modality-specific prompts and to 34.11 after adding $\tilde{x}_{m_1}$6. For train image 30%, text 100% and test 65%–65%, the progression is 34.47, 41.24, and 42.17. The paper concludes that modality-specific prompts improve both performance and robustness for unseen missing-modality cases, and that the orthogonality term further guides $\tilde{x}_{m_1}$7 and $\tilde{x}_{m_1}$8 to acquire distinct information from each modality.

6. Scalability, limitations, and related terminology

A central scalability claim is the reduction in prompt count from exponential to linear in the number of modalities. Missing-aware prompting requires $\tilde{x}_{m_1}$9 prompts for an $\tilde{x}_{m_2}$0-modality task because each nonempty missingness pattern receives its own prompt. By contrast, the modality-specific design uses one prompt per modality, yielding $\tilde{x}_{m_2}$1 prompts. In the explicit two-modality case, the modality-complete prompt is not separately parameterized but is obtained by element-wise summation of the image-specific and text-specific prompts (Jang et al., 2023).

The reported experiments are limited to two modalities, image and text, and to classification tasks. The paper identifies extension to more modality settings and tasks as future work, specifically mentioning action recognition with RGB, depth, and IR as inputs. A plausible implication is that the prompt-count advantage becomes more significant as modality count grows. A further plausible implication, stated in the source discussion as an implicit limitation, is that pairwise orthogonality constraints would scale as $\tilde{x}_{m_2}$2 in the number of regularization terms even if the number of prompts remains linear. The same discussion also notes that orthogonality across high-dimensional prompts may be only a crude proxy for diversity rather than a guarantee of semantic disentanglement.

Two common misconceptions are addressed directly by the design. First, the method is not a prompt-per-case system; it is a prompt-per-modality system. Second, orthogonality is not exact mutual orthogonality in a geometric or algebraic hard-constraint sense, but approximate orthogonality enforced by minimizing absolute cosine similarity. The phrase “Asymmetric Orthogonal Prompting” therefore names a structural pattern rather than a formal framework title used by the source paper.

The terminology also benefits from disambiguation. In a distinct mathematical literature, “orthogonality” and “asymmetry” refer to orthogonal duality functions for asymmetric particle systems such as ASEP$\tilde{x}_{m_2}$3 and ASIP$\tilde{x}_{m_2}$4, where multivariate $\tilde{x}_{m_2}$5-Krawtchouk and $\tilde{x}_{m_2}$6-Meixner polynomials serve as orthogonal duality functions (Carinci et al., 2020). That usage concerns observables, reversible measures, and self-duality in interacting particle systems, not multimodal prompt tuning. Within contemporary multimodal learning, however, the phrase most naturally denotes the design pattern exemplified by modality-specific prompts plus an orthogonality regularizer for robust learning with missing modalities.