Negative-Guided Multimodal Prompting Alignment

• The paper introduces NMPA, a multimodal alignment mechanism that bridges textual rationales with visual prompts for improved Ov-GSR.
• It employs a two-level alignment, using scene-level glimpse and instance-level gaze rationales with hard negative guidance to sharpen discrimination.
• Empirical results show significant gains on seen, rare, and unseen situations compared to baselines and direct distillation methods.

Searching arXiv for the cited papers to ground the article in current preprints. Negative-Guided Multimodal Prompting Alignment (NMPA) is a multimodal alignment mechanism introduced within the Multimodal Interactive Prompt Distillation (MIPD) framework for Open-vocabulary Grounded Situation Recognition (Ov-GSR). In that setting, NMPA aligns textual rationales generated by a teacher Multimodal LLM (MLLM) with visual prompts derived from teacher visual features, while using negative rationales as informative hard negatives. Its defining structure is two-level: glimpse rationales are aligned with scene-aware prompts for holistic activity semantics, and gaze rationales are aligned with instance-perception prompts for entity- and role-level grounding. The aligned teacher knowledge is then distilled into a smaller frozen CLIP-based student, with reported gains on seen, rare, and unseen situations (Cai et al., 19 Jul 2025).

1. Problem setting and conceptual role

NMPA arises from the problem of distilling knowledge from a large MLLM teacher into a smaller Ov-GSR student so that the student can recognize seen, rare, and unseen situations. The teacher is InstructBLIP in experiments, and the student is a frozen CLIP-based GSR model. The motivating diagnosis is that a standard student GSR model is usually closed-set and overfits to seen situations, lacks the generalization and commonsense-like semantic richness of MLLMs, and tends to be biased toward frequent or seen classes, especially under class imbalance (Cai et al., 19 Jul 2025).

Within MIPD, NMPA is the bridge between semantic rationales and visual features/prompts. The paper states that directly distilling only raw teacher outputs is not enough; instead, the teacher’s knowledge should be organized into scene-level holistic semantics and instance-level perceptual semantics. NMPA is therefore designed to align positive semantic descriptions with the correct visual content and to push away semantically confusing negatives. In the paper’s formulation, this is especially important for Ov-GSR because unseen and rare situations often differ by subtle semantic details (Cai et al., 19 Jul 2025).

The resulting interpretation of NMPA is narrower than generic text–image matching. It is a structure-aware alignment module embedded in a distillation pipeline, and its multimodal scope is explicitly split between global activity understanding and local role/entity understanding. This makes NMPA a specialized alignment mechanism for grounded situation recognition rather than a general-purpose prompt-tuning method.

2. Rationales, prompts, and representational split

The method adopts a human-like reasoning split between glimpse and gaze rationales. Glimpse rationales are coarse, scene-level descriptions of “what is happening,” while gaze rationales are detailed, instance-level descriptions of “which entities are involved and how.” These are represented as $\mathbf{P}_{gli}$ and $\mathbf{P}_{gaz}$, and the paper states that the rationales are text-encoded as

$\mathbf{P} \in \mathbb{R}^{L \times D},$

where $L$ is rationale length and $D$ is embedding dimension (Cai et al., 19 Jul 2025).

The Judgmental Rationales Generator (JRG) constructs four rationale types:

$$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$

Positive glimpse and gaze rationales are generated by an MLLM from the input image $I$, scored by an LLM-judge against the target situation $s$, and iteratively refined if the score is below a threshold. The threshold is set to $N=8$ in practice. Negative rationales are then created by modifying the positive or pseudo rationales while preserving semantic similarity, specifically by changing activity, entities, and attributes such as action, object, and pattern. The resulting negatives are therefore hard negatives rather than arbitrary mismatches (Cai et al., 19 Jul 2025).

NMPA aligns these rationales to two corresponding prompt families. The scene-aware prompt is

$$\mathbf{P}_{sce} \in \mathbb{R}^{D \times \left( 2p(H + W - 2p) \right)},$$

and is appended to the edges of the visual feature map from the frozen teacher encoder:

$\mathbf{P}_{gaz}$0

The instance-perception prompt is

$\mathbf{P}_{gaz}$1

based on instance coordinates

$\mathbf{P}_{gaz}$2

The paper characterizes $\mathbf{P}_{gaz}$3 as absorbing holistic scene visual knowledge and glimpse semantic knowledge, and $\mathbf{P}_{gaz}$4 as absorbing regional visual cues and gaze semantic knowledge (Cai et al., 19 Jul 2025).

Component	Associated knowledge	Representation
$\mathbf{P}_{gaz}$5 / $\mathbf{P}_{gaz}$6	Glimpse / gaze rationale	$\mathbf{P}_{gaz}$7
$\mathbf{P}_{gaz}$8	Holistic scene visual knowledge	Edge-attached prompt
$\mathbf{P}_{gaz}$9	Entity-level regional visual cues	Box-conditioned prompt

This factorization is central to the method’s semantics. Rather than treating prompt alignment as a single channel, NMPA decomposes it into a global branch for activity semantics and a local branch for role grounding.

3. Alignment mechanism and the meaning of “negative-guided”

NMPA uses cross-attention to align positive rationales with the corresponding visual prompts. For the glimpse branch, the paper defines

$\mathbf{P} \in \mathbb{R}^{L \times D},$0

and for the gaze branch,

$\mathbf{P} \in \mathbb{R}^{L \times D},$1

Here, $\mathbf{P} \in \mathbb{R}^{L \times D},$2 and $\mathbf{P} \in \mathbb{R}^{L \times D},$3 are projection parameters, while $\mathbf{P} \in \mathbb{R}^{L \times D},$4 and $\mathbf{P} \in \mathbb{R}^{L \times D},$5 are simple cross-attention operations (Cai et al., 19 Jul 2025).

The operational meaning of negative-guided is given by a separate contrastive-style term:

$\mathbf{P} \in \mathbb{R}^{L \times D},$6

where $\mathbf{P} \in \mathbb{R}^{L \times D},$7 is cosine similarity. The paper’s explanation is that the model should not collapse positive and negative representations, but instead use negative rationales as informative hard negatives, while keeping them close enough to remain meaningful. The negative rationale is explicitly described as staying closely aligned with positive text features while introducing variations in attribute information, which helps distinguish unseen and rare situations (Cai et al., 19 Jul 2025).

This yields a precise interpretation of the module. Positive rationales provide semantic anchoring, while negative rationales define nearby semantic alternatives that must be rejected. NMPA therefore does not merely maximize agreement across modalities; it structures agreement by pairing each positive alignment path with semantically adjacent exclusions. This suggests that the method’s discriminative effect comes from local semantic boundary shaping rather than from undifferentiated cross-modal attraction alone.

4. Optimization, distillation, and inference

The paper states a general distillation form in which student parameters are optimized using both student–teacher distillation loss and supervised situation recognition loss. The explicit task loss for situation recognition is

$\mathbf{P} \in \mathbb{R}^{L \times D},$8

where $\mathbf{P} \in \mathbb{R}^{L \times D},$9 is cross-entropy, $L$0 is the activity prediction, and $L$1 is the role/entity prediction (Cai et al., 19 Jul 2025).

The feature distillation term is an $L$2 loss:

$L$3

where $L$4 and $L$5 are the aligned teacher features, and $L$6 and $L$7 are the student visual and role features. The full objective is

$L$8

with $L$9 the bounding-box localization loss (Cai et al., 19 Jul 2025).

The end-to-end pipeline is described as follows. An input image enters a frozen teacher MLLM and the student Ov-GSR model. The teacher generates pseudo glimpse and gaze rationales. JRG judges, refines, and converts them into positive and negative rationales. The teacher’s visual features are equipped with $D$0 for scene-level knowledge and $D$1 for instance-level knowledge. NMPA aligns $D$2 with $D$3 and $D$4 with $D$5 using cross-attention, and contrasts them against $D$6 and $D$7. The aligned teacher knowledge is then distilled into the student with classification loss, $D$8 distillation loss, negative-guided loss, and box regression loss. At inference, rationales are no longer needed; the student alone predicts the situation and grounded roles (Cai et al., 19 Jul 2025).

The paper explicitly describes scene-aware prompts as a glimpse-based knowledge distiller and instance-perception prompts as a gaze-based knowledge distiller. This division clarifies that NMPA is not only a prompt-alignment device but also a modality-structured distillation interface.

5. Empirical evidence and ablation findings

The reported empirical results are organized around improvements over a baseline and over direct distillation. In Table 4, the comparison is Baseline, w/ Dist, and w/ MIPD. On all verb/value, the scores are 36.78 / 29.44, 38.76 / 31.54, and 41.96 / 34.11, respectively. On unseen, the scores are 2.80 / 1.85, 5.60 / 3.47, and 7.40 / 4.08. The paper presents this as evidence that MIPD improves over direct rationale distillation and that prompt alignment with negative guidance matters (Cai et al., 19 Jul 2025).

Table 5 studies prompt combinations involving $D$9, $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$0, $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$1, and $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$2. The key finding reported is that using all prompts together gives the best performance:

• All: 41.96 / 34.11
• Rare: 28.30 / 22.37
• Unseen: 7.40 / 4.08

The paper interprets this as showing that $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$3 improves scene/activity understanding, $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$4 improves entity recognition, and the full combination best aligns holistic and instance-level knowledge (Cai et al., 19 Jul 2025).

Table 6 compares the proposed prompt design with PadPrompt:

• Baseline: 36.78 / 29.44
• Pad: 39.50 / 31.95
• Ours: 41.96 / 34.11

Table 7 isolates the contribution of negative-guided loss. Without good negative rationale guidance, the scores are:

• All: 40.63 / 32.78
• Rare: 26.90 / 20.74
• Unseen: 6.60 / 3.83

With the full model, they are:

• All: 41.96 / 34.11
• Rare: 28.30 / 22.37
• Unseen: 7.40 / 4.08

Table 8 examines rationale quality. The reported settings are:

• Pseudo: 37.15 / 29.31
• Refined D5: 40.26 / 32.83
• Refined G8: 41.40 / 33.35
• Refined D8: 41.96 / 34.11

The paper states that pseudo rationales are weaker, refined rationales help, and score 8 refinement is best (Cai et al., 19 Jul 2025).

Setting	Reported scores
Baseline	36.78 / 29.44
w/ Dist	38.76 / 31.54
w/ MIPD	41.96 / 34.11
Unseen Baseline	2.80 / 1.85
Unseen w/ Dist	5.60 / 3.47
Unseen w/ MIPD	7.40 / 4.08

Taken together, these ablations locate NMPA’s contribution in three places: the prompt design, the use of high-quality refined rationales, and the negative-guided loss. The qualitative analysis reported in the paper further states that MIPD correctly predicts the main activity, semantic roles, and grounding boxes in unseen and rare cases where competing methods misclassify the activity and/or roles (Cai et al., 19 Jul 2025).

6. Relation to other negative-guided multimodal methods

A broader research pattern places NMPA within a family of negative-guided multimodal alignment strategies. In Multimodal Negative Learning (MNL), the dominant modality guides the weak modality by suppressing non-target classes rather than by forcing full target-class imitation. Its core claim is that weak modalities often benefit more from ruling out wrong classes than from directly increasing the correct-class probability, and its objective applies guidance only to the complement of the one-hot label $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$5 rather than to the entire class distribution (Gong et al., 23 Oct 2025). This is not NMPA, but it shares the same negative-guided logic: alignment is mediated through exclusion and margin shaping rather than only through positive matching.

A related design appears in zero-shot video action recognition, where semantic alignment is performed with both positive prompts and class-dependent negative prompts such as “This video is NOT [CLASS$$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$6]”. The model computes a negative prompt loss

$$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$7

with $$\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).$$8, and reports that adding negative prompt loss changes HMDB from 46.0 to 54.8 and UCF from 71.3 to 80.5 before the full objective reaches 55.2 on HMDB-51 and 82.2 on UCF-101 (Wang et al., 18 Apr 2026). That work explicitly frames negative prompts as modeling “non-class” semantics.

In remote sensing open-vocabulary detection, RS-MPOD does not instantiate NMPA directly, but its implications section identifies a broader direction toward negative-guided prompting and stronger multimodal prompt alignment, particularly under semantic ambiguity and distribution shift (Yang et al., 2 Feb 2026). This suggests that NMPA belongs to a wider methodological trend in which multimodal systems rely not only on positive semantic grounding but also on structured negative evidence to stabilize alignment under open-vocabulary, cross-domain, or imbalanced conditions.

Across these related works, the common principle is that negative information is not treated as auxiliary noise. Instead, it becomes a first-class alignment signal: in NMPA through near-miss rationale contrast, in MNL through non-target suppression, and in zero-shot video recognition through explicit negative prompt embeddings. The specific implementation differs by task, but the underlying objective is closely aligned—preserve informative multimodal structure while improving discrimination against plausible alternatives.