---
title: Negative-Guided Multimodal Prompting Alignment
url: https://www.emergentmind.com/topics/negative-guided-multimodal-prompting-alignment-nmpa
type: topic
---

# Negative-Guided Multimodal Prompting Alignment

Searching arXiv for the cited papers to ground the article in current preprints.
Negative-Guided Multimodal Prompting Alignment (NMPA) is a multimodal alignment mechanism introduced within the Multimodal Interactive Prompt Distillation (MIPD) framework for Open-vocabulary Grounded Situation Recognition (Ov-GSR). In that setting, NMPA aligns textual rationales generated by a teacher Multimodal Large Language Model (MLLM) with visual prompts derived from teacher visual features, while using negative rationales as informative hard negatives. Its defining structure is two-level: glimpse rationales are aligned with scene-aware prompts for holistic activity semantics, and gaze rationales are aligned with instance-perception prompts for entity- and role-level grounding. The aligned teacher knowledge is then distilled into a smaller frozen CLIP-based student, with reported gains on seen, rare, and unseen situations [2507.14686].

## 1. Problem setting and conceptual role

NMPA arises from the problem of distilling knowledge from a large MLLM teacher into a smaller Ov-GSR student so that the student can recognize **seen**, **rare**, and **unseen** situations. The teacher is **InstructBLIP in experiments**, and the student is a **frozen CLIP-based GSR model**. The motivating diagnosis is that a standard student GSR model is usually **closed-set** and overfits to seen situations, lacks the **generalization** and **commonsense-like semantic richness** of MLLMs, and tends to be biased toward frequent or seen classes, especially under class imbalance [2507.14686].

Within MIPD, NMPA is the bridge between **semantic rationales** and **visual features/prompts**. The paper states that directly distilling only raw teacher outputs is not enough; instead, the teacher’s knowledge should be organized into **scene-level holistic semantics** and **instance-level perceptual semantics**. NMPA is therefore designed to align positive semantic descriptions with the correct visual content and to push away semantically confusing negatives. In the paper’s formulation, this is especially important for Ov-GSR because unseen and rare situations often differ by subtle semantic details [2507.14686].

The resulting interpretation of NMPA is narrower than generic text–image matching. It is a structure-aware alignment module embedded in a distillation pipeline, and its multimodal scope is explicitly split between global activity understanding and local role/entity understanding. This makes NMPA a specialized alignment mechanism for grounded situation recognition rather than a general-purpose prompt-tuning method.

## 2. Rationales, prompts, and representational split

The method adopts a human-like reasoning split between **glimpse** and **gaze** rationales. **Glimpse rationales** are coarse, scene-level descriptions of “what is happening,” while **gaze rationales** are detailed, instance-level descriptions of “which entities are involved and how.” These are represented as $\mathbf{P}_{gli}$ and $\mathbf{P}_{gaz}$, and the paper states that the rationales are text-encoded as
$$
\mathbf{P} \in \mathbb{R}^{L \times D},
$$
where $L$ is rationale length and $D$ is embedding dimension [2507.14686].

The **Judgmental Rationales Generator (JRG)** constructs four rationale types:
$$
\mathbf{P}_{gli+}, \mathbf{P}_{gaz+}, \mathbf{P}_{gli-}, \mathbf{P}_{gaz-} = \text{JRG}(I, Intructions).
$$
Positive glimpse and gaze rationales are generated by an MLLM from the input image $I$, scored by an **LLM-judge** against the target situation $s$, and iteratively refined if the score is below a threshold. The threshold is set to **$N=8$** in practice. Negative rationales are then created by modifying the positive or pseudo rationales while preserving semantic similarity, specifically by changing activity, entities, and attributes such as action, object, and pattern. The resulting negatives are therefore **hard negatives** rather than arbitrary mismatches [2507.14686].

NMPA aligns these rationales to two corresponding prompt families. The **scene-aware prompt** is
$$
\mathbf{P}_{sce} \in \mathbb{R}^{D \times \left( 2p(H + W - 2p) \right)},
$$
and is appended to the **edges** of the visual feature map from the frozen teacher encoder:
$$
\mathbf{X}_T = T_{model}(I), \qquad \mathbf{X}_T \in \mathbb{R}^{H \times W \times D}.
$$
The **instance-perception prompt** is
$$
\mathbf{P}_{ins} \in \mathbb{R}^{D \times b_i \times H' \times W'},
$$
based on instance coordinates
$$
b_i \in \mathbb{R}^4.
$$
The paper characterizes $\mathbf{P}_{sce}$ as absorbing holistic scene visual knowledge and glimpse semantic knowledge, and $\mathbf{P}_{ins}$ as absorbing regional visual cues and gaze semantic knowledge [2507.14686].

| Component | Associated knowledge | Representation |
|---|---|---|
| $\mathbf{P}_{gli}$ / $\mathbf{P}_{gaz}$ | Glimpse / gaze rationale | $\mathbf{P} \in \mathbb{R}^{L \times D}$ |
| $\mathbf{P}_{sce}$ | Holistic scene visual knowledge | Edge-attached prompt |
| $\mathbf{P}_{ins}$ | Entity-level regional visual cues | Box-conditioned prompt |

This factorization is central to the method’s semantics. Rather than treating prompt alignment as a single channel, NMPA decomposes it into a global branch for activity semantics and a local branch for role grounding.

## 3. Alignment mechanism and the meaning of “negative-guided”

NMPA uses cross-attention to align positive rationales with the corresponding visual prompts. For the glimpse branch, the paper defines
$$
\mathbf{X}^{glimpse}_T = \delta_{glimpse}([\mathbf{X}_T + \mathbf{P}_{sce}]W^q, \mathbf{P}_{gli+}W^{kv}),
$$
and for the gaze branch,
$$
\mathbf{X}^{gaze}_T = \delta_{gaze}([\mathbf{X}_T + \mathbf{P}_{ins}]W^q, \mathbf{P}_{gaz+}W^{kv}).
$$
Here, $W^q$ and $W^{kv}$ are projection parameters, while $\delta_{glimpse}$ and $\delta_{gaze}$ are simple cross-attention operations [2507.14686].

The operational meaning of **negative-guided** is given by a separate contrastive-style term:
$$
\mathcal{L}_{neg} = - [\text{sim}(\mathbf{X}^{glimpse}_T, \mathbf{P}_{gli-}) + \text{sim}(\mathbf{X}^{gaze}_T, \mathbf{P}_{gaz-})],
$$
where $\text{sim}$ is cosine similarity. The paper’s explanation is that the model should **not collapse** positive and negative representations, but instead use negative rationales as **informative hard negatives**, while keeping them close enough to remain meaningful. The negative rationale is explicitly described as staying closely aligned with positive text features while introducing variations in attribute information, which helps distinguish unseen and rare situations [2507.14686].

This yields a precise interpretation of the module. Positive rationales provide semantic anchoring, while negative rationales define nearby semantic alternatives that must be rejected. NMPA therefore does not merely maximize agreement across modalities; it structures agreement by pairing each positive alignment path with semantically adjacent exclusions. This suggests that the method’s discriminative effect comes from local semantic boundary shaping rather than from undifferentiated cross-modal attraction alone.

## 4. Optimization, distillation, and inference

The paper states a general distillation form in which student parameters are optimized using both **student–teacher distillation loss** and **supervised situation recognition loss**. The explicit task loss for situation recognition is
$$
\mathcal{L}_{sit} = \mathcal{L}_{ce}(\mathbf{\varepsilon}_s^{v},v^b) + \sum_{\text{f}_r \in \mathcal{F}_v ^b} \mathcal{L}_{ce}(\mathbf{\varepsilon}_s^{r},\text{f}_r),
$$
where $\mathcal{L}_{ce}$ is cross-entropy, $\mathbf{\varepsilon}_s^{v}$ is the activity prediction, and $\mathbf{\varepsilon}_s^{r}$ is the role/entity prediction [2507.14686].

The feature distillation term is an $L_1$ loss:
$$
\mathcal{L}_{dis}  = \left| \mathbf{X}_T^{glimpse} - \mathbf{X}_S \right| + \left| \mathbf{X}_T^{gaze} - \mathbf{X}_S^r \right|,
$$
where $\mathbf{X}_T^{glimpse}$ and $\mathbf{X}_T^{gaze}$ are the aligned teacher features, and $\mathbf{X}_S$ and $\mathbf{X}_S^r$ are the student visual and role features. The full objective is
$$
\mathcal{L} = \mathcal{L}_{neg}+\mathcal{L}_{sit}+\mathcal{L}_{dis} + \mathcal{L}_{box},
$$
with $\mathcal{L}_{box}$ the bounding-box localization loss [2507.14686].

The end-to-end pipeline is described as follows. An input image enters a frozen teacher MLLM and the student Ov-GSR model. The teacher generates pseudo glimpse and gaze rationales. JRG judges, refines, and converts them into positive and negative rationales. The teacher’s visual features are equipped with $\mathbf{P}_{sce}$ for scene-level knowledge and $\mathbf{P}_{ins}$ for instance-level knowledge. NMPA aligns $\mathbf{P}_{gli+}$ with $\mathbf{P}_{sce}$ and $\mathbf{P}_{gaz+}$ with $\mathbf{P}_{ins}$ using cross-attention, and contrasts them against $\mathbf{P}_{gli-}$ and $\mathbf{P}_{gaz-}$. The aligned teacher knowledge is then distilled into the student with classification loss, $L_1$ distillation loss, negative-guided loss, and box regression loss. At inference, rationales are no longer needed; the student alone predicts the situation and grounded roles [2507.14686].

The paper explicitly describes scene-aware prompts as a **glimpse-based knowledge distiller** and instance-perception prompts as a **gaze-based knowledge distiller**. This division clarifies that NMPA is not only a prompt-alignment device but also a modality-structured distillation interface.

## 5. Empirical evidence and ablation findings

The reported empirical results are organized around improvements over a baseline and over direct distillation. In Table 4, the comparison is **Baseline**, **w/ Dist**, and **w/ MIPD**. On **all verb/value**, the scores are **36.78 / 29.44**, **38.76 / 31.54**, and **41.96 / 34.11**, respectively. On **unseen**, the scores are **2.80 / 1.85**, **5.60 / 3.47**, and **7.40 / 4.08**. The paper presents this as evidence that MIPD improves over direct rationale distillation and that prompt alignment with negative guidance matters [2507.14686].

Table 5 studies prompt combinations involving $\mathbf{P}_{sce}$, $\mathbf{P}_{ins}$, $\mathbf{P}_{gli}$, and $\mathbf{P}_{gaz}$. The key finding reported is that using **all prompts together** gives the best performance:
- **All**: **41.96 / 34.11**
- **Rare**: **28.30 / 22.37**
- **Unseen**: **7.40 / 4.08**

The paper interprets this as showing that $\mathbf{P}_{sce} + \mathbf{P}_{gli}$ improves scene/activity understanding, $\mathbf{P}_{ins} + \mathbf{P}_{gaz}$ improves entity recognition, and the full combination best aligns holistic and instance-level knowledge [2507.14686].

Table 6 compares the proposed prompt design with PadPrompt:
- **Baseline**: **36.78 / 29.44**
- **Pad**: **39.50 / 31.95**
- **Ours**: **41.96 / 34.11**

Table 7 isolates the contribution of negative-guided loss. Without good negative rationale guidance, the scores are:
- **All**: **40.63 / 32.78**
- **Rare**: **26.90 / 20.74**
- **Unseen**: **6.60 / 3.83**

With the full model, they are:
- **All**: **41.96 / 34.11**
- **Rare**: **28.30 / 22.37**
- **Unseen**: **7.40 / 4.08**

Table 8 examines rationale quality. The reported settings are:
- **Pseudo**: **37.15 / 29.31**
- **Refined D5**: **40.26 / 32.83**
- **Refined G8**: **41.40 / 33.35**
- **Refined D8**: **41.96 / 34.11**

The paper states that pseudo rationales are weaker, refined rationales help, and **score 8 refinement is best** [2507.14686].

| Setting | Reported scores |
|---|---|
| Baseline | 36.78 / 29.44 |
| w/ Dist | 38.76 / 31.54 |
| w/ MIPD | 41.96 / 34.11 |
| Unseen Baseline | 2.80 / 1.85 |
| Unseen w/ Dist | 5.60 / 3.47 |
| Unseen w/ MIPD | 7.40 / 4.08 |

Taken together, these ablations locate NMPA’s contribution in three places: the prompt design, the use of high-quality refined rationales, and the negative-guided loss. The qualitative analysis reported in the paper further states that MIPD correctly predicts the main activity, semantic roles, and grounding boxes in unseen and rare cases where competing methods misclassify the activity and/or roles [2507.14686].

## 6. Relation to other negative-guided multimodal methods

A broader research pattern places NMPA within a family of negative-guided multimodal alignment strategies. In **Multimodal Negative Learning (MNL)**, the dominant modality guides the weak modality by suppressing non-target classes rather than by forcing full target-class imitation. Its core claim is that weak modalities often benefit more from ruling out wrong classes than from directly increasing the correct-class probability, and its objective applies guidance only to the complement of the one-hot label $\overline y$ rather than to the entire class distribution [2510.20877]. This is not NMPA, but it shares the same negative-guided logic: alignment is mediated through exclusion and margin shaping rather than only through positive matching.

A related design appears in zero-shot video action recognition, where semantic alignment is performed with both positive prompts and **class-dependent negative prompts** such as “This video is NOT [CLASS\(_k\)]”. The model computes a negative prompt loss
$$
\mathcal{L}_{N} = \sum_i \mathrm{CE}(P_{N,i}, \hat{y}_i),
$$
with $\lambda_N = 0.1$, and reports that adding negative prompt loss changes HMDB from **46.0** to **54.8** and UCF from **71.3** to **80.5** before the full objective reaches **55.2** on HMDB-51 and **82.2** on UCF-101 [2604.17062]. That work explicitly frames negative prompts as modeling **“non-class” semantics**.

In remote sensing open-vocabulary detection, RS-MPOD does not instantiate NMPA directly, but its implications section identifies a broader direction toward **negative-guided prompting** and stronger **multimodal prompt alignment**, particularly under semantic ambiguity and distribution shift [2602.01954]. This suggests that NMPA belongs to a wider methodological trend in which multimodal systems rely not only on positive semantic grounding but also on structured negative evidence to stabilize alignment under open-vocabulary, cross-domain, or imbalanced conditions.

Across these related works, the common principle is that negative information is not treated as auxiliary noise. Instead, it becomes a first-class alignment signal: in NMPA through near-miss rationale contrast, in MNL through non-target suppression, and in zero-shot video recognition through explicit negative prompt embeddings. The specific implementation differs by task, but the underlying objective is closely aligned—preserve informative multimodal structure while improving discrimination against plausible alternatives.

Source: https://www.emergentmind.com/topics/negative-guided-multimodal-prompting-alignment-nmpa