Expression-Aware Visual Prompting Module

• Expression-Aware Visual Prompting Module is a neural architecture component that adaptively modulates visual features using expression-specific signals.
• It employs CAM-based masking, learnable patch prompts, and global/local attention to focus on subtle and task-relevant visual cues.
• Its integration in FER, empathetic tutoring, and visual-language models significantly improves performance metrics and task-specific accuracy.

An Expression-Aware Visual Prompting Module is a class of neural architecture components that adaptively modulate visual input—typically at the feature or token level—in response to expression-relevant or prompt-specific signals. Its purpose is to enhance downstream models’ sensitivity to salient details (e.g., subtle facial cues or prompt-relevant objects) that standard backbone encoders, even with large-scale pretraining, tend to underexploit. These modules appear in several recent frameworks spanning open-set video-based facial expression recognition, empathetic LLM tutoring grounded in facial cues, and dynamic multimodal visual-LLMs, each deploying distinct mechanisms for extracting, weighting, and integrating expression- or prompt-conditioned visual evidence.

1. Core Principles and Motivation

Expression-aware visual prompting addresses a recognized limitation in fixed-encoder architectures—particularly foundation vision-LLMs such as CLIP—where model capacity is diffusely allocated, often missing expression-rich or prompt-relevant content. This module family is motivated by the observation that:

• Subtle visual states (e.g., nuanced emotions or object properties) often concentrate in spatially or semantically sparse regions.
• Prompt or context conditioning (defined by user queries, class labels, or inferred states) provides additional constraints that can focus model attention and computational resources.
• Standard adapters or vanilla cross-attention blocks typically fail to deliver sufficiently targeted modulation, treating all patches or tokens identically regardless of task demands (Zhang et al., 2024).

The mechanism, therefore, centers on learning or inferring spatial, temporal, or semantic regions linked to critical expressive or prompt-directed cues and injecting learnable, class- or prompt-specific tokens or modulations at those points.

2. Architecture: General Patterns and Instantiations

Recent expressions of this concept include the Human Expression-Sensitive Prompting (HESP) visual module for OV-FER (Liu et al., 2024), the AU-based frame selection/description pipeline for LLM tutoring (Feng et al., 10 Mar 2026), and the prompt-aware adapter for MLLMs (Zhang et al., 2024).

HESP Visual Prompting Module (Liu et al., 2024):

• Input Processing: Each video $V_n$ is sampled into $N$ frames $\{V_n(1), \ldots, V_n(N)\}$.
• Expression-Sensitive Mask: The first frame is fed through a frozen CLIP encoder $\phi_V$, extracting a Class Activation Map (CAM). CAM is thresholded to a binary mask $M_n$ highlighting expression-relevant ($l \times l$) regions.
• Learnable Visual Prompts: For each known class $k$, a tensor $\delta_V(k)$ (shape $3 \times l \times l$) is trained. At forward pass, mask $M_n$ selects the target patch where $N$0 is injected in place of the original pixels.
• Prompted Frame Synthesis:

$N$1

• Frame Encoding: Each $N$2 is passed through $N$3.
• Temporal Aggregation: Video-level feature is $N$4.

Prompt-Aware Adapter (Zhang et al., 2024):

• Global Attention: Projects global text embedding to vision space, concatenates to visual patches, and performs self-attention for scene-level prompt guidance.
• Local Attention: Projects both patch-level visual features and per-token text features into a shared space, computes a normalized similarity matrix (ℝSOFTMAX), and reweights visual features per their prompt relevance.

LLM Empathetic Tutoring Pipeline (Feng et al., 10 Mar 2026):

• Expression Feature Extraction: For a face video, an Action Unit Estimation Model (AUM) predicts an AU intensity vector per frame.
• Temporal Pooling or Saliency Scoring: Per-frame AU scores identify salient (peak-expression) frames or aggregate intensity descriptors.
• Integration: AU-derived signals are used to (i) prepend textual AU-based natural language descriptions to LLM prompts or (ii) select peak frames for visual input into MLLMs.

3. Mathematical Formulation and Mechanisms

A representative instantiation (HESP (Liu et al., 2024)) uses hard thresholded CAM-based masks as binary attention indicators. At each frame,

$N$5

where $N$6 is a binary mask (derived from CAM), $N$7 the raw frame, and $N$8 the class-specific patch prompt.

A soft-attention variant (noted but not default) applies a per-frame attention map via a softmax over flattened CAM scores, then mixes $N$9 and $\{V_n(1), \ldots, V_n(N)\}$0 per $\{V_n(1), \ldots, V_n(N)\}$1 intensity.

In prompt-aware adapters (Zhang et al., 2024), global and local attention mechanisms are defined mathematically:

• Global cross-modal token $\{V_n(1), \ldots, V_n(N)\}$2, with $\{V_n(1), \ldots, V_n(N)\}$3 as global prompt embedding; concatenated with vision tokens for multi-head self-attention.
• Local weighting uses pairwise similarity between all visual and prompt tokens: $\{V_n(1), \ldots, V_n(N)\}$4; aggregated patch importance $\{V_n(1), \ldots, V_n(N)\}$5, finally $\{V_n(1), \ldots, V_n(N)\}$6 produces prompt-adaptive output tokens.

4. Integration with Cross-Modal Pipelines

Combination with Textual Prompting:

In HESP, textual learnable prompts $\{V_n(1), \ldots, V_n(N)\}$7 are encoded by CLIP’s text encoder, producing a feature bank $\{V_n(1), \ldots, V_n(N)\}$8. Visual and textual branches generate class probability distributions, which are averaged: $\{V_n(1), \ldots, V_n(N)\}$9 This joint matching forces agreement between modalities, improving open-set rejection and closed-set classification.

Adapters for MLLMs:

Prompt-adaptive visual tokens from the adapter are introduced as special tokens in the LLM input stream, compatible with standard token embedding dimensions. This allows flexible insertion of visually focused information at the sequence level guided by the prompt (Zhang et al., 2024).

Facial-Expression-Grounded LLMs:

In empathetic tutoring, AU-derived descriptors are either injected as prepended AU→text natural language or as selected peak-attention frames, directly affecting the LLM’s output distribution and empathetic responsiveness (Feng et al., 10 Mar 2026).

5. Empirical Performance and Ablation Results

HESP (OV-FER) (Liu et al., 2024):

• Metrics: AUROC and OSCR; strong improvements: AUROC +17.93% (absolute), OSCR +106.18% vs. CLIP+ARPL baseline.
• Visual mask ablation: visual-only prompt boosts AUROC by +10.7%, OSCR +41.5%; combined with text, results are additive.
• Spatial mask analysis: $\phi_V$0 consistently localizes mouth and eye regions, critical for emotion reasoning.
• t-SNE visualization: demonstrates cluster compaction and better separation of known/unknown emotion classes.
• Negative representation and fusion losses further regularize and harmonize modality agreement.

Prompt-Aware Adapter (Zhang et al., 2024):

• COCO-QA performance: object (+4.6 pts), counting (+4.4), color (+10.2), and position (+4.7) over prompt-unaware baseline.
• MME perceptual tasks: perception score up 75.2 points; removal of global/local attention each drops performance sharply.
• Qualitative examples show adaptive focus on prompt-queried regions (counting objects, detecting color/position).

Empathetic LLM Tutoring (Feng et al., 10 Mar 2026):

• AU-based prompting outperforms text-only and random-frame baselines on empathy (average human pairwise score $\phi_V$1, $\phi_V$2).
• No decrement in pedagogical or textual cue responsiveness.
• Model-dependent tradeoffs for text-vs-image conditioning; GPT-5.1 shows stronger effects for AU→text, Claude and Gemini prefer frame-level visual conditioning in some trials.

6. Domain Applications and Limitations

Expression-aware visual prompting modules are central to domains demanding fine-grained perception grounded in context or task, notably:

• Open-set video-based FER, where they enable robust discrimination of both known and unseen emotion classes (Liu et al., 2024).
• Multimodal LLM-driven tutoring systems, inducing empathetic behavior via lightweight, interpretable, and training-free pipelines (Feng et al., 10 Mar 2026).
• Visual question answering and spatial reasoning tasks, where attention focusing via prompt-aware adapters reduces LLM cognitive load and boosts accuracy on perception-intensive benchmarks (Zhang et al., 2024).

Limitations include:

• For visual prompt injection methods (e.g., patch-based in HESP), localization accuracy is bottlenecked by CAM thresholds and fixed mask reuse.
• Prompt-aware adaptation does not resolve semantic ambiguity in prompt interpretation and relies on pre-existing context cues.
• AU-based pipelines inherit limits from AU detector accuracy and may not generalize to non-facial expressions or multi-person interactions.

7. Comparative Summary Table

Module	Expression Cue Extraction	Visual Prompt Mechanism	Downstream Impact
HESP (Liu et al., 2024)	CAM mask on CLIP vision encoder	Patch-level learnable prompt ($\phi_V$3), mask injection	+17.9% AUROC, +106% OSCR; robust open-set FER
Prompt-Aware Adapter (Zhang et al., 2024)	CLIP text encoder, global+local prompt	Global scene + local token attention weighting	+5.5 pts VQA; +75.2 MME; localized reasoning
LLM Empath Tutor (Feng et al., 10 Mar 2026)	AU regression (IR-50)	AU→Text or saliency-guided frame selection	$\phi_V$4 (Q2 empathy); backbone-agnostic

These approaches demonstrate how injection, modulation, and integration of expression- or prompt-aware visual features yield substantial benefits in communication, recognition, and generative multimodal systems.