InsightVQA: High-Dimensional Emotion-Cognitive Visual Question Answering Benchmark

Abstract: Visual emotion understanding requires models not only to recognize emotional states, but also to why they arise and perform higher-level cognitive reasoning. However, existing benchmarks mainly focus on emotion recognition, offering limited support for grounded understanding and response-oriented analysis. To address this gap, we introduce \textbf{InsightVQA}, a large-scale dataset for hierarchical visual question answering on emotion understanding and cognitive reasoning. Building from 351K images collected from six public sources, we apply a rigorous multi-stage filtering pipeline to curate 138K high-confidence images. Each image is annotated at three hierarchical levels: perception QA for emotion and valence recognition, grounded understanding QA constructed from visual trigger extraction through constraint-guided generation, and cognition QA centered on response intent prediction and sequential insight reasoning. In total, InsightVQA contains 725K QA pairs. We further present \textbf{InsightVQA-Bench}, a high-quality evaluation benchmark comprising 30K samples for fine-grained evaluation. To support evaluation, we introduce \textbf{InsightNet}, an emotion-tuned baseline for MLLMs. Results demonstrate that InsightVQA poses significant challenges for grounded emotion understanding and reasoning.

• The paper introduces a hierarchical benchmark with 138,008 balanced images and 725,000 QA pairs spanning emotion perception, evidence-grounded understanding, and cognitive response reasoning.
• The paper uses clustering, visual emotion experts, grounded trigger extraction, language-model verification, and human review to achieve strong annotation reliability, including Fleiss’ Kappa of 0.91.
• The paper’s InsightNet baseline reaches 76.25% perception accuracy and 69.18% cognition accuracy, showing that domain-specific hierarchical tuning substantially outperforms larger general-purpose models.

Motivation and problem formulation

InsightVQA (2606.02171) addresses a specific gap in affective multimodal research: existing visual emotion benchmarks concentrate on recognition—predicting discrete categories or dimensional scores—while providing little supervision for explaining why an emotion arises or for reasoning about appropriate responses. The authors ground their design in Theory of Mind literature, arguing that human-centered visual understanding requires inferring latent mental states (beliefs, intentions, emotions) rather than performing low-dimensional label prediction. They operationalize this as a three-stage hierarchical VQA task: Perception (emotion category and valence recognition), Understanding (grounding emotional triggers in verifiable visual evidence), and Cognition (response intent prediction and sequential insight reasoning). This decomposition is the paper's central claim: that emotion understanding should be evaluated as structured, multi-level reasoning rather than classification.

Dataset construction

The dataset is built from 351,165 images crawled from six public sources using a search matrix spanning five macro-dimensions (Action, Facial, Style, Entity, Concept) with over 1,500 emotion keywords. A notable empirical finding emerges during annotation: general-purpose VLMs exhibit severe zero-shot affective bias. QwenVL and InternVL assigned over 50% of the pool to "contentment," which led the authors to exclude these models entirely from perception annotation—a result worth noting for anyone relying on off-the-shelf VLMs for affective labeling.

Instead, the pipeline uses a tri-verification scheme: K-Means clustering ($K{=}9$) over CLIP features fine-tuned on EmoSet produces pseudo-labels, cross-validated against two dedicated Visual Emotion Analysis experts (SMKD-VEA and EmotionCLIP). An image is retained only if at least one VEA model agrees with the clustering prediction; neutral clusters and edge samples are discarded, reducing the pool to 214,015 images. Confidence-aware class balancing then caps each of eight emotion categories at 20,000 samples, prioritizing three-way consensus and high-confidence two-way agreement. After human review purges rejected samples, the final perception foundation comprises 138,008 class-balanced images.

The higher annotation layers are LLM-generated but heavily constrained:

• Understanding: Qwen3-VL-32B-Instruct extracts 6–15 candidate trigger phrases per image across low-level appearance, mid-level scene, and high-level agent cues, with prompts explicitly banning emotion words to prevent label leakage. Triggers are verified via spaCy filtering and Grounding DINO spatial grounding (with exemptions for non-localizable evidence such as gaze or lighting). DeepSeek-V3 generates three QA types per image—visual attribution, contextual synthesis, counterfactual reasoning—with strict trigger-allocation constraints (at least four distinct triggers when available; no visual details beyond cited triggers).
• Cognition: conditioned on verified understanding outputs and few-shot exemplars retrieved from a 24-item expert-written bank, Gemini-3.1-flash-lite-preview generates one of eight closed-set response intents, a rationale, a one-to-three-step insight sequence (semantic/somatic/regulatory), behavioral constraints, and a safety note.

Quality control is cascaded and iterative: rule-based validation, NLI consistency checking with DeBERTa-v3-large-mnli (rejecting contradictions at $p > 0.9$), and VLM re-examination, with up to three repair iterations per sample.

Human verification yields strong reliability figures: on 6,400 stratified source images (each reviewed by 3 of 8 annotators), the pass rate is 92.52% with Fleiss' Kappa of 0.91. On 7,200 generated annotations, pass rates are 92.64% (answer correctness), 95.00% (visual grounding), and 94.58% (reasoning rationality). These numbers support the claim that LLM-generated annotations grounded in verified triggers can reach near-expert quality, though roughly 5–7% of generated content still fails verification even after iterative repair.

The full dataset contains 725K QA pairs (276K perception, 330K understanding, 119K cognition). InsightVQA-Bench holds out 10% of data as a 30K-sample evaluation set, converting generative QA into discriminative formats: SCQs with LLM-generated distractors for perception and understanding, plus a Situational Judgment Test (SJT) in which models must rank candidate insight sequences from best to worst. Compared against prior affective VQA resources (EmoVIT, EEmoDB, EIBench, FaceBench, UNIFER, among others), InsightVQA is the only benchmark combining scale, grounded evidence, and cognitive-reasoning supervision.

InsightNet baseline and results

InsightNet applies LoRA fine-tuning ($r=32$, $\alpha=32$, dropout 0.05) to Qwen2.5-VL-7B on the InsightVQA training split, with LoRA chosen explicitly as implicit regularization against overfitting on emotionally biased data. The results are stark:

Model	Perception ACC	Understanding F1	Cognition Rank	Cognition Top-1	Cognition ACC
Qwen2.5-VL-7B	57.95	88.24	34.95	51.27	30.92
GPT-4o	50.63	88.34	77.51	61.06	50.34
Claude-3.7-sonnet	56.37	88.72	81.77	63.49	61.87
Gemini-2.5-flash	56.83	89.05	80.56	63.51	56.08
EmoViT	53.69	84.50	–	12.53	42.86
InsightNet	76.25	90.56	82.79	71.21	69.18

Two observations stand out. First, all evaluated MLLMs—including frontier closed-source models—cluster tightly around 87–89 F1 on understanding while scoring only 50–58% accuracy on eight-way perception, indicating that open-ended understanding metrics may be saturating or insufficiently discriminative relative to the difficulty of categorical perception. Second, InsightNet's gains over its base model (+18 points perception accuracy, +38 points cognition accuracy) demonstrate that domain-specific instruction tuning on hierarchically structured data yields substantially larger improvements than model scale alone: Qwen2.5-VL-72B reaches only 57.24 cognition accuracy versus InsightNet's 69.18. The authors interpret the cross-task pattern as evidence of hierarchy synergy—perception grounding enables understanding, which enables cognitive planning—though this is an observational claim, not an ablated causal result.

Limitations and open questions

Several caveats bear directly on the reported results. The understanding and cognition layers are LLM-generated (DeepSeek-V3, Gemini-3.1-flash-lite-preview) with automated verification; despite human spot-checks on 7,200 samples, systematic biases inherited from the generator models cannot be ruled out, and the "grounded" nature of triggers depends on Grounding DINO thresholds and deliberate exemptions for non-localizable cues. The cognition layer's closed set of eight response intents and its SJT ranking format impose a specific structure on "cognitive reasoning" whose ecological validity is asserted rather than independently validated. Class balancing caps categories at 20,000 images, meaning rare emotions like disgust rely predominantly on single-model verification rather than three-way consensus. Finally, the evaluation uses BERTScore-based F1 for understanding, which the near-uniform baseline scores suggest may not discriminate well among strong models; whether the benchmark separates models more finely under stricter human-judged evaluation remains an open question, as does whether InsightNet's gains generalize beyond in-distribution InsightVQA-Bench data.

Conclusion

InsightVQA contributes a large-scale, hierarchically annotated resource (138K images, 725K QA pairs) and a 30K-sample benchmark that extends visual emotion evaluation from recognition toward grounded explanation and response-oriented cognition. Its tri-verification pipeline and documented VLM affective bias offer practical lessons for dataset construction, and the InsightNet results quantify how far current MLLMs—from 7B open-source to frontier closed-source—fall short on hierarchical affective reasoning when trained without such data. The principal unresolved issues concern the independence of LLM-generated supervision and the generalization of benchmark performance outside the curated distribution.