Visual Question Answering with Grounding (VQA-G)

• Visual Question Answering with Grounding (VQA-G) is a vision-language approach that not only predicts answers but also identifies the precise visual evidence required for correct inference.
• It employs techniques like attention supervision, capsule-based transformers, and explicit mask prediction to align responses with relevant image or video regions.
• Evaluation metrics such as IoU, mAP, and FPVG assess both answer accuracy and the fidelity of grounding to mitigate shortcut learning and enhance model robustness.

Visual Question Answering with Grounding (VQA-G) denotes a class of vision-language systems that, given an image (or video) and a natural-language question, must not only produce an answer but also explicitly identify (i.e., ground) the visual evidence supporting that answer. Grounding in VQA-G is operationalized as selecting regions, bounding boxes, polygons, temporal segments, or visual entities within the input modality that are causally or semantically required for correct inference. The integration of grounding supervision and evaluation in VQA frameworks is motivated by the need for interpretability, robustness, user trust, and mitigation of shortcut learning.

1. Task Definition and Formalization

VQA-G extends traditional Visual Question Answering, which involves predicting an answer $A$ for a given image $I$ and question $Q$, by additionally requiring the system to output a visual grounding $R$ or set of regions/evidence corresponding to $Q$. The output is thus:

$\text{VQA-G:} \quad (I, Q) \mapsto (A, R)$

where $R$ could be a bounding box, mask, polygon, object indices, temporal interval (for video), or heatmap corresponding to the visual evidence necessary for $A$.

Several works set out formal logical criteria for grounding. (Reich et al., 2024) defines Visual Grounding (VG) formally: the model's inference for $A$ must rely on question-relevant image regions. It further specifies axiomatic requirements: correctness implies grounding ($A \rightarrow VG$) and lack of grounding implies incorrectness ($I$0). The Visually-Grounded Reasoning (VGR) framework codifies this as:

$I$1

with $I$2 as shortcut-free reasoning. For video, the analogous output includes a temporal interval $I$3 alongside the answer (Xiao et al., 2023).

2. Datasets and Evaluation Protocols

Several datasets provide grounding annotations to enable the development and benchmarking of VQA-G systems:

• VizWiz-VQA-Grounding (Chen et al., 2022): 9,998 real (I, Q, A, R) tuples (images, questions posed by visually impaired users, crowd-validated single answer, polygonal groundings). The dataset targets authentic, unconstrained photos and questions, with region annotations covering a broad range of visual phenomena (object, color, text).
• Visual7W (Hu et al., 19 Apr 2026, Fukui et al., 2016): Free-form QA with bounding-box referent annotations.
• VQA-HAT, VQA-X, GQA, CLEVR-Answers: Provide attention overlays, polygons, or object indices primarily on images; automatically generated for synthetic or compositional domains (Chen et al., 2022).
• NExT-GQA (Xiao et al., 2023): Extends VideoQA with segment-level (start, end) groundings for 8,900+ temporal questions.
• RefCOCOg [(Chen et al., 30 Sep 2025)]: Used for supervised training of object-level groundings.

Evaluation metrics include:

• Intersection-over-Union (IoU):

$I$4

• Pointing Game (PG): Accuracy of maximum attention landing within ground-truth region.
• mAP@IoU: Mean average precision at several IoU thresholds.
• FPVG (Reich et al., 2023): Combines faithfulness (did model attend to relevant region) and plausibility (does answer change if that region is masked):

$I$5

Faithful and plausible grounding are further partitioned in (Reich et al., 2024) as GGC, GGW, BGC, BGW.

3. Grounding Methodologies

VQA-G approaches are characterized by explicit architectural or loss-based mechanisms for grounding.

• Attention Supervision Mining: Automatically derive pseudo-ground-truth attention maps from region descriptions (e.g., Visual Genome) and inject them into attention modules via KL losses. Example: Attn-MFB, Attn-MFH (Zhang et al., 2018).
• Multimodal Fusion with Attention: Bilinear pooling (e.g., Multimodal Compact Bilinear Pooling, MCB (Fukui et al., 2016)) fuses spatial image features and question encodings, predicting soft attention maps for spatial grounding.
• Capsule-Based Transformers: Replace grid image features with text-guided capsule encodings, masking capsules by semantic relevance (Khan et al., 2022); enables object-like grounding without reliance on external detectors.
• Explicit Mask Prediction: Predict soft masks (heatmaps, bounding boxes) conditioned on the question, trained by regressing to ground-truth polygons, boxes, or segmentation maps (GDINO, (Chen et al., 30 Sep 2025)).
• Compositional Lattice Retrieval: VQA-Lattice-based Retrieval (VLR (Reich et al., 2022)) constructs a scene graph, parses the question to a sequence of operations, and aligns the answering path through the evidence graph to produce explicit grounding.
• Contrastive and Auxiliary Losses: Encourage interpretability and prevent shortcut reasoning by enforcing that answer prediction depends on grounded visual evidence rather than spurious correlations (Le et al., 2022, Reich et al., 2024).
• Causal and Self-Interpretable Training: For video, Equivariant & Invariant Grounding (EIGV (Li et al., 2022)) separates causal (answer-critical) from environment (background) clips via Gumbel-Softmax masks and dedicated equivariant/invariant losses.

Recent methods integrate automatic data generation, verification, and prompt refinement for dataset construction: AutoVQA-G employs a generate–evaluate–refine loop with chain-of-thought consistency judgments to ensure high-fidelity (I, Q, A, R) tuples (Hu et al., 19 Apr 2026).

4. Shortcut Learning, Out-of-Distribution Robustness, and Faithfulness

Empirical analysis reveals that standard VQA models often exploit dataset biases or language shortcuts, achieving high accuracy with poor visual grounding (Reich et al., 2024, Reich et al., 2023). In such settings, models answer correctly without utilizing question-relevant regions (high BGC rate in FPVG categorization). This is exacerbated in out-of-distribution (OOD) splits (e.g., GQA-CP, VQA-CPv2), where language–answer priors are suppressed, exposing a lack of genuine grounding.

Mitigation strategies include:

• Information Infusion (INF): Replace or correct visual features to ensure all question-relevant objects are represented consistently during training (Reich et al., 2024, Reich et al., 2024).
• OOD splits enforcing grounding: The GQA-AUG OOD protocol modifies images to ensure that the correct answer is possible only if the model attends to the question-relevant region (Reich et al., 2024).
• Evaluations restricted to TVG (True Visual Grounding) subsets, filtering to samples where all required objects are indeed detectable (Reich et al., 2024).

Faithful grounding, as opposed to merely plausible overlapping with human attention, is crucial for debiasing and interpretability. The FPVG metric explicitly operationalizes this requirement (Reich et al., 2023).

5. Practical Architectures and Training Protocols

A wide range of architectures are adapted for VQA-G:

Model/Approach	Grounding Mechanism	Strengths/Weaknesses
Attention-mined bilinear pooling	MCB/MFB/MFH + pseudo-attention supervision (Zhang et al., 2018)	Improved grounding, scalable supervision via mining
Capsule-based Transformers	Text-guided capsules, detector-free (Khan et al., 2022)	Enhanced object-level grounding, SOTA on GQA, VQA-HAT
Lattice-based Retrieval (VLR)	Lattice over scene-graph; IR-style inference (Reich et al., 2022)	Maximal grounding, robust OOD, symbolic, lower ID accuracy
Retrieval-Augmented Generation + Grounding Head	Text-anchored box prediction, targeted retrieval (Chen et al., 30 Sep 2025)	Improved truthfulness, reduced hallucination
EIGV (video)	Causal/environmental split with equiv/inv objectives (Li et al., 2022)	Intrinsic interpretability, improved accuracy
AutoVQA-G	Self-improving annotation loop, CoT verification (Hu et al., 19 Apr 2026)	Produces higher-fidelity VQA-G data than GPT-4o+tools

Training objectives combine cross-entropy for classification with additional grounding losses: KL-divergence for attention, regression for boxes/polygons, cosine/ranking losses for alignment, and auxiliary contrastive or causal losses as appropriate.

6. Empirical Observations, Open Challenges, and Future Directions

Benchmarking shows that current VQA/G models often exhibit moderate to low IoU scores on authentic, diverse settings (VizWiz: best models ~27–33% IoU on all, worse on small regions) (Chen et al., 2022). Pretraining on in-domain data is crucial for transfer; models trained on synthetic or unrelated datasets underperform. Error analysis reveals issues with small or text-based groundings, noisy detection, and language bias.

Future work and open directions include:

• End-to-end integration of semantic validation: Ensuring that detected features correspond to ground-truth objects required for grounding (Reich et al., 2024).
• Scalable annotation and dataset construction: Automatic refinement and CoT-based verification to reduce reliance on expensive human labeling (Hu et al., 19 Apr 2026).
• Robustness to visual quality variation and OOD data: Mitigating reliance on language priors and improving compositional generalization (Reich et al., 2022, Reich et al., 2024).
• Extending to dense spatial substructures: Beyond boxes and segments—dense rationale annotations, open-vocabulary, and relational grounding.
• Transparent evaluation and calibration: Reliable faithfulness/plausibility metrics, model abstention for uncertain queries, and hallucination management (Chen et al., 30 Sep 2025).

7. Significance and Impact

VQA-G advances the field by enforcing interpretability, reducing spurious correlations, and paving the way toward trustworthy multimodal reasoning systems. Explicit grounding aligns system behavior with human expectations, enables user-centric applications (e.g., assistive tech for visually impaired), and offers new axes for evaluating model robustness and generalization. The development of datasets, benchmarks, and metrics tailored to grounding is catalyzing research toward models that "answer for the right reasons" rather than exploiting shortcuts or dataset bias. As research progresses, integration of VQA-G principles in large-scale multimodal LLMs and real-world applications remains a central objective for the community (Chen et al., 2022, Reich et al., 2024, Hu et al., 19 Apr 2026).