GridVQA-X: Multimodal Explainability Benchmark

• GridVQA-X is a diagnostic framework for evaluating cross-modal explainability by distinguishing robust spatial-relational reasoning from shortcut-driven behavior.
• It leverages synthetic VQA datasets and paired models—one relying on true spatial cues and the other on Bag-of-Words shortcuts—to generate mathematically guaranteed ground-truth explanations.
• The framework assesses local and global explanation methods using metrics like IoU and Relevance Mass Accuracy to quantify attribution faithfulness.

GridVQA-X is a diagnostic framework for evaluating multimodal explainability methods in vision–language settings, introduced to determine whether an explainer can distinguish genuine cross-modal spatial-relational reasoning from shortcut-driven behavior in multimodal models (Belsare et al., 2 Jun 2026). It combines a synthetic VQA dataset family, a closed-world synthesis logic with mathematically guaranteed explanations, paired models trained on identical architectures but divergent data regimes, and evaluation protocols for both local attributions and global interaction measures. Within this framework, the core contrast is between a model that learns robust spatial-relational reasoning and a model that is structurally induced to rely on Bag-of-Words cross-modal shortcuts, so that faithful explainability methods should expose different reasoning pathways for the two.

1. Conceptual scope and motivation

GridVQA-X is defined as “the first diagnostic framework specifically designed to evaluate cross-modal explainability” (Belsare et al., 2 Jun 2026). Its motivating problem is that modern vision–LLMs and large vision–LLMs achieve strong task performance through tightly fused visual and textual representations, while existing Multimodal Explainable AI methods lack causal ground truth for assessing whether their explanations reflect actual model reasoning. In real multimodal datasets, there is no exact causal account of which pixels and tokens drive a prediction, and models often exploit shortcuts such as answer priors, Bag-of-Words attribute matching, visual majority heuristics, or partial logical decomposition. As a result, an explainer may either faithfully expose a shortcut or hallucinate a plausible spatial-relational rationale, and standard evaluation protocols cannot reliably tell the difference (Belsare et al., 2 Jun 2026).

The framework addresses this problem by constructing a controlled synthetic environment in which the reasoning structure of each VQA instance is known exactly. A visual scene is represented as a set of objects

$$\mathcal{V} = \{o_1, \dots, o_N\}, \quad o_i = (c_i, s_i, p_i),$$

where $c_i$ is color, $s_i$ is shape, and $p_i$ is position in a discrete grid. A question $\mathcal{Q}$ specifies target objects $\mathcal{T}$, anchors $\mathcal{A}$, and spatial relations encoded as constraints on positions. This closed-world synthesis logic allows the generator to produce unique, mathematically defined ground-truth explanations rather than proxy annotations or human rationales (Belsare et al., 2 Jun 2026).

A source of terminological ambiguity is that earlier grid-based VQA work used “GridVQA-X” only as a hypothetical name for grid-centric video QA systems or benchmarks, especially in discussions of image-grid prompting and transcript-based VideoQA (Kim et al., 2024, Chowdhury et al., 30 May 2025). In the 2026 usage, however, GridVQA-X denotes a multimodal explainability benchmark rather than a video question answering architecture.

2. Synthetic data generation and task structure

GridVQA-X is built around a synthetic VQA dataset family called GridVQA, with two parallel datasets: $\mathcal{D}_{\text{pure}}$, which forces true spatial reasoning, and $\mathcal{D}_{\text{spur}}$, which makes a specific shortcut perfectly predictive (Belsare et al., 2 Jun 2026). Each sample consists of an $S \times S$ grid image populated with geometric shapes and a question asking either existence or count under varying degrees of spatial-relational complexity.

Each sample is parameterized by a 4-tuple

$c_i$0

where $c_i$1 is the depth, $c_i$2 is the question type, $c_i$3 is the form, and $c_i$4 is the density (Belsare et al., 2 Jun 2026). Depth counts the number of anchors or relational hops. The question-type taxonomy separates non-spatial attribute-only questions from spatial questions over shapes, colors, or mixed shape–color compositions, and also includes comparison questions over counts. Form distinguishes counting from existence, while density controls scene sparsity versus distractor-rich scenes.

This design explicitly probes spatial-relational reasoning, attribute binding, and compositional complexity. A depth-1 query may require a single directional constraint such as a target object left of an anchor; depth-2 and depth-3 queries require intersections of multiple valid regions. This suggests that the benchmark’s primary difficulty is not low-level perception but the correct use of conjunctions of relational constraints under distractor pressure (Belsare et al., 2 Jun 2026).

The generator also defines distractors as

$c_i$5

and, using do-calculus, asserts that for interventions on distractors,

$c_i$6

Under this construction, only anchors and targets have nonzero causal effect, so $c_i$7 is the unique ground-truth causal explanation (Belsare et al., 2 Jun 2026). This is the formal basis for evaluating explanation faithfulness in the benchmark.

3. Paired models, shortcut induction, and causal contrast

A central feature of GridVQA-X is the use of paired models with identical architecture but different training environments. Both models use MDETR, a transformer-based, end-to-end multimodal detection and QA model, but one is trained on $c_i$8 and the other on $c_i$9 (Belsare et al., 2 Jun 2026). The first, $s_i$0, must learn the full spatial-relational cause in order to attain high accuracy. The second, $s_i$1, can attain perfect performance on its own data by exploiting a deliberately embedded Bag-of-Words shortcut.

The framework formalizes a shortcut heuristic as a function $s_i$2 that uses only a proper subset $s_i$3 of the true causal features. A spurious correlation exists if

$s_i$4

Four shortcut families are enumerated: answer priors, Bag-of-Words spatial shortcut, visual feature dominance or majority, and logical decomposition through dropped anchors (Belsare et al., 2 Jun 2026).

The most important injected shortcut is Case-1, the Bag-of-Words spatial shortcut. For a relational query $s_i$5, the shortcut predicts the answer from the number of objects matching the target attributes, ignoring the relation $s_i$6. In $s_i$7, the generator ensures that no target-attribute objects occur outside the valid relational region, so

$s_i$8

In $s_i$9, by contrast, the generator enforces Spatial Independence: for every relational query, it inserts adversarial distractors that match the target attributes but violate the spatial relation, making

$p_i$0

This destroys the shortcut and forces use of the full relation (Belsare et al., 2 Jun 2026).

For multi-anchor queries, the framework also prevents partial-logic heuristics. If $p_i$1 is the valid region for anchor $p_i$2, and $p_i$3, then the confuser region for dropping anchor $p_i$4 is

$p_i$5

The stated “Generalized Robust Intersection Guarantee” shows that if $p_i$6 is nonempty and contains at least one adversarial target-attribute object, then any model ignoring anchor $p_i$7 will always over-count and incur training loss (Belsare et al., 2 Jun 2026). This makes the paired models a controlled behavioral contrast: a faithful explainer should reveal distinct mechanisms rather than merely distinct predictions.

4. Ground-truth explanations and evaluation protocol

Because the generative process identifies anchors, targets, and distractors exactly, each sample carries pixel or object masks for $p_i$8, $p_i$9, and noncausal objects (Belsare et al., 2 Jun 2026). The true explanation is the union of anchor and target masks. Any attribution mass assigned to distractors is, by construction, unfaithful for $\mathcal{Q}$0. For $\mathcal{Q}$1, however, distractor-sensitive or shortcut-sensitive explanations can be informative if they expose the model’s actual heuristic.

GridVQA-X evaluates both local and global explanation methods. Local explanations include saliency maps over image regions and text tokens and cross-modal attribution matrices. Global explanations include scalar synergy scores that estimate how much predictive power arises from interaction between modalities rather than additive unimodal contributions (Belsare et al., 2 Jun 2026).

Three principal metrics organize the evaluation. The first is Intersection-over-Union, used after thresholding an explanatory heatmap against the ground-truth mask. The second is Relevance Mass Accuracy, defined as

$\mathcal{Q}$2

which measures how much attribution mass lies inside a specified mask (Belsare et al., 2 Jun 2026). The third is the Additive Fallacy Check for global methods. If $\mathcal{Q}$3 is a global synergy score at depth $\mathcal{Q}$4, then a faithful estimator should increase with depth for $\mathcal{Q}$5, because more anchors require more cross-modal composition, and remain roughly flat for $\mathcal{Q}$6, because its shortcut does not become more compositional as depth increases (Belsare et al., 2 Jun 2026).

The protocol comprises three scenarios. In pure evaluation, the explainer is applied to $\mathcal{Q}$7 on $\mathcal{Q}$8, where attributions should focus on anchors and targets and synergy should increase with depth. In spurious evaluation, the explainer is applied to $\mathcal{Q}$9 on $\mathcal{T}$0, where it should reveal decreased reliance on directional cues and increased reliance on shortcut objects. In cross-evaluation, the explainer is applied to $\mathcal{T}$1 on $\mathcal{T}$2, where the model fails on spatial queries and a faithful explanation should expose misaligned or shortcut-focused attributions (Belsare et al., 2 Jun 2026).

5. Experimental findings on local and global explainers

GridVQA-X evaluates local explainers DIME, MultiSHAP, and MultiViz-gradient, and global methods including Emap and InterSHAP (Belsare et al., 2 Jun 2026). The principal empirical finding is that none of the tested methods reliably distinguishes the shortcut learner $\mathcal{T}$3 from the genuine spatial reasoner $\mathcal{T}$4.

On the question of diagnosing shortcut learning, MultiViz shows nearly identical Relevance Mass Accuracy on ground-truth masks for both models, around $\mathcal{T}$5, and is characterized as effectively model-blind (Belsare et al., 2 Jun 2026). DIME produces diffuse heatmaps that spread over large portions of the grid, so that attribution overlaps the correct targets even for the shortcut model and therefore creates the appearance of faithfulness. MultiSHAP yields higher RMA for $\mathcal{T}$6, around $\mathcal{T}$7, than for $\mathcal{T}$8, around $\mathcal{T}$9, which the authors interpret as a preference for independent, localized detectors over non-linear spatial intersections (Belsare et al., 2 Jun 2026). This suggests that Shapley-style interaction scores can systematically favor shortcut structure when the shortcut is sparse and locally detectable.

On $\mathcal{A}$0 with $\mathcal{A}$1, the local methods exhibit distinct failure modes. DIME’s IoU remains low, around $\mathcal{A}$2, even when RMA increases in dense scenes, indicating a volume effect rather than genuine relational localization (Belsare et al., 2 Jun 2026). MultiSHAP achieves IoU above $\mathcal{A}$3, but its heatmaps highlight all objects sharing target color or shape, including adversarial distractors, which means it is functioning as an attribute detector rather than a relational explainer. MultiViz has extremely low IoU, below $\mathcal{A}$4, despite moderate RMA, reflecting poor spatial localization (Belsare et al., 2 Jun 2026).

The global methods fail the Additive Fallacy Check. For Mixed queries of Form 0 on $\mathcal{A}$5, Emap’s interaction fraction decreases from $\mathcal{A}$6 at depth 1 to $\mathcal{A}$7 at depth 3, and InterSHAP drops from $\mathcal{A}$8 to $\mathcal{A}$9 over the same depth range (Belsare et al., 2 Jun 2026). These trends invert the expected monotonicity for a true compositional reasoner and therefore misrepresent deeper relational queries as less synergistic. The broader conclusion is that existing methods often confound additive dependence on both modalities with genuine cross-modal composition.

6. Technical implementation, related grid-based research, and implications

The paired models in GridVQA-X are trained in two phases with MDETR. The first phase is phrase grounding or visual grounding, using

$\mathcal{D}_{\text{pure}}$0

where $\mathcal{D}_{\text{pure}}$1 is L1 box regression loss and $\mathcal{D}_{\text{pure}}$2 is generalized IoU loss. The second phase predicts the answer using class-weighted cross-entropy,

$\mathcal{D}_{\text{pure}}$3

with $\mathcal{D}_{\text{pure}}$4 inversely proportional to class frequency in the batch (Belsare et al., 2 Jun 2026). The paper characterizes these stages as explanation-guided training because grounding supervision aligns internal cross-attentions with causal objects.

The empirical behavioral divergence between the paired models is strong. On its own dataset, $\mathcal{D}_{\text{pure}}$5 achieves approximately $\mathcal{D}_{\text{pure}}$6 accuracy on $\mathcal{D}_{\text{pure}}$7. Under cross-evaluation on $\mathcal{D}_{\text{pure}}$8, its overall accuracy drops to about $\mathcal{D}_{\text{pure}}$9, with depth-2 Mixed queries at $\mathcal{D}_{\text{spur}}$0 and depth-3 Mixed queries at $\mathcal{D}_{\text{spur}}$1, while Attribute-Only questions remain at $\mathcal{D}_{\text{spur}}$2 (Belsare et al., 2 Jun 2026). This validates the claim that the spurious model is not performing spatial composition.

GridVQA-X belongs to a broader line of grid-based research, but its function differs sharply from prior work. Earlier studies used grids as an input representation for video question answering or as a substitute for region features in VQA. IG-VLM converted videos into temporally ordered image grids and used a single off-the-shelf vision–LLM in zero-shot VideoQA, showing that image grids can encode temporal information without video-data training (Kim et al., 2024). Grid-LoGAT used grid-based local and global area transcription to produce structured text transcripts from frames for privacy-preserving VideoQA, with a 2×3 grid as the default layout (Chowdhury et al., 30 May 2025). “In Defense of Grid Features for Visual Question Answering” argued that grid features can match region-based VQA performance while running more than an order of magnitude faster, reporting $\mathcal{D}_{\text{spur}}$3 on VQA 2.0 test-std (Jiang et al., 2020). These works treat grids primarily as representational or efficiency devices, whereas GridVQA-X uses a grid world to obtain causal identifiability for explanation evaluation.

A related but distinct line is high-fidelity spatial VQA generation from 2D geometry. GRAID showed that qualitative spatial relationships can be reliably derived from 2D bounding boxes and deterministic rules, generating more than $\mathcal{D}_{\text{spur}}$4 million VQA pairs and reporting $\mathcal{D}_{\text{spur}}$5 human-validated accuracy against a $\mathcal{D}_{\text{spur}}$6 baseline from a recent data-generation pipeline (Elmaaroufi et al., 25 Oct 2025). This suggests a methodological affinity with GridVQA-X: both depend on controlled geometry and explicit constraints to separate true spatial reasoning from confounded or hallucinated proxies.

The principal implication of GridVQA-X is that current multimodal explainability methods can produce plausible yet unfaithful accounts of model behavior, especially when cross-modal shortcuts are available (Belsare et al., 2 Jun 2026). Its main limitation is the synthetic domain: scenes are discrete grids of geometric objects, and the framework is restricted to spatial-relational reasoning and attribute binding rather than temporal reasoning, commonsense, or open-ended natural-language rationales. A plausible implication is that the framework is best understood not as a substitute for real-world evaluation but as a necessary diagnostic layer for determining whether an explanation method can pass a causally identifiable test before being trusted on natural multimodal data.