ProtoVQA: Explainable Fine-Grained VQA

• The paper introduces ProtoVQA, a prototype-based framework for explainable fine-grained VQA that uses question-aware prototypes and spatially constrained matching to generate human-verifiable explanations.
• It leverages a shared multimodal space by encoding images with DeiT and questions with DeBERTa, forming prototypes via the formula: P = Reshape(F(E_q[:m × k])) ∈ ℝ^(m × k × D), with cosine similarity computed between image patches and prototype sub-patches.
• Performance results on Visual7W show an accuracy of 70.23% and improvements in explanation faithfulness (VLAS@1 increased by +66.4% over Bi-CMA), highlighting its practical benefits in safety-critical applications.

to organize experimental outcomes. Need ensure table cells brief.

Need not mention "I". Need neutral encyclopedic tone. Need no references to how wrote it. No prefaces. Need markdown only. Need technical depth. No meta commentary. Need likely use formulas from data. We can include some. Need careful to use exact formulas from data maybe corrected from typos? Should maintain factual fidelity. There are some malformed equations in data for (Diao et al., 20 Sep 2025). Need perhaps use only the formulas that are clear and appear. Could use those with notation. But note some notation is malformed because brackets. We can present what data contains. Need not invent corrected formulas maybe okay if slight formatting needed. But "Every concrete claim... must appear verbatim in data block below." We can repeat formulas as given. Yet the formulas have typos. Need not overdo. Could include only those fully clear:

• $$P = \mathrm{Reshape}(\mathcal{F}(E_q[:m \times k])) \in \mathbb{R}^{m \times k \times D}$$
• similarity equation
• argmax with masks
• prototype score equation
• VLAS equation maybe. Need not fix denominator if malformed? It says $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$ missing brace. We can present in prose rather than formula: VLAS counts how many examples achieve sufficient overlap, with $\theta=0.5$. That stays factual. Might avoid malformed formulas.

Need arxiv search tool. Let's do search for ProtoVQA and (Bara et al., 2022) maybe. ProtoVQA most directly denotes the prototype-based visual question answering framework introduced in “ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering” (Diao et al., 20 Sep 2025). It is a unified prototypical framework for explainable fine-grained VQA in which question-aware prototypes serve as reasoning anchors, spatially constrained matching enforces coherent evidence selection, and a shared prototype backbone supports both answering and grounding. A recurrent source of confusion is nomenclature: “ProtoVQA” is not the official name of the earlier privacy-preserving edge/cloud VQA system introduced in “Privacy Preserving Visual Question Answering” (Bara et al., 2022), although that system is sometimes treated as a related prototype-style or private VQA pipeline. In current usage, the formal title ProtoVQA refers to the 2025 prototype-based explainable framework.

1. Problem setting and terminological scope

ProtoVQA is situated in the branch of VQA concerned with fine-grained, human-verifiable explanations rather than answer prediction alone. The motivating applications named for the framework include medical imaging, autonomous systems, and other safety-critical domains, where a model must provide not only accurate answers but also explanations that humans can easily understand and verify (Diao et al., 20 Sep 2025). The framework therefore targets a stricter notion of interpretability than generic saliency visualization.

The method is motivated by three limitations attributed to prior approaches. First, prototype-based interpretability methods had mostly been developed for purely visual tasks and did not explicitly bridge the visual–language semantic gap required by VQA. Second, fixed prototypes were described as ill-suited to the dynamic, question-dependent nature of VQA, in which relevant evidence changes with the query and often depends on geometric variation and part-level cues. Third, many attention-based or post-hoc explanation methods were characterized as showing where the model looked without necessarily exposing an internal causal reasoning path, and as producing only coarse visual saliency rather than fine-grained grounding (Diao et al., 20 Sep 2025).

A separate but related line of work is the privacy-preserving hybrid VQA architecture of “Privacy Preserving Visual Question Answering” (Bara et al., 2022). That system addresses what leaves an edge device and how to prevent recovery of the original image from intermediate representations. It is therefore related by topic but distinct in objective: ProtoVQA is centered on explainable fine-grained grounding, whereas the privacy-preserving system is centered on non-differentiable symbolic transmission.

2. Core architecture

ProtoVQA consists of three principal components: visual feature extraction, text/question encoding in a shared projection space, and question-aware prototype learning (Diao et al., 20 Sep 2025). An image $I \in \mathbb{R}^{H \times W \times 3}$ is encoded by a pre-trained DeiT vision transformer, producing a CLS token and patch tokens. The patch features are then enhanced by subtracting the CLS token, which emphasizes local patch-level evidence relative to the global image context.

The question $Q = [q_1,\ldots,q_{l_q}]$ is encoded by DeBERTa to obtain token embeddings $E_q$, which are projected into the shared multimodal space $\mathbb{R}^D$ using a learnable projector $\mathcal{F}$. This shared space is used for image patches, question tokens, and answer representations. The shared embedding geometry is central to the framework because prototypes are learned as text-conditioned entities but must be matched against visual evidence.

Question-aware prototypes are formed by reshaping the first $m \times k$ projected question tokens:

$$P = \mathrm{Reshape}(\mathcal{F}(E_q[:m \times k])) \in \mathbb{R}^{m \times k \times D}.$$

This yields $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$0 prototypes, each containing $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$1 sub-patch prototypes. In the reported experiments, the framework uses $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$2 prototypes per class, $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$3 sub-patches per prototype, and spatial radius $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$4 (Diao et al., 20 Sep 2025). Each prototype is therefore a small structured set of subparts rather than a single vector, which allows the evidence search to adapt to the question.

3. Spatially constrained matching and shared reasoning backbone

The main interpretability mechanism in ProtoVQA is spatially constrained prototype matching. For each prototype $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$5, cosine similarity is computed between every image patch and every prototype sub-patch:

$\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$6

At each iteration, the model selects the best patch–subpatch pair under two masks:

$\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$7

where $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$8 is a binary availability mask that prevents reuse of selected patches and $\text{VLAS} = \frac{\sum_{i=1}^{N} \mathcal{I}(M_i \cap G_i > \theta)}{N_{QA},$9 is an adjacency mask that enforces spatial continuity (Diao et al., 20 Sep 2025). After a patch is selected, it is marked unavailable and the adjacency mask is updated so that only patches within radius $\theta=0.5$0 remain eligible.

The prototype score is then computed as a weighted sum of selected similarities,

$\theta=0.5$1

with learnable slot weights $\theta=0.5$2 (Diao et al., 20 Sep 2025). This converts prototype matching into a chain of spatially coherent selections rather than an unconstrained nearest-neighbor search over disconnected patches. The stated significance of this design is that it discourages scattered evidence and yields contiguous, semantically meaningful regions that are easier to inspect.

The same prototype-matching backbone is used for both grounding and answer prediction. For grounding tasks, coordinate inputs $\theta=0.5$3 are projected into the shared space through a dedicated projector. For multiple-choice textual answering, each answer is encoded by DeBERTa and projected through the same feature projector $\theta=0.5$4, with shared weights copied from the question branch and frozen (Diao et al., 20 Sep 2025). The matched patch features are concatenated with the processed answer features and passed to a classifier. The framework therefore does not separate explanation from reasoning into independent modules; the evidence selected by prototype matching directly supports the answer.

4. Evaluation protocol and quantitative results

ProtoVQA is evaluated on Visual7W, a grounded VQA benchmark containing 327,939 QA pairs, 47,300 COCO images, 4 answer options per question, 1.3M candidate answers total, and 561,459 object-level groundings over 36,579 categories (Diao et al., 20 Sep 2025). This benchmark is particularly suitable because it provides both answer supervision and grounding annotations.

The reported training setup uses NVIDIA A800 80GB hardware, 200 epochs, the Adam optimizer, learning rate $\theta=0.5$5, batch size 64, image resolution $\theta=0.5$6, and patch size $\theta=0.5$7 (Diao et al., 20 Sep 2025). The baseline set includes SUPER, QOI_Attention, SDF of VLT, STL, CFR, BriVL, CTI, and Bi-CMA.

On the Visual7W test set, ProtoVQA achieves competitive answer accuracy relative to strong Transformer baselines.

Method	Accuracy
ProtoVQA	70.23%
Bi-CMA (ViT-patch16)	70.53%
Bi-CMA (ViT-patch16 finetuned)	73.07%
SDF of VLT	65.93%

The framework’s principal quantitative claim concerns explanation faithfulness, measured by the Visual-Linguistic Alignment Score (VLAS). VLAS is defined over QA pairs using the union of matched patch boxes as the model-attended region $\theta=0.5$8, the ground-truth evidence region $\theta=0.5$9, an indicator function $I \in \mathbb{R}^{H \times W \times 3}$0, and threshold $I \in \mathbb{R}^{H \times W \times 3}$1; it measures how often the attended region exceeds the overlap threshold with ground-truth evidence (Diao et al., 20 Sep 2025).

Method	VLAS@1	VLAS@3
SDF of VLT	0.2013	0.0847
Bi-CMA	0.2466	0.1123
ProtoVQA	0.4103	0.2466

ProtoVQA is reported to improve over Bi-CMA by +66.4% relative improvement on VLAS@1 and +119.6% relative improvement on VLAS@3 (Diao et al., 20 Sep 2025). The numerical pattern indicates that the method is not only competitive in answer accuracy but substantially stronger in grounding-oriented explanation quality.

5. Interpretability profile, qualitative evidence, and limitations

The qualitative examples reported for ProtoVQA emphasize part-level and relation-sensitive evidence selection. Examples include “Which item can be used for communication?”, where the model highlights the telephone region; “Which is framing a white sideways boat?”, where selected patches align with the boat frame; “Which ear is the left ear of the right giraffe?”, which demonstrates part-level localization; and “Which flower tub, with red flowers in it, is beside a parking meter?”, which requires object identity together with spatial context (Diao et al., 20 Sep 2025). Additional examples in the appendix extend this pattern to human or animal anatomy, object identification, interaction-based queries, and spatial relation queries.

The framework’s explanation mechanism is explicitly intended to do more than indicate generic saliency. Its central claim is that question-aware prototypes act as semantic anchors and that spatially coherent matching makes the resulting evidence region more inspectable and more faithful to the answering process (Diao et al., 20 Sep 2025). This suggests a stricter notion of interpretability than attention heatmaps that are not tightly coupled to a reusable reasoning substrate.

Several limitations are stated. Improving faithfulness while preserving performance remains open, and prototype-based explanations may still not be perfectly causal. The evaluation scope is restricted to general-purpose VQA benchmarks such as Visual7W. The architecture is also limited to multiple-choice answering and grounding-style tasks, rather than free-form, prompt-based, generative VQA with LLMs (Diao et al., 20 Sep 2025). A plausible implication is that questions requiring highly abstract reasoning or evidence that is diffuse and non-local may be less well matched by a small number of spatially constrained prototypes, although this is an inference rather than a separately benchmarked claim.

6. Relation to privacy-preserving symbolic VQA

The privacy-preserving system introduced in “Privacy Preserving Visual Question Answering” is a distinct VQA architecture that can be relevant when the term “ProtoVQA” is used informally (Bara et al., 2022). Its central idea is a hybrid edge/cloud pipeline: a small visual perception model runs on-device, converts the image into a symbolic scene representation, and sends only that non-differentiable symbolic output to a downstream QA model such as MCAN. The privacy argument is that the intermediate output is symbolic rather than feature-based and is therefore “close to impossible” to use for reconstructing the original image.

Its symbolic representation is formed from top predicted classes, top predicted attributes, and bounding-box geometry. For each detected object, the top 5 class predictions are taken and the GloVe embeddings of those class names are concatenated, giving a class representation of $I \in \mathbb{R}^{H \times W \times 3}$2. For attributes, the top 5 attribute predictions are combined as a confidence-weighted sum of GloVe embeddings, giving $I \in \mathbb{R}^{H \times W \times 3}$3. Two bounding-box encodings are used, producing $I \in \mathbb{R}^{H \times W \times 3}$4. The concatenation

$I \in \mathbb{R}^{H \times W \times 3}$5

is then padded to $I \in \mathbb{R}^{H \times W \times 3}$6 before being passed to MCAN (Bara et al., 2022).

The visual model is based on EfficientDet-D0 with an added attribute prediction head, trained on Visual Genome with 1600 object classes and 400 attributes (Bara et al., 2022). The detector uses non-max suppression at IoU 0.5. In the reported comparison, Bottom-Up has AP 32.50, AR 45.67, object-class F1 78.34, and attribute F1 27.93, whereas the proposed EfficientDet-based detector has AP 39.30, AR 50.48, object-class F1 74.87, and attribute F1 50.20. On VQA 2.0, without captions, Bottom-Up end-to-end achieves 67.08 overall, Bottom-Up symbolic GloVe 62.49 overall, the EfficientDet raw variant 56.67 overall, the EfficientDet symbolic GloVe variant 55.41 overall, and the EfficientDet symbolic BERT variant 54.17 overall. With captions, Bottom-Up raw achieves 65.16 overall, Bottom-Up symbolic GloVe 64.58 overall, caption only 57.55 overall, the EfficientDet raw variant 60.10 overall, and the EfficientDet symbolic GloVe variant 59.76 overall (Bara et al., 2022).

Model compression is a major claim in that system: the visual model is stated to be more than 25 times smaller than current state-of-the-art vision models and 100 times smaller than end-to-end state-of-the-art VQA models, with parameter counts of 153M for Bottom-Up and 5.75M for the proposed detector (Bara et al., 2022). The conceptual link to ProtoVQA is limited but clear: both approaches attempt to make the visual side of VQA more inspectable than opaque dense-feature pipelines, but they do so for different reasons. ProtoVQA emphasizes faithful grounding and question-conditioned interpretability, whereas the privacy-preserving architecture emphasizes non-differentiability, on-device perception, and restricted information transfer.