Closed Visual Question Answering (CVQA)

• CVQA is a visual question answering paradigm where an image is converted into textual context to select the correct answer from a fixed set.
• The method employs a reading comprehension framework using QANet, combining image captions and external factual sentences to enhance answer accuracy.
• Key challenges include dependency on caption quality and handling abstract candidate answers, with future work aimed at joint training and advanced OCR integration.

Closed Visual Question Answering (CVQA) refers to the class of visual question answering problems in which each query consists of an image, a natural-language question, and a finite set of candidate answers from which exactly one is to be selected. This contrasts with open-ended VQA, where the answer is generated freely as text. In CVQA, the challenge is to exploit visual and linguistic cues to disambiguate among multiple-choice options, often demanding context integration, commonsense reasoning, and, occasionally, incorporation of external knowledge bases. The paradigm has provided a fertile ground for evaluating multimodal understanding under well-controlled, answer-constrained conditions.

1. Reformulating CVQA as Reading Comprehension

A key development in recent CVQA research is the reframing of the problem from multimodal fusion to pure text-based reading comprehension. Li et al. (Li et al., 2018) propose that, by translating both the image and any necessary supplemental information into natural language, the multimodal complexity can be sidestepped. Each image is first rendered as a “context” paragraph $S$ by aggregating region descriptions—either ground-truth annotations (e.g., from Visual Genome) or automatically generated dense captions. For knowledge-intensive tasks, externally retrieved factual sentences $\{F_j\}$ can be appended, yielding $S = [D_1; \dots; D_K; F_1; \dots; F_M]$. The question $Q = [q_1, \dots, q_m]$ and each candidate answer $A_i = [a^i_1, \dots, a^i_{r_i}]$ are then composed, and the decision is cast as a reading-comprehension selection problem. This reduction is directly applicable to closed VQA, as all components are rendered in natural language.

2. Model Architecture and Workflow

The approach adopts a reading comprehension backbone, specifically QANet, characterized by three principal stages: word-plus-character embeddings, a stack of encoder blocks (depthwise separable convolutions, multi-head self-attention, and layer normalization), and a context–query attention mechanism. For each triplet $(S, Q, A_i)$, the question and candidate answer are concatenated to form $T_i = [Q; A_i]$, which serves as the “query” to the model.

The architecture computes two parallel attention streams for each candidate: context–question attention (yielding $M^{(q)}$) and context–answer attention ($M^{(a)}_i$). Each stream is processed through cascaded model-encoder layers, whose outputs are average-pooled and concatenated to form a fixed-length vector $$h_i = [v^{(a)}_{0,i}; v^{(a)}_{1,i}; v^{(q)}_{0}; v^{(q)}_{1}]$$. This vector is fed through a two-layer MLP with ReLU and dropout, producing a scalar logit $\{F_j\}$0, which is then passed through a sigmoid to yield the predicted answer probability $\{F_j\}$1. During inference, the answer with the highest $\{F_j\}$2 is selected, $\{F_j\}$3.

Training uses each correct triplet along with two randomly sampled negatives, minimizing binary logistic (cross-entropy) loss per triplet:

$\{F_j\}$4

where $\{F_j\}$5 for the correct answer and $\{F_j\}$6 otherwise. Regularization is achieved via dropout and weight decay.

3. Integration of External Knowledge

CVQA often requires reasoning over world knowledge not present in the image. In the reading-comprehension framework, external facts are incorporated straightforwardly by appending textual fact sentences to the image-derived context. For example, in FVQA (Fact-based VQA), structured knowledge base queries are issued to retrieve relevant facts, which are then included in $\{F_j\}$7. Critically, no further architectural changes are needed; the same model processes this extended context. Empirical results demonstrate that such integration yields robust performance improvements, with a top-1 FVQA accuracy of $\{F_j\}$8, surpassing previous methods by over $\{F_j\}$9 (Li et al., 2018).

4. Quantitative Benchmarks and Analysis

The model is evaluated on the Visual7W “Telling” task, a canonical closed-VQA benchmark. The following table presents overall top-1 accuracy scores reported in (Li et al., 2018):

Model (Context)	Accuracy (%)
LSTM+CNN baseline	52.1
Visual7W official	55.6
MCB [Fukui et al.]	62.2
MLP [Goyal et al.]	67.1
MAN [Zhang et al.]	59.4
KDMN-NoKG [Zhu et al.]	62.0
RC (ground-truth desc.)	69.8
RC (densecap)	60.2
RC (densecapJoint)	61.2

When human-annotated region descriptions are used, the reading-comprehension formulation outperforms the prior state-of-the-art (MLP) by more than 2.7 points. With predicted captions (from dense captioners), the method matches or outperforms previous approaches that do not extensively leverage external knowledge. A per-question-type breakdown (using automatic captions by [Johnson et al.]) is as follows:

Type	Proportion (%)	Accuracy (%)
What	47.8	58.4
Where	16.5	64.9
When	4.5	75.1
Who	10.0	70.2
Why	6.3	56.3
How	14.9	50.8
Overall	100.0	60.2

A sharp 9–10 point accuracy drop is observed between ground-truth and automatic captions, underlining the central importance of high-quality language-based image descriptions.

5. Advantages, Limitations, and Potential Extensions

The textual reformulation of CVQA leverages advances in machine reading comprehension, including efficient contextual encoders and large-scale language pretraining. By bypassing multimodal feature fusion, it facilitates easy integration of external knowledge—new facts are incorporated simply by extending the context string. However, performance is closely tied to the fidelity of the language surrogate for the image; omissions or errors at the captioning stage are irrecoverable further downstream. Abstract or non-exact candidate answers (e.g., “During the daytime”) that do not appear verbatim in the context present additional challenges, as inference must occur beyond co-occurrence.

Potential future directions enumerated by (Li et al., 2018) include end-to-end joint training of image captioning and reading comprehension modules to bias the captioner toward generation of VQA-relevant details, integration of OCR and scene-graph–based representations to capture fine-grained textual or relational cues, and large-scale pretraining on synthetic or text-only QA datasets for robustness to missing or noisy facts.

6. Conclusion and Significance

CVQA, when cast as a textual multiple-choice reading comprehension problem, offers a compelling alternative to traditional multimodal modeling by unifying modality information at the language level. This approach provides competitive, frequently superior empirical results on standard benchmarks, and presents a structurally simple, extensible framework for both observation-based and knowledge-intensive visual question answering (Li et al., 2018). The central determinant of system performance becomes the quality and completeness of the linguistic image representation, positioning advances in automatic captioning and structured knowledge retrieval as primary levers for further progress in closed VQA.