Knowledge-Based Visual Question Answering

• Knowledge-Based Visual Question Answering (KB-VQA) is a paradigm that combines visual, textual, and external knowledge to answer questions beyond direct image cues.
• The ROCK model employs BERT-based retrieval and multimodal fusion to integrate visual features, subtitles, and annotated knowledge, boosting accuracy by about 6.5%.
• The KnowIT VQA dataset, curated from 'The Big Bang Theory', exemplifies how annotated episodic knowledge enhances reasoning over video content.

Knowledge-Based Visual Question Answering (KB-VQA) refers to methods that require a system to answer questions about visual content by leveraging both the visual input and external, often structured, knowledge. Unlike conventional VQA, which is typically grounded only in the information visible in an image (or a video), KB-VQA requires integrating multimodal representations with outside knowledge—such as textual facts, contextual cues, or commonsense—often in a structured, retrievable database or as explicit annotations. KB-VQA is essential for answering questions that cannot be resolved from visual or textual content alone, such as those involving episodic context, world knowledge, or information implicitly accumulated across a narrative.

1. Dataset Construction and Annotation

A central contribution to KB-VQA is the design of datasets that demand external knowledge for answering. The KnowIT VQA dataset is an example tailored for video: it is constructed from 207 episodes of “The Big Bang Theory” sitcom and comprises 12,087 video clips annotated with a total of 24,282 question–answer pairs. Each video clip is paired with:

• A question
• One correct answer and three plausible distractors
• A short natural language “knowledge” sentence per clip, encapsulating the episodic or contextual insight critical for correct answering

Questions are classified by their core reasoning requirement: visual, textual, temporal, or knowledge-based, yielding several axes of reasoning complexity. Annotators provide knowledge sentences that serve as minimal context required to answer—e.g., “Penny has just moved in”—allowing the dataset to capture dependencies often only clear to seasoned viewers of the show.

This multifaceted annotation protocol ensures simultaneous integration of visual cues, textual dialogue/subtitles, temporal structure (event coherence across segments), and external knowledge (Garcia et al., 2020).

2. Model Architectures for KB-VQA

The ROCK model demonstrates a modular approach to fusing multimodal context with auxiliary knowledge. The architecture has three primary components:

A. Knowledge Base (KB):

A set $\mathcal{K} = \{w_1, w_2, ..., w_N\}$, where each $w_j$ is a natural language fact or contextual snippet curated by human annotators.

B. Knowledge Retrieval:

Uses a BERT-based similarity scoring function: Given question–answer pair $q_i$ and knowledge sentence $w_j$, a BERT-style model computes $s_{ij}$; training leverages binary cross-entropy loss to maximize $s_{ij}$ for relevant (i = j) pairs and minimize for irrelevant ones. At inference, the top-$k$ knowledge entries are selected for a given question.

C. Multimodal Video Reasoning:

The model encodes each modality:

• Visual: Concatenated ResNet50 frame features, bag-of-words for object/concept detection, face recognition (main characters), and generated captions
• Textual: Subtitle sequence and generated captions These features, along with the focused knowledge from the retriever, are concatenated and projected in a BERT-based language encoder ("BERT-reasoning") which outputs a combined representation. For classification, multimodal features $[V||L]$ (visual $V$, aggregated language $L$) are scored as

$\text{score} = W \cdot [V || L] + b,$

selecting the candidate answer with the maximal score. This approach allows flexible integration of diverse content streams and is particularly suited to domains where respective sources provide complementary, non-overlapping cues (Garcia et al., 2020).

3. Knowledge Integration Strategies

Explicit knowledge integration is critical in closing the gap between what models can extract from video frames, subtitles, and what is required for high-fidelity answering. In the KnowIT VQA setting:

• Each question–clip pair is associated with a knowledge sentence that bridges event-based or character-based context that might not be visually observable (e.g., unseen relationships, off-screen events).
• Retrieval is not only over candidate knowledge but also over alternate candidate answers, supporting dense distractor selection.
• The retrieval and fusion steps allow the model to factor in not only on-screen events but also episodic, relational, or background knowledge—simulating human reliance on “watching experience.”

This integration is essential when scene understanding alone is insufficient, and the incorporation of even short knowledge snippets can shift answer prediction accuracy by over 6.5% compared to non-knowledge models.

4. Performance Analysis and Human Comparison

Quantitative evaluation on the KnowIT VQA dataset reveals the following:

• ROCK achieves a substantial boost—about 6.5% absolute increase in answer accuracy—over earlier multimodal approaches (such as TVQA) that lack explicit knowledge integration.
• Human accuracy, even for “Rookie” annotators unfamiliar with the show but provided with the same data, remains markedly higher (approximately 74.8%) than ROCK.
• The disparity is further exacerbated for expert populations (“Masters”) and for questions requiring composite, knowledge-rich reasoning, highlighting the inherent difficulty of narrative and temporal coherence in video.

The performance gap underscores that while external, context-specific knowledge is necessary for challenging KB-VQA, it does not alone suffice: representing long-term, multi-hop, or narratively intertwined events remains a bottleneck.

5. Limitations and Implications for KB-VQA Research

Key limitations identified relate to:

• Incomplete temporal coherence: even with knowledge sentences, models still struggle to carry narrative dependencies across episodes or scenes.
• Imperfect fusion: While multi-modal concatenation followed by a linear classifier and BERT-based encoding improves performance, it likely underexploits deeper entanglement between visual, textual, and knowledge sources.
• Dataset specificity: The performance gain is highly dependent on the quality and coverage of annotated knowledge; the generalization to other domains or less-structured sources may reduce efficacy.

Implications for future KB-VQA research include:

• The necessity to develop architectures that can reason over temporal event structures (narrative-based VQA), possibly with memory or graph-based mechanisms.
• The importance of modular designs that allow seamless integration of explicit annotated knowledge, multimodal feature streams, and advanced retrieval mechanisms.
• The value of developing and evaluating against datasets that tightly couple visual, textual, temporal, and contextual knowledge, as in KnowIT VQA, to expose persistent model deficiencies (Garcia et al., 2020).

6. Broader Impact and Directions

The introduction of KnowIT VQA and models such as ROCK establish:

• The paradigm of episodic, knowledge-enhanced video VQA as a testbed for next-generation visual reasoning models.
• That bridging the gap toward human-like performance in video understanding is less about scaling vision architectures and more about tightly weaving in knowledge representation and retrieval, especially in narrative-rich domains.

This framework has set the stage for research focusing on:

• More sophisticated, possibly attention-based, knowledge integration and fusion techniques.
• Continued development of datasets combining explicit human-annotated episodic knowledge, visual streams, and multilayered textual data.
• Further systematic evaluation on human-annotated tasks that faithfully mimic the multimodal, background-dependent tasks encountered in the real world.

In summary, Knowledge-Based Visual Question Answering as defined and operationalized in the referenced work constitutes a fundamental advance in simulating the human experience of watching, comprehending, and answering complex narratives, validating the centrality of external, context-specific knowledge in multimodal AI.