---
title: Document Visual Question Answering
url: https://www.emergentmind.com/topics/document-visual-question-answering-vqa
type: topic
---

# Document Visual Question Answering

Document Visual Question Answering (VQA) is a research area focused on developing machine learning models capable of answering natural language questions by reasoning over the visual and structural content of documents. These systems integrate components for visual feature extraction, text detection, and sophisticated multimodal reasoning to process documents that may include diverse visual layouts, dense text, tables, diagrams, or forms. In contrast to conventional VQA, which primarily deals with photographs, document VQA leverages optical character recognition (OCR), spatial layout modeling, and fusion of visual and linguistic signals to solve tasks—ranging from span extraction to relational reasoning or evidence retrieval—over both single documents and multi-page or multi-document collections [2008.08899][2501.03939][2104.14336][2304.06447][2301.04883].

## 1. Problem Formulation and Task Variants

Document VQA tasks are framed as answering questions \( Q \) posed about document images \( D \) or image collections \( D = \{d_1, d_2, \ldots, d_n\} \). The task formulation and requirements depend on task granularity and context:

- **Single-Document QA:** Given a document image \( d \) and question \( Q \), extract an answer \( A \) as a text span or short answer present in \( d \). Questions span various categories (figure, table, layout, yes/no, handwriting) and typically require reading or structural understanding of the page [2008.08899].
- **Collection-Level and Multi-Image QA:** The input is a set of images \( D \) and question \( Q \). The system must:
  - Retrieve an evidence subset \( R \subseteq D \) containing all documents or pages necessary for reasoning.
  - Produce an answer \( A = f_{\mathrm{ans}}(Q, R) \), where \( f_{\mathrm{ans}} \) may involve aggregation, logic, or arithmetic across multiple documents [2104.14336][2301.04883].
  - Some benchmarks (e.g., SlideVQA) explicitly annotate both evidence (slide indices) and answers, requiring multi-hop and numerical reasoning [2301.04883].

Question and answer types include extraction (free text, region index), binary (yes/no), counts, range queries, and relational answers (parent/child section linkages, arithmetic expressions) [2008.08899][2304.06447][2301.04883].

## 2. Datasets and Annotation Protocols

Document VQA has been advanced by several benchmark datasets, each characterizing distinct aspects of the task:

| Dataset          | Images / Pages | Questions / QA Pairs | Special Features                                 | Reference         |
|------------------|----------------|---------------------|--------------------------------------------------|-------------------|
| DocVQA           | 12,767         | 50,000              | Layout-diverse docs, 9 Q types                   | [2008.08899]      |
| DocCVQA          | 14,362         | 20                  | Collection-level, evidence subset, complex queries| [2104.14336]      |
| PDF-VQA          | 111,538        | 1,012,263           | Multi-page PDFs, element/structural/relational Q  | [2304.06447]      |
| SlideVQA         | 52,000+        | 14,500+             | Slide decks, multi-hop, arithmetic, annotations   | [2301.04883]      |

Annotation protocols leverage OCR outputs with manual correction for ground-truth fidelity. Task-specific annotation includes linking answers to evidence slides/pages, constructing explicit arithmetic expressions for numerical questions, and QA template filling for structural relations. A key trend is the move toward large-scale, multi-page, and semantically-rich annotation, enabling robust multi-document reasoning evaluation [2104.14336][2304.06447][2301.04883].

## 3. Model Architectures and Reasoning Paradigms

Document VQA models can be grouped into three architectural families [2501.03939]:

**A. Two-Stage (OCR → Reasoner) Pipelines:**
- First, OCR detects and recognizes text regions, extracting tokens and bounding boxes (e.g., via Tesseract or CRAFT).
- Next, extracted text/position features and the query are encoded by a neural reasoner (often a Transformer or BERT/QA model).
- Sample: Text-spotting coupled with BERT span-extraction, sometimes employing key–value extraction and structured queries [2104.14336].

**B. End-to-End Document-Aware Transformers:**
- Unified vision-language Transformers ingest image patches/tokens (from CNN or ViT), positional/layout encodings, OCR tokens, and question embeddings.
- Self- and cross-attention layers enable joint reasoning over spatial, textual, and visual signals.
- Examples: LayoutLMv2 (BERT with layout/visual features), DocFormer (self/cross-attention), M4C (pointer network over OCR-object tokens) [2501.03939][2304.06447].
- Graph-based GCN: PDF-VQA introduces GCNs over document elements (nodes with visual+textual features, spatial/hierarchical edges) for structural and relational reasoning [2304.06447].

**C. Large Vision-Language Models (LVLMs) with Document Adapters:**
- LVLMs (BLIP-2, Donut, InstructBLIP) are adapted for documents by inserting OCR/layout adapters or prompt tuning.
- Donut uses a ViT-VQGAN encoder and decoder, predicting answer tokens directly from image inputs.
- Adapters map detected text or layout signals into LVLM embedding spaces, enabling document grounding with minimal model modification [2501.03939].

Multi-document tasks have seen the emergence of unified sequence-to-sequence models: SlideVQA frames both evidence retrieval and answer extraction as a single generation problem via a transformer-based encoder–decoder sequence output “evidence indices || answer” [2301.04883].

## 4. Evaluation Metrics and Benchmarking

Evaluation protocols are multi-faceted, commonly divided by task type:

- **Span/Extractive QA:** Average Normalized Levenshtein Similarity (ANLS), granting partial credit for variants and near-matches:
  \[
  \mathrm{ANLS} = \frac{1}{N} \sum_{i=1}^N \max_j S(a^j_i, \hat{a}_i)\,,\quad S(a,b) = \max\bigl(0,1 - \frac{\mathrm{Lev}(a,b)}{\max(|a|,|b|)}\bigr)
  \]
- **Retrieval:** Mean Average Precision (MAP), Precision@k, and Recall@k for evidence document selection tasks. Combined DocCVQA metrics multiply retrieval $F_r$ and QA ANLS or accuracy for holistic scoring [2104.14336].
- **Multi-Answer List Matching:** ANLSL (ANLS for lists via optimal bipartite matching), capturing order-agnostic multi-item accuracy [2104.14336].
- **Sequence Models:** Main metrics jointly average evidence-selection F1 and answer F1 (SlideVQA). Jaccard overlap for evidence slide indices.
- **Discrete Tasks:** Exact match for binary/counting/region-class answers, F1 for multi-token spans, and accuracy for element or relation prediction [2304.06447].

Representative model performance on public benchmarks includes:
- DocVQA (Test): Top ANLS ∼0.85 (PingAn-OneConnect-Gammalab-DQA); Baseline M4C: ∼0.52; LayoutLMv2: ∼0.70; Donut: ∼0.60 [2501.03939][2008.08899].
- DocCVQA: Retrieval MAP ∼73% (Text-spotting+BERT) with ANLSL 0.45–0.71 across baselines; Database+Database achieves highest QA scores [2104.14336].
- PDF-VQA: Graph GCN model outperforms six baselines (acc. 78.9% overall, +10–12% over pointer-based M4C on reasoning tasks) [2304.06447].
- SlideVQA: Unified seq2seq models achieve ∼40 joint F1 (vs. human ∼85); numerical reasoning remains most challenging [2301.04883].

## 5. Advances in Multimodal and Multidocument Reasoning

Recent developments focus on modeling hierarchical document structure, spatial relationships, and evidence aggregation:

- **Graph-based Reasoning:** PDF-VQA demonstrates that representing documents as heterogeneous graphs (nodes: elements; edges: spatial/hierarchical) and applying Relational GCNs yields improved multi-hop relational understanding and layout-sensitive predictions. Ablation studies show that removing hierarchy or spatial edges drops accuracy by 3–4% [2304.06447].
- **Unified Generation:** SlideVQA’s seq2seq approach is empirically superior to cascaded retrieval+reader baselines for evidence selection and answer synthesis, especially for multi-slide and arithmetic questions [2301.04883].
- **Multi-page and Cross-document Pretraining:** Pretraining document Transformers to directly handle multiple pages or documents, leveraging retrieval supervision and cross-page attention, is considered a next step for improving collection-level VQA [2104.14336].
- **Numeric and Symbolic Reasoning:** Datasets such as SlideVQA annotate intermediate arithmetic formulas, pushing models to ground entities and reason symbolically, with embedded neural arithmetic components suggested as future directions [2301.04883].

## 6. Applications, Challenges, and Outlook

Document VQA underpins essential workflows in finance (invoice extraction, cross-form consistency checks), insurance (multi-page claim analysis), archive research (historical record aggregation), contract understanding, and compliance auditing [2104.14336][2501.03939]. Robust deployment requires accurate OCR, domain-adaptive layout modeling, and low-latency inference on privacy-sensitive data.

Ongoing challenges include:
- Propagation and correction of OCR errors.
- Generalization to diverse and unseen document templates or handwriting.
- Integrating structured data (tables, charts) with unstructured text for seamless downstream QA.
- Cross-document and hierarchical multi-level reasoning for paginated or collection-based corpora.
- Scarcity of cross-lingual and low-resource document QA resources.
- Explainability through evidence highlighting, supporting regulatory or user-facing transparency [2501.03939].

Future research is expected to further integrate graph neural architectures, layout-aware large vision–language models, and domain-specific adapters, with a focus on unified retrieval–reasoning frameworks capable of interactive, chain-of-thought elaboration over visually and linguistically complex document ecosystems [2104.14336][2304.06447][2301.04883][2501.03939].

Source: https://www.emergentmind.com/topics/document-visual-question-answering-vqa