Bounding Box DocVQA: Spatially Grounded Answers

• Bounding Box DocVQA is a framework that integrates explicit bounding box annotations to extract and localize answers in document images.
• It leverages unified datasets and dual-stream architectures to enhance both spatial evidence grounding and textual answer accuracy.
• Empirical evaluations highlight a performance gap between textual correctness and spatial localization, guiding future improvements in document AI.

Bounding Box DocVQA refers to a line of research, datasets, and model architectures in Document Visual Question Answering (DocVQA) that explicitly incorporate spatial bounding box grounding of answers within the document image. The core objective is not only accurate answer extraction from complex visual documents, but also reliable localization of the evidence or answer region, thereby offering verifiable, interpretable outputs for practical applications and rigorous evaluation of spatial-semantic alignment.

1. Foundations and Motivation

Traditional DocVQA methods focus on extracting appropriate answers—often text strings—from document images in response to natural language queries. However, these systems often underperform in localizing the exact evidence region (“answer box”), limiting their interpretability, traceability, and suitability for use cases such as information retrieval, regulatory compliance, and transparent document analytics. Bounding Box DocVQA explicitly demands models return, along with each answer, one or more bounding boxes $[x_1, y_1, x_2, y_2]$ in image or normalized coordinates that tightly enclose the answer’s supporting text, table, or visual content (Chen et al., 12 Sep 2025, Yu et al., 19 Nov 2025, Giovannini et al., 6 Jan 2025).

This approach operationalizes the principle that robust Document AI should both “know what to answer” and “know where the answer is,” facilitating human-in-the-loop verification, evidence highlighting, and fine-grained spatial reasoning.

2. Datasets with Bounding Box Annotations

Bounding Box DocVQA requires substantially annotated datasets with spatially grounded question–answer pairs. Several major corpora have emerged:

• BoundingDocs v2.0: Consolidates ten public datasets, uniformly re-OCR’d (Amazon Textract) for token and coordinate normalization. Each QA pair is linked to a precise answer box (or boxes), enabling questions formulated for both information extraction and visually rich document understanding tasks (Giovannini et al., 6 Jan 2025).
• BBox-DocVQA: Constructs a large-scale dataset (3,671 papers, ≈42.4k pages, 32k QA pairs) with explicit region-level grounding via a Segment–Judge–and–Generate pipeline. Each QA is associated with one or more bounding boxes derived from high-quality page segmentation (SAM, ViT-H), semantic filtering (Qwen2.5-VL), and question–answer generation (GPT-5). A smaller human-verified subset (1,623 QAs, 80 papers) provides meticulous oracle-quality ground-truth for benchmarking (Yu et al., 19 Nov 2025).

These resources enable fine-grained study of both page-level and subpage-level reasoning, covering a wide range of document types, regions, and challenge scenarios.

3. Model Architectures and Spatial Encoding

Bounding Box DocVQA architectures incorporate spatial localization mechanisms at several levels:

• Dual-Stream Decoupled Models: Approaches such as DocExplainerV0 (“D.E.”) attach an independent bounding-box regression head atop frozen vision–language backbones (e.g., SmolVLM-2.2B, QwenVL2.5-7B). D.E. uses a SigLiP2 dual encoder to generate visual $f_{\text{vis}}$ and textual $f_{\text{txt}}$ features, projects them onto a common latent space, fuses, and regresses the bounding box as $[x_1, y_1, x_2, y_2] \in [0,1]^4$. The answer generator (VLM) is treated as a black box; only the bounding box module is trained via coordinate regression (Smooth L1 loss) (Chen et al., 12 Sep 2025).
• Spatial Embeddings in LLMs: In models inspired by LayTextLLM, each bounding box is projected (via a single-layer “Spatial Layout Projector”) to a learned embedding and interleaved as a token in the input sequence. This “one bbox one token” scheme sharply reduces sequence length compared to naive coordinate representations and fully leverages autoregressive transformer architectures (Lu et al., 2024).
• Prompting Schemes with Spatial Tags: For non-spatial transformer architectures, bounding box information is injected as explicit tags within tokenized OCR inputs (e.g., “[WORD]@[x₁/1000,y₁/1000,x₂/1000,y₂/1000]”), and spatial embeddings are added to token encodings via linear projections (Giovannini et al., 6 Jan 2025).

These advances support both end-to-end models and plug-and-play adapters for proprietary or frozen large-scale VLMs.

4. Evaluation Metrics and Experimental Benchmarks

Assessment of Bounding Box DocVQA models is inherently multi-objective:

• Textual Accuracy: Typically evaluated via Average Normalized Levenshtein Similarity (ANLS), Exact Match (EM), or task-adapted F1 scores (Chen et al., 12 Sep 2025, Giovannini et al., 6 Jan 2025, Lu et al., 2024).
• Spatial Localization: The principal metric is mean Intersection over Union (IoU) between predicted and ground-truth boxes:

$$\mathrm{IoU}(B_{\text{pred}}, B_{\text{gt}}) = \frac{|B_{\text{pred}} \cap B_{\text{gt}}|}{|B_{\text{pred}} \cup B_{\text{gt}}|}$$

Extensions include mean Average Precision (mAP) at threshold $\tau$ and Recall@K for multi-box cases (Yu et al., 19 Nov 2025).

• Parsing Robustness: When answers must be output as structured JSON, the JSON parsing error rate is also tracked (Giovannini et al., 6 Jan 2025).

Empirical findings consistently reveal a substantial performance gap between strong answer accuracy and effective spatial localization. For example, standard VLM prompts achieve ANLS up to 0.72 but with MeanIoU often $<$0.06, whereas plug-in regressors such as DocExplainerV0 drive MeanIoU up by $4\times$–$15\times$ (to $\sim$0.18) with negligible impact on answer accuracy. However, perfect OCR-grounded methods still set the upper bound (MeanIoU $\sim$0.45–0.49) (Chen et al., 12 Sep 2025).

A sample quantitative comparison is given below (select models on BoundingDocs v2.0, (Chen et al., 12 Sep 2025)):

Model / Prompting	ANLS	MeanIoU
QwenVL2.5 zero-shot	0.691	0.048
QwenVL2.5 + DocExplainerV0	0.689	0.188
Naive OCR baseline (Qwen)	0.690	0.494

In BBox-DocVQA, even the largest VLMs (Qwen3-32B) show mean IoU $\leq$ 0.20–0.35 and substantial declines in multi-region/page scenarios (Yu et al., 19 Nov 2025).

5. Dataset Construction, Prompt Design, and Best Practices

Dataset schema and prompt design are central:

• Pipeline for Unified Datasets: BoundingDocs v2.0 harmonizes multiple datasets by re-OCR’ing, matching keys and answers to Textract lines/words (via Jaccard over word sets), and templating/rephrasing questions for natural language variety. All spatial coordinates are normalized (integer $[0,1000]$ or unit $[0,1]$ intervals) (Giovannini et al., 6 Jan 2025).
• Prompt Variants: Empirical study demonstrates that embedding explicit bounding boxes in token sequences (“Reph-Reph-bbox” variant) yields the highest accuracy, especially on layout-heavy tasks—e.g., a +8.4 ANLS* point gain on SP-VQA (Giovannini et al., 6 Jan 2025).
• Multi-Region and Multi-Page: BBox-DocVQA introduces categories (SPSBB, SPMBB, MPMBB) covering single/multiple regions and pages. Oracle studies show that providing ground-truth crops—i.e., perfect region grounding—can improve answer accuracy by 10–25 percentage points, demonstrating the critical role of spatial evidence in DocVQA (Yu et al., 19 Nov 2025).

Recommendations for practitioners include training/fine-tuning with rephrased questions and normalized bounding-box cues, and deploying robust post-processing (e.g., regex fallback) for nontrivial QA formatting (Giovannini et al., 6 Jan 2025).

6. Challenges, Limitations, and Future Directions

Despite significant architectural innovations, a persistent challenge remains: there is a marked discrepancy between models’ ability to answer correctly and to reliably localize supporting content. Models frequently exploit document-level statistical priors or global token context instead of true region grounding, especially in complex, multi-page, or highly structured documents (Chen et al., 12 Sep 2025, Yu et al., 19 Nov 2025). Localization degrades as task complexity increases (SPSBB→MPMBB), with IoU and accuracy both dropping.

Areas identified for further research and improvement include:

• Region proposal and evidence selection: Training VLMs with explicit region/evidence selection modules (Yu et al., 19 Nov 2025).
• Chain-of-thought with grounding awareness: Integrating multi-step, spatially aware reasoning into generation flows.
• Extended metrics: Adopting recall@K and mAP measures for richer evaluation of multi-candidate region proposals.
• Model robustness: Enhancing resistance to OCR noise, layout variability, and imperfect spatial cues through hybrid representations and refined annotation schemas.

A plausible implication is that continued advances in bounding-box grounded DocVQA will be pivotal to building explainable Document AI systems capable of both accurate retrieval and auditable, interpretable output.

7. Significance and Impact

Bounding Box DocVQA constitutes a rigorous new standard for document AI research and deployment by demanding spatially explicit output. It bridges the gap between answer string generation and spatial evidence localization, enabling applications in legal, financial, and scientific document automation that require high confidence, traceability, and human verifiability. Its emergence has established new benchmarks (BoundingDocs, BBox-DocVQA), revealed key weaknesses in state-of-the-art VLMs, and catalyzed innovation in spatial-semantic architectures (Chen et al., 12 Sep 2025, Yu et al., 19 Nov 2025, Giovannini et al., 6 Jan 2025, Lu et al., 2024).

It is anticipated that Bounding Box DocVQA, with its focus on fine-grained grounding and explicit spatial-semantic alignment, will remain central to future multimodal document research and real-world vision–language applications.