Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents

Abstract: Institutional documents contain substantial amounts of operational and analytical information embedded within figures and tables. Current approaches for extracting visual content from documents are largely built around generic document layout analysis, where figures and tables are treated as uniformly relevant document objects rather than semantically meaningful analytical artifacts. In this work, we introduce a benchmark dataset and evaluation framework for \textit{data snapshot extraction}, the task of identifying and localizing semantically meaningful visual artifacts within institutional documents. The benchmark spans humanitarian reports, World Bank policy research working papers, and project appraisal documents, and includes annotations for figures and tables that contain reusable analytical information. Using this dataset, we benchmarked multiple open-source layout detection models and evaluated both detection performance and spatial extraction quality. Our results show that current models struggle to generalize to operational institutional documents despite strong performance on conventional academic benchmarks. Common failure modes include confusion between analytical and non-analytical content, fragmentation of composite analytical artifacts, and incomplete extraction of contextual information required for interpretation. These findings highlight a persistent gap between generic document layout analysis and operationally useful data snapshot extraction. We release the source PDFs, annotation dataset, metadata, and source code to support future research in operational document intelligence. The dataset is available at https://huggingface.co/datasets/ai4data/data-snapshot and the source code is available at https://github.com/worldbank/ai4data/tree/main/experimental/data-snapshot.

• The paper introduces a novel benchmark dataset for distinguishing semantically relevant data snapshots from generic visual objects in institutional documents.
• The paper demonstrates that transformer-based models achieve superior spatial accuracy (e.g., IoU up to 0.919) while YOLO-based models show higher recall with tradeoffs in precision.
• The paper identifies failure modes such as semantic confusion and incomplete extraction, highlighting the need for specialized operational document intelligence pipelines.

Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents

Scope and Motivation

The paper "Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents" (2606.06242) addresses a pivotal challenge in operational document intelligence: the extraction of semantically meaningful analytical artifacts (termed "data snapshots") from institutional documents. The authors distinguish between generic layout detection—identifying figures and tables indiscriminately—and the operational necessity to isolate only those visual artifacts which possess semantic and analytical relevance. This distinction is especially critical in contexts such as humanitarian reports, policy research papers, and project appraisal documents, where data and actionable insights are often embedded within complex visual structures and not conveyed in narrative text.

To rigorously benchmark existing solutions, the authors introduce a dataset comprising annotated visual artifacts across three document corpora: ReliefWeb/UNHCR humanitarian reports, World Bank policy research working papers (PRWP), and refugee-focused project appraisal documents (PADs). The annotation protocol distinguishes between merely visual objects and those carrying operational or analytical utility. This dataset—a resource exposing diverse document compositions—constitutes a foundation for evaluating and developing future document intelligence models.

A salient empirical finding is that analytically relevant data snapshots appear on only a minority of document pages (median prevalence ≈ 20% per document), underscoring the inefficiency of indiscriminate multimodal processing pipelines. This motivates the need for specialized snapshot localization mechanisms prior to downstream analysis.

Figure 1: Distribution of the fraction of pages containing at least one data snapshot within the PRWP and Refugee PAD corpora; the operational prevalence is typically one page in five.

Benchmark Construction and Corpus Characteristics

Three corpora were used to construct the benchmark, each representing distinct operational document genres:

• UNHCR/ReliefWeb: Humanitarian analysis reports with dense infographics, monitoring maps, and summary cards.
• PRWP: Policy research papers with econometric tables, analytical figures, and comparative analysis layouts.
• Refugee PADs: Project appraisal documents featuring financing matrices, implementation tables, and operational dashboards.

The annotation process employed semi-assisted human-in-the-loop review, leveraging models for pre-labeling but ultimately relying on manual validation to ensure semantic correctness and contextual completeness. The released dataset, including bounding-box annotations, document metadata, and benchmarking code, guarantees reproducibility and supports further research.

Evaluated Layout Detection Models

Four open-source document layout detection models were benchmarked, covering both transformer-based and YOLO-family document detector architectures:

• TF-ID-Large: Transformer-based, trained on scientific papers with explicit caption annotation.
• DocLayout-YOLO: YOLOv10 adaptation, fine-tuned on diverse synthetic and human-annotated layout datasets.
• YOLOv11 (medium variant): Lightweight YOLO-based detector, fine-tuned on DocLayNet.
• YOLOv26 (medium variant): Latest YOLO-based detector, trained on updated DocLayNet v1.2.

Each model's inference pipeline integrated bounding-box area filtering to suppress detections of non-analytical small graphical elements.

Evaluation Metrics and Framework

Detection efficacy was quantified via precision and recall at a fixed IoU threshold of 0.5, enabling interpretable matching of predictions to ground-truth analytical artifacts. Beyond geometric localization, spatial extraction quality was assessed through:

• Area Recall: Proportion of actual artifact area captured.
• Area Precision: Proportion of predicted region corresponding to ground-truth.
• IoU: Intersection-over-union for spatial alignment.

This dichotomy of metrics exposes the tension between generic object localization and semantically complete extraction, especially in documents containing non-standard visual layouts.

Quantitative Results and Model Analysis

A pronounced tradeoff was observed between detection performance and extraction quality:

• TF-ID-Large demonstrated highest spatial accuracy for both figures (IoU = 0.877, Area Recall = 0.938) and tables (IoU = 0.919, Area Recall = 0.946), reflecting robust inclusion of titles, legends, and explanatory captions.
• YOLO-based models exhibited higher recall for tables (0.862--0.893) and figures (0.761--0.802), yet suffered from lower precision, frequently misclassifying decorative or operationally irrelevant content.
• DocLayout-YOLO consistently outperformed other YOLO variants, likely due to greater training diversity.

These results highlight the inadequacy of generic layout detectors for operational document intelligence, as precision (correct identification of semantically relevant artifacts) remains modest despite high recall.

Failure Modes and Domain Gaps

Qualitative analysis reveals systematic failure modes:

• Semantic confusion: Decorative images misidentified as figures; formatting tables and tables of contents detected as analytical tables.
• Fragmentation: Composite dashboards and multi-panel infographics split into disparate detections rather than captured as a unified analytical artifact.
• Incomplete contextual extraction: Models omitting critical titles, captions, footnotes, or legends even when satisfying IoU thresholds.

Such failure modes, endemic across document genres, illustrate both semantic and visual domain gaps between conventional layout benchmarks and operational requirements in institutional documents.

Implications for Document Intelligence and AI Development

The results substantiate several practical and theoretical implications:

• Operational document AI pipelines require semantic snapshot extraction: Downstream tasks—retrieval, indexing, summarization, multimodal reasoning—depend on contextually complete and analytically relevant visual crops. Deduplication of irrelevant or incomplete artifacts is thus paramount.
• Humanitarian and policy research documents necessitate models calibrated for composite and non-standard layouts: Training data diversity, annotation philosophy, and boundary definitions critically influence model generalization.
• Current open-source detectors are insufficient for large-scale deployment in institutional contexts: There is substantial room for developing specialized models and segmentation strategies (potentially hierarchical or non-rectangular), targeting semantic artifact boundaries.

Future advancements may include multimodal, context-aware layout models with dynamic cropping anchored in downstream analytical requirements, as well as new datasets encompassing broader operational genres and annotation protocols reflecting hierarchical or relational artifact structure.

Limitations and Path Forward

Primary limitations include subjectivity in annotation, reliance on rectangular bounding boxes for artifact crops, focus on localization rather than downstream semantic interpretation, and coverage limited to development/humanitarian corpora. Addressing these, future benchmarks may incorporate segmentation masks, hierarchical annotations, and expanded domain coverage.

Conclusion

The study rigorously benchmarks open-source layout detection models on operational institutional documents, revealing a persistent gap between academic layout benchmarks and semantically meaningful analytical artifact extraction. The public dataset, code, and findings facilitate development of models and evaluation methodologies aligned with real-world document intelligence workflows. The extraction of data snapshots—a distinct, operationally critical task—demands future research in model architectures, annotation protocols, and evaluation metrics tailored to the demands of institutional document ecosystems.