SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Abstract: Vision LLMs (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.

• The paper introduces SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that enables causal analysis of model failures.
• It employs a combinatorial design varying document length, layout, modality, and question type, allowing independent ablation of each factor.
• Empirical results highlight performance drops with increasing context, cross-modal reasoning challenges, and positional sensitivity issues in VLMs.

SynthDocBench: A Diagnostic Benchmark for Long-Context Visual Document Understanding

Motivation and Context

Vision-LLMs (VLMs) have demonstrated significant advances on established visual document understanding (VDU) benchmarks. However, current benchmarks largely fail to capture the full complexity of real-world multi-modal, long-context documents. These documents integrate a diverse range of modalities—including charts, tables, and dense text—distributed across dozens of pages, with compositional queries requiring evidence synthesis across distant content. Existing benchmarks typically isolate either chart reasoning or long-document tasks, which precludes controlled analysis of model failure modes. Moreover, the confounded nature of real-document benchmarks prevents attribution of specific weaknesses, such as those arising from context length, cross-modal evidence integration, or layout variation.

Figure 1: SynthDocBench occupies a previously unexplored region, bridging high document length, high textual density, and dense chart grounding for diagnostic evaluation.

Benchmark Design and Synthetic Generation Pipeline

SynthDocBench addresses the limitations of prior VDU benchmarks by introducing a fully synthetic, controlled, and factorized benchmark for long-context VDU. The benchmark construction builds on a combinatorial design space, where document length, layout archetype, modality composition, and question type are explicitly varied as orthogonal axes. Each axis can be ablated independently, enabling rigorous causal analysis of model behavior as a function of each document property.

Figure 2: End-to-end synthetic visual document generation pipeline with style, layout, chart synthesis, QA generation, and manifest alignment.

The document generation process proceeds via several coupled components:

• Topic-grounded content generation: Given a seed topic $\tau$, evidence is retrieved, sanitized, and reorganized into a structured backbone with explicit section boundaries for controlled allocation of information.
• Layout archetype sampling: A stochastic process (including a 40% random layout override) assigns each document a distinct visual grammar, which prevents models from shortcutting on trivial topic--layout correlations and induces structural diversity.
• Synchronized visualization synthesis: Each chart is rendered using D3.js and also serialized as a structured metadata block containing axiomatic ground truth for downstream QA derivation.
• Automated validation and assembly: Numeric and structural correctness are enforced via metadata recomputation and consistency checks, eliminating the annotation bottleneck inherent in real-document construction.

QA Pipeline, Difficulty Control, and Evidence Integration

The QA generation pipeline parses each synthetic document into structured evidence channels and synthesizes chart-reading, cross-modal, and multi-hop question types. Each question is tagged with a difficulty label ($L1$--$L5$), which serves to stratify the depth of required reasoning, from direct value lookups to multi-source aggregations. A robust automated and manual quality-control regime ensures answer determinism, high evidentiary support, and strong annotation precision.

Figure 3: Controlled QA generation pipeline, extracting structured evidence, synthesizing queries, and validating cross-modal/multihop composition.

SynthDocBench contains 200 reports (mean length 51.1 pages, ~20,568 words, 16.7 charts per document) and 1,788 QA pairs, evenly distributed across chart reading, composition (multi-hop), and cross-modal subsets.

Figure 4: Document composition distribution in terms of pages, tokens, and charts demonstrates unimodal, tightly controlled content statistics.

Model Evaluation Protocol

All candidate VLMs operate under a strict vision-only evaluation regime: models receive only rasterized page images, never latent HTML, metadata, or manifest. Input images are concatenated 5-page strips (144 DPI), ensuring the input format is compatible with diverse model APIs and context constraints. The evaluation pipeline leverages deterministic reference answers and employs a judge model (GPT-5) for automatic scoring on a 0–10 rubric, with rigorous scoring validation (agreement $r \ge 0.94$) across alternative judge LLMs (Gemini, Claude).

Figure 5: Pipeline for vision-only inference—image strip input, model response, and GPT-5 reference-based scoring.

Main Empirical Results and Surfaces Failure Modes

SynthDocBench surfaces a range of failure modes not observed on prior VDU benchmarks:

1. Performance Degradation with Context Length

Unlike prior benchmarks where model performance saturates, all models except Gemini-3.1-Pro and Qwen3.5-VL-122B exhibit monotonic performance loss as document context and evidence complexity increase. While Gemini-3.1-Pro achieves 0.725 ACC (mean score 7.19), no other model approaches this regime; the best open-weight system (Qwen3.5-VL-122B) remains 7 points below. Chart reading, previously considered "solved," degrades sharply in long-context scenarios.

2. Cross-Modal Reasoning is a Limiting Factor

All evaluated models are consistently weakest on cross-modal questions, with cross-modal ACC trailing chart reading or complex multi-hop by 13–16 pp. Models are unable to reliably integrate evidence across modalities separated by multiple pages; OCR-only and vision-only models each have distinct failure loci depending on the subset Figure 6.

Figure 6: Vision-only input outperforms on chart-reading, while OCR excels on evidence-recoverable compositional tasks.

3. Systematic Positional Sensitivity: Lost-in-the-Middle

Models exhibit position-based evidence retrieval bias: chart questions referencing the middle third of a document are significantly more difficult for 5 of 8 models, with Qwen3.5-VL-122B ACC dropping by 18.5 pp from early to middle buckets—a pattern consistent with recency-primacy effects and inefficient long-range memory (cf. "lost-in-the-middle" in LLMs).

4. Visual Hallucination and Failure of Precise Decoding

Visual hallucination—popular plausible but incorrect value extraction—dominates the hardest error cases, especially on dense value-reading charts and for multi-series selection tasks, indicating that pixel-accurate decoding and precise visual-textual alignment remain open challenges. This persists even on structurally controlled, annotation-perfect synthetic evaluation.

Fine-Grained Category and Ablation Analysis

Fine-grained analysis reveals category-specific deficiencies (Figure 7, Figure 8, Figure 9). Trend/pattern detection is generally tractable for vision models, but extraction-based, multi-source integration, and technical quantitative queries induce divergent error signatures across models. Ablation over image context and prompting strategies demonstrates that context density and vision input granularity have significant impact on model performance, with best performance for multi-page concatenation and resolution settings tuned per-model.

Figure 10: Chart type distribution—all 24 types including low-resource and visually ambiguous forms are well-represented, precluding type-specific shortcutting.

Theoretical and Practical Implications

SynthDocBench exposes substantial generalization and compositional weaknesses in current VLMs, attributable to limitations in long-range visual-context fusion, cross-modal evidence retrieval, and chart grounding in dense and variable layouts. The strong empirical gaps between leading open and proprietary models, and between single- and long-context benchmarks, validate the need for controlled, scenario-grounded synthetic benchmarks in vision-language research. The sharp performance drop for chart extraction and multi-hop reasoning in long documents suggests that recent overfitting to formatting and layout priors persists even in the best public-weighted and commercial VLMs.

By separating document properties and question design axes, SynthDocBench enables diagnosis of failure mechanisms that can inform both pretraining data curation (e.g., style and chart diversity), architectural innovations (e.g., context windows, retrieval-augmented attention), and multimodal prompting strategies.

Future Directions

The diagnostic methodology realized by SynthDocBench is extensible to richer multimodal scenarios, including complicated forms, mixed layouts, tabular data, and cross-document queries. Expanding chart rendering backends, scaling synthetic distributions, and introducing noise or real-to-synthetic domain adaptation would further stress-test frontier VLMs and catalyze advances in visual retrieval, data-efficient fine-tuning, and error-resilient cross-modal reasoning.

Conclusion

SynthDocBench provides the vision-language community with a rigorous, controlled, and scalable evaluation suite that decisively reveals current model limitations on long-context document understanding and compositional visual reasoning. Its design enables benchmarking, regression analysis, and ablation unmatched by current real-document benchmarks, and its results will steer model development toward architectures and datasets grounded in true multi-page, multi-modal document complexity (2607.10400).