MMLongBench-Doc: Long-Context Document Benchmark

• MMLongBench-Doc is a long-context, multimodal document understanding benchmark designed for visual question answering over lengthy PDFs with complex layouts.
• It evaluates models’ ability to localize, integrate, and reason with evidence across multiple pages and diverse modalities like text, tables, charts, and images.
• Empirical results reveal that even leading LVLMs, such as GPT-4o, struggle with cross-page reasoning and hallucination detection in these challenging settings.

MMLongBench-Doc is a long-context, multi-modal document understanding benchmark for visual question answering over lengthy PDF documents, designed to evaluate whether large vision-LLMs can localize, integrate, and reason over evidence distributed across pages, modalities, and layout structures. It is built on 130 PDF documents with 1,062 expert-annotated questions, with documents averaging 49.4 pages and 20,970.9 tokens; 33.2% of the questions are cross-page and 22.8% are unanswerable, making hallucination detection part of the benchmark rather than an auxiliary concern (Ma et al., 2024).

1. Scope and defining characteristics

MMLongBench-Doc targets long-context document understanding rather than single-page DU. The benchmark is motivated by the observation that real documents such as financial reports, research reports, academic papers, guidelines, manuals, brochures, and slide decks often span tens or hundreds of pages and combine rich text, complex layout, and multiple visual modalities including tables, charts, figures, and photos. In this setting, a model must retrieve relevant pages, identify the right regions within those pages, integrate evidence across pages and modalities, and avoid producing unsupported answers (Ma et al., 2024).

Its design is explicitly distinguished from earlier LVLM benchmarks that are single-page or short-context. The motivating comparison includes DocVQA, ChartQA, InfographicVQA, and TAT-DQA on the single-page side, and MP-DocVQA, DUDE, SlideVQA, and FinanceBench on the multi-page side. The stated limitation of those settings is that they rarely require non-trivial cross-page reasoning, usually involve shorter and less information-dense documents, and only weakly probe unanswerable cases or hallucinations. MMLongBench-Doc is therefore positioned as a benchmark where answers may depend simultaneously on different evidence sources—text, image, chart, table, and layout structure—and on evidence located at different page indices within long PDFs (Ma et al., 2024).

A concise statistical summary is as follows.

Property	Value
Documents	130
Questions	1,062
Average pages	49.4
Average tokens	20,970.9
Cross-page questions	33.2%
Unanswerable questions	22.8%

This design implies that MMLongBench-Doc is not merely a larger DocVQA set. A plausible implication is that it operationalizes long-context DU as the conjunction of localization, multi-modal grounding, cross-page composition, and abstention under missing evidence, rather than as a pure extraction task.

2. Corpus construction and annotation

The document collection combines 76 documents from existing DU datasets and 54 newly collected documents from arXiv, ManualsLib, and general web search. Documents with fewer than 15 pages were discarded, documents restricted by license were removed, and some sources such as DUDE, SlideVQA, and FinanceBench were down-sampled for balanced domain distribution. The resulting seven document categories are Research Report, Financial Report, Academic Paper, Brochure, Guideline, Administration / Industrial File, and Tutorial / Workshop (Ma et al., 2024).

Question authoring and curation were performed by 10 expert-level annotators, described as PhD students or above and proficient in English. Annotators were trained, assigned 2–4 documents per batch, and typically spent 60–90 minutes per document, with the first 15–30 minutes used for skimming and the remainder for refinement and question creation. For every question, annotators provided the reference answer, evidence sources, evidence locations, and answer format. Final answer formats are String, Integer, Float, and List, with counts 239, 280, 158, and 143 respectively (Ma et al., 2024).

The benchmark contains 878 newly annotated questions and 184 derived from existing datasets. The details also report an intermediate review stage in which 425 original questions were audited, with 32.2% revised and 46.1% removed, leaving 211 retained or edited questions at that stage. New annotation was explicitly used to compensate for bias toward single-page, text-only questions by increasing cross-page, unanswerable, table-based, chart-based, and image-based items (Ma et al., 2024).

Quality control follows a three-stage semi-automatic pipeline. First, GPT-4o answers each question without seeing the document; if it can answer correctly, the item is treated as low document-relevant and removed, which eliminated 85 samples. Second, GPT-4o is rerun with the document, and discrepancies with the reference answer are sent back to annotators; 13.8% of samples were identified as problematic and revised. Third, annotators cross-check one another’s work, yielding Cohen’s kappa $= 0.42$ and about 17.5% inconsistent items, after which two lead authors act as meta-annotators and make final decisions (Ma et al., 2024).

Evidence is labeled by dominant source as text, layout, table, chart, or image. The corresponding counts are 296 pure-text questions, 119 layout questions, 212 table questions, 168 chart questions, and 282 image questions, with the explicit caveat that many questions require multiple sources. By evidence-page type, the benchmark contains 467 single-page questions, 353 cross-page questions, and 242 unanswerable questions (Ma et al., 2024).

3. Task formulation and evaluation protocol

The task is open-ended visual question answering over multi-page documents. For LVLMs, each PDF page is rendered to a PNG image at 120 DPI and page images are passed to the model; proprietary models such as GPT-4o, GPT-4V, and Gemini-1.5-Pro receive all page screenshots directly, Claude-3-Opus concatenates pages to fit its image limit, and many open-source LVLMs use a concatenation strategy in which all pages are merged into 1 or 5 images because they degrade severely beyond about 5 images. For text-only baselines, documents are converted with Tesseract OCR and fed as lossy-parsed OCR documents, truncating the text if it exceeds the model context window (Ma et al., 2024).

Evaluation is decomposed into three steps following MathVista. First, the model generates a free-form response. Second, a unified LLM-based extractor, specifically GPT-4o, converts that response into a short-form answer consistent with the expected answer type. Third, a rule-based scorer compares the extracted answer with the gold answer. String answers use either strict equality or ANLS with threshold $\tau = 0.5$, integers require exact match, floats are correct within a 1% relative tolerance, and lists are scored by sorting the predicted and reference lists, computing scalar scores element-wise, and taking the minimum element score as the list score (Ma et al., 2024).

The benchmark reports generalized accuracy and generalized F1, the latter balancing performance on answerable and unanswerable questions. Performance is also broken down by evidence source, evidence-page type, document type, and evidence position. A human check on 100 responses from GPT-4o and Gemini-1.5-Pro found that the automatic pipeline differed from human judgment on only 4–6 cases per 100, which supports the reliability of the extraction-plus-scoring protocol (Ma et al., 2024).

The benchmark’s task distribution is structurally important. Single-page questions measure localization within a single page, cross-page questions require multi-page retrieval and integration, and unanswerable questions are designed so that the document contains no sufficient evidence and the correct behavior is to say the question is unanswerable or unknown. This suggests that MMLongBench-Doc jointly evaluates retrieval, multimodal fusion, cross-page reasoning, and calibration.

4. Empirical results and diagnostic patterns

Experiments cover 14 LVLMs and 10 LLMs. The headline result is that long-context DU remains difficult: the best-performing LVLM, GPT-4o, reaches ACC 40.8 and F1 42.7%, while the second-best, GPT-4V, reaches F1 31.4%. Twelve of the 14 LVLMs—everything except GPT-4o and GPT-4V—perform worse than their LLM counterparts operating on lossy-parsed OCR documents, which shows that end-to-end multimodal processing over long PDFs is not automatically superior to a text-only pipeline (Ma et al., 2024).

Performance varies strongly by evidence source and page type. GPT-4o is relatively balanced across TXT 43.6, LAY 43.5, CHA 43.1, TAB 49.9, and IMG 41.6, whereas OCR-based LLMs retain competitive performance on text and tables but degrade on charts and images because OCR incompletely captures those modalities. Models perform consistently better on single-page than on cross-page questions. For GPT-4o, SIN ACC is 54.0, MUL ACC is 37.5, and UNA ACC is 19.8. Gemini-1.5-Pro adopts a more conservative refusal strategy and scores UNA ACC 69.4 in LVLM mode, while GPT-4o and Claude-3-Opus are more aggressive and therefore more hallucination-prone on unanswerable items (Ma et al., 2024).

The evidence position analysis shows a monotonic difficulty trend over page index: questions whose evidence lies on early pages are easier, and accuracy decreases steadily as evidence moves later in the document. Industrial and administrative files are relatively easier, plausibly because of standardized formats and text-heavy content, whereas academic papers and financial reports are much harder for most models (Ma et al., 2024).

An oracle evidence-page experiment isolates the localization bottleneck by giving models only pages that contain evidence. Over 820 answerable questions, Gemini-1.5-Pro and InternLM-XC2-4KHD improve by more than 20 percentage points overall, and single-page questions can improve by about 30 points; GPT-4o improves by about 10 points. Yet even under oracle conditions, Gemini-1.5-Pro and InternLM-XC2-4KHD remain around 40% and 30% accuracy respectively. This indicates that long-context failure is not reducible to page retrieval alone; perception, grounding, and reasoning remain major bottlenecks after localization is solved (Ma et al., 2024).

Manual error analysis of 72 GPT-4o failures identifies seven categories: perceptual error, irrelevant answer, incomplete evidence, hallucinated evidence, extractor error, reasoning error, and knowledge lacking. The presence of incomplete evidence and hallucinated evidence in the same analysis is especially informative: one failure mode is missing relevant pages or items, while another is fabricating support not present in the document (Ma et al., 2024).

5. Position within the long-context multimodal benchmark ecosystem

MMLongBench-Doc occupies a specific niche within a broader ecosystem of long-context benchmarks. MileBench evaluates multimodal long context across a wide range of downstream tasks and diagnostic settings, including temporal multi-image tasks, semantic multi-image tasks, and multimodal needle-in-a-haystack tests, but it is not specifically document-centric (Song et al., 2024). MMLongBench extends this broader perspective further by standardizing cross-modal input lengths from 8K to 128K tokens across Visual RAG, NIAH, many-shot ICL, summarization, and long-document VQA; in that broader benchmark, MMLongBench-Doc is one of the DocVQA components, and long-document VQA exhibits the highest average correlation with other long-context categories, making it the strongest single proxy for overall long-context vision-language capability (Wang et al., 15 May 2025).

Complementary benchmarks refine adjacent subproblems. MMDocBench focuses on fine-grained OCR-free visual document understanding at page scale, with 15 main tasks, 4,338 QA pairs, and 11,353 supporting regions, emphasizing region grounding rather than cross-page reasoning; it can therefore be understood as a fine-grained, single-page complement to MMLongBench-Doc’s long-document setting (Zhu et al., 2024). MMDocIR, in turn, recasts part of the MMLongBench-Doc problem as retrieval by introducing page-level and layout-level multimodal document retrieval over long documents; its role is to evaluate which pages or layout elements should be retrieved before a QA model reasons over them (Dong et al., 15 Jan 2025).

This partitioning is methodologically useful. MMLongBench-Doc stresses end-to-end long-PDF understanding, MMDocBench isolates fine-grained page-level perception and grounding, MMDocIR isolates retrieval, and MMLongBench studies how these skills interact with other long-context multimodal capabilities under a common length-control regime. A plausible implication is that progress on MMLongBench-Doc depends on improvements in at least three separable subsystems: long-range retrieval, visual text and layout perception, and multi-step reasoning over multi-page evidence.

6. Corrections, training use, and later developments

A later study on training long-context visual document models treats MMLongBenchDoc as the central benchmark for long-context visual document QA and introduces MMLBD-C, a corrected version intended to reduce erroneous and low-quality examples. In that study’s working version of the original benchmark, the authors report 1,091 examples, flag 342 through a recursive consistency pipeline, manually modify 251, and remove 16. Reported correction types include document mismatch, underspecified questions, typos, incorrect answers, and answer expansion for “Not answerable” cases (Veselka, 16 Feb 2026).

The same study shows that benchmark-specific training choices materially affect MMLongBenchDoc performance. Feeding page images with explicit textual page indices—literally the format “Page 1: <image> Page 2: <image> …”—provides a simple structural cue that boosts long-document performance. On MMLBD-C, adding page indices in both training and evaluation raises a Mistral-based model from 42.3 to 45.1, a gain of 2.8 points. The study also reports that training on context lengths that match evaluation context lengths outperforms training on longer contexts, and that a Qwen3 VL 32B model with supervised finetuning reaches 57.3 on MMLBD-C and 56.3 on original MMLongBenchDoc, essentially matching a much larger Qwen3 VL 235B A22B model that scores 56.7 on the original benchmark (Veselka, 16 Feb 2026).

These later results recast MMLongBenchDoc from a passive evaluation set into a training target and methodological driver. The benchmark becomes the operational criterion for continued pretraining, supervised finetuning, preference optimization, page-index formatting, and synthetic self-improvement pipelines. The same study further argues that visual long-context training transfers to long-context text performance, implying that MMLongBenchDoc probes competencies that are not purely document-specific but are part of a broader long-context reasoning regime (Veselka, 16 Feb 2026).

Taken together, MMLongBench-Doc has two roles in the literature. First, it is a benchmark demonstrating that long-context multi-page PDF understanding with multimodal evidence and unanswerable questions is still far from solved. Second, it has become an anchor benchmark for training and diagnosing long-context visual document models, especially once corrected variants such as MMLBD-C are introduced to improve measurement fidelity (Ma et al., 2024).