Doc-750K: Multimodal Document Dataset
- Doc-750K is a large-scale, document-level multimodal dataset that provides comprehensive contexts by combining full texts, page images, and in-document Q&A pairs.
- It addresses challenges in retrieval-augmented generation by offering native, end-to-end long-context processing to reduce fragmented contexts and cascading errors.
- The dataset features 758K questions, 3.1M images, and 5.2B text tokens from academic sources, supporting advanced long-context multimodal reasoning.
Searching arXiv for the specified paper to ground the article. Doc-750K is a large-scale, document-level multimodal dataset introduced alongside Docopilot for multi-page document understanding in multimodal LLMs (MLLMs). It was designed to address three problems associated with retrieval-augmented generation (RAG) on document tasks: fragmented retrieval contexts, multi-stage error accumulation, and extra time costs. Rather than retrieving fragments, Doc-750K provides native, end-to-end long context over complete documents, combining text and page images with real, in-document question–answer pairs and extensive cross-page dependencies. The dataset is used as the core supervised fine-tuning corpus for retrieval-free document understanding in Docopilot (Duan et al., 19 Jul 2025).
1. Motivation and problem setting
Doc-750K was motivated by the observation that existing MLLMs remain inadequate on complex, multi-page document comprehension. In the formulation accompanying the dataset, RAG offers only a partial solution because retrieval can break the natural structure of documents and their visual layout, retrieval mistakes can cascade through multi-stage pipelines, and retrieval introduces both latency and engineering complexity (Duan et al., 19 Jul 2025).
The central design choice is therefore to make entire documents natively available to the model. This includes complete text, page images, and question–answer pairs aligned to source content, so that the model can resolve document-level dependencies without external retrieval. The intended effect is not merely improved answer accuracy, but also more coherent responses and better response time under long-context multimodal processing.
This suggests a shift in emphasis from passage selection to direct long-context reasoning. In practical terms, the dataset frames document understanding as an end-to-end multimodal sequence modeling problem rather than a retrieve-then-read pipeline.
2. Scale, composition, and representational formats
Doc-750K comprises 251K conversations totaling 758K questions, 3.1M document page images, and roughly 5.2B text tokens. It is sourced predominantly from scientific and academic documents, specifically Sci-Hub, arXiv, and OpenReview, with OpenReview and arXiv together contributing 75.4% of all samples. Academic papers constitute roughly 32.6% of all samples, and multimodal data accounts for about 88.8% of the dataset. A notable property is that 31.6% of question–answer pairs are real question–answer pairs directly derived from the document’s own content rather than synthetic prompts (Duan et al., 19 Jul 2025).
| Aspect | Value | Notes |
|---|---|---|
| Conversations | 251K | Core conversational units |
| Questions | 758K | Total QA count |
| Document page images | 3.1M | Multi-page page-image corpus |
| Text tokens | roughly 5.2B | Full-document textual scale |
| Real QA pairs | 31.6% | Directly derived from source documents |
| Multimodal share | about 88.8% | Text and image content |
| OpenReview + arXiv share | 75.4% | Share of all samples |
| Academic papers share | roughly 32.6% | Portion of all samples |
The dataset uses two principal representational formats. In the interleaved text–image format, a PDF parser, MinerU, extracts sequences such as "<text>\n<image>\n<text>\n<image>", which avoids OCR errors and supports language-side reasoning over clean text while preserving explicit references to figures and tables. In the multi-image format, each -page document is rendered as page images in natural order, preserving layout and cross-page visual structure and supporting OCR-free visual reading and layout understanding.
For training and evaluation, the same document can be used in either format. At inference time, adjacent pages are frequently concatenated vertically to reduce patch count while keeping page order intact. This dual-format design balances text fidelity against layout fidelity: the interleaved format sacrifices some layout, whereas the multi-image format preserves the page-level visual organization.
3. Task design and cross-page dependency structure
Doc-750K includes document-level VQA together with eight proxy tasks built from natural document structure. The tasks named in the dataset description are abstract writing, paper titling, caption writing, experiment writing, translation, review writing, and reply writing; the corpus also contains multi-turn dialogues, with 87K multi-turn questions and 164K single-turn questions (Duan et al., 19 Jul 2025).
These tasks are intended to require cross-page scanning, layout awareness, and integration of figures, tables, and text. Scientific articles in the dataset often require answers that depend jointly on title, abstract, main text, figures, and tables. In many cases, the relevant figure or table caption appears on a different page from the narrative text that references it. The paper’s qualitative examples emphasize four kinds of document-level behavior: multi-page retrieval, backward queries, consistency checks across follow-up questions, and counting across pages.
The task construction embeds cross-page dependency directly into supervision. “Experiments Writing” and “Caption Writing” require locating and integrating relevant figures or tables with surrounding text. “Review” and “Reply” exploit the OpenReview review–reply structure. Multi-turn QA encodes sequential reasoning and cross-page references inside dialogue. Answers are typically free-form text that quotes relevant spans and then synthesizes a response, whereas evaluation on single-page VQA benchmarks uses “single word or phrase” answers.
A plausible implication is that Doc-750K supervises not only answer extraction but also discourse-level grounding over long multimodal contexts. Because the tasks originate from native scholarly document structure, the resulting supervision is document-grounded rather than instruction-only.
4. Annotation pipeline and quality control
The annotation pipeline differs by source. For OpenReview, human-written reviews and authors’ replies are directly extracted and turned into conversational pairs. For Sci-Hub and arXiv, structured tasks are derived from natural sections and metadata of well-formed papers, including “Abstract Writing,” “Paper Titling,” “Caption Writing,” “Experiments Writing,” and “Translation.” For other documents, GPT-4o is used to generate a small fraction of question–answer pairs with a conservative prompt that constrains questions to be answerable only by the provided paper and requires the answer to quote the original text beginning with “According to the original text …,” including references such as “Figure 1” or “Table 2” (Duan et al., 19 Jul 2025).
Only 4.8% of samples are LLM-generated, and those samples are labeled in metadata as model-generated. Quality control is described explicitly: a random sample of 500 generated question–answer pairs was manually reviewed, and 498/500, greater than 99%, were judged relevant and correct. More broadly, quality is maintained by relying primarily on human-origin QA from OpenReview and structured, rule-based construction for arXiv and Sci-Hub tasks.
The dataset is explicitly designed to minimize instruction-only data and maximize document-grounded question–answer pairs. This suggests that annotation quality is treated not simply as syntactic fluency, but as source alignment and recoverability from the original document. The use of deterministic section-based tasks for papers further constrains the supervision to well-formed document semantics.
5. Data organization, preprocessing, and use in long-context training
Doc-750K is not released with fixed train/validation/test partitions; instead, it is used as the core supervised fine-tuning corpus, while evaluation is conducted on external benchmarks including DocVQA, ChartQA, InfoVQA, MP-DocVQA, MMLongBench-Doc, DocGenome, and MM-NIAH (Duan et al., 19 Jul 2025).
Training uses dynamic high-resolution image tiling with tile resolution 448 and a maximum of 24 tiles per input, together with a maximum sequence length of 32K tokens. A multimodal data packing algorithm concatenates multiple conversation samples into long sequences to reduce padding waste under an image threshold of 48 and a token threshold of 32K. During packing, tokens can attend only within their original sample, preserving sample integrity while increasing throughput. In multi-page VQA inference, adjacent pages are vertically concatenated, with a maximum of 18 concatenated images.
Handling of tables and charts depends on format. In the interleaved format, figures and tables are represented as <image> tokens interspersed in relevant text, and “Caption Writing” pairs the visual asset with surrounding text to teach the linkage. In the multi-image format, charts and tables remain embedded in page images and are processed OCR-free by the visual encoder.
Docopilot uses Doc-750K in a ViT–MLP–LLM architecture consisting of a pre-trained Vision Transformer, InternViT-300M, a two-layer MLP projector, and a pre-trained LLM: InternLM2-1.8B for Docopilot-2B and InternLM2.5-7B for Docopilot-8B. Cross-modal fusion is implemented by feeding visual tokens as soft prompts into the LLM. Retrieval-free long-context training relies on multimodal data packing, Ring Attention, and Liger Kernel. Fine-tuning is conducted for one epoch with batch size 128, AdamW, learning rate , cosine schedule, and weight decay 0.01 for 2B and 0.05 for 8B. No explicit loss equations are presented.
6. Comparative position and empirical significance
Doc-750K is positioned against both single-page and multi-page document benchmarks. Single-page VQA datasets such as DocVQA, ChartQA, and InfoVQA primarily focus on individual pages and restricted modalities, whereas Doc-750K is designed for entire documents with explicit cross-page dependencies and includes structured writing tasks and multi-turn QA. Compared with multi-page document benchmarks such as MP-DocVQA and DUDE, which are reported as having 46K and 41K QA pairs respectively, Doc-750K contributes 758K QA pairs and 3.1M images, and extends beyond VQA to long-form grounded writing and multi-turn interaction (Duan et al., 19 Jul 2025).
The comparison with Docmatix is quantitative: Doc-750K is reported with 758K questions, 3,103,494 images, and 5,200,000,000 tokens, whereas Docmatix is reported with 2,444,750 images, 9,500,000 QA pairs, and 390,000,000 tokens. Doc-750K therefore has fewer QA pairs than Docmatix but far more tokens per sample due to full document contexts. The stated interpretation is that this better exercises long-context modeling.
The evaluation evidence in the associated paper is used to argue that Doc-750K materially improves document understanding. Docopilot-8B achieves MP-DocVQA ANLS of 81.3; MMLongBench-Doc Accuracy 28.8 and F1 23.0; DocGenome classification accuracy 93.8, Title edit distance 2.0, Abstract edit distance 19.7, Single-Page accuracy 53.9, and Multi-Page accuracy 51.9; MM-NIAH Accuracy Short 71.2, Medium 57.4, Long 55.3, and Overall 61.8. On single-page VQA benchmarks, Docopilot-8B reaches 92.0 on DocVQA, 83.3 on ChartQA, and 73.3 on InfoVQA. The paper further reports that in DocGenome multi-page QA, Docopilot improves by 12.6% over InternVL2-8B.
Ablation results isolate the contribution of Doc-750K on MMLongBench-Doc. Starting from a baseline InternVL2-2B with Accuracy 10.5 and F1 10.8, supervised fine-tuning without Doc-750K yields Accuracy 18.4 and F1 9.4; adding Sci-Hub data gives 18.5 and 15.2; adding arXiv data gives 20.5 and 15.5; and adding OpenReview data, i.e., full Doc-750K, gives 21.8 and 16.0. These staged gains are presented as evidence that the dataset’s multi-source, document-grounded coverage is essential for long-context comprehension.
The latency comparison is part of the same empirical argument. Docopilot-2B has an average token output latency of 35.9 ms on MMLongBench-Doc, equal to the non-RAG baseline, while InternVL2-2B + RAG is 82.9 ms, reported as +130% slower. Docopilot-8B runs at 81.0 ms, while InternVL2-8B + RAG is 113.4 ms. The significance attributed to Doc-750K is therefore not limited to accuracy; it is bound to a retrieval-free operating regime with lower latency than RAG-based alternatives.
7. Limitations, release status, and prospective extensions
A key limitation is domain coverage. Doc-750K is currently dominated by academic documents, which yields strong scientific comprehension but leaves gaps in enterprise document types such as forms, invoices, slides, and manuals (Duan et al., 19 Jul 2025). The associated authors state that they plan to extend coverage to broader domains and generalize proxy tasks accordingly.
A second limitation is the OCR-versus-layout trade-off between the two data formats. The interleaved text–image representation avoids OCR errors but loses some layout fidelity. The multi-image representation preserves layout but depends on OCR-free visual reading and may introduce OCR-like mistakes in complex visuals. Long-range limits also remain: although the training regime supports sequences up to 32K tokens and reports strong gains, dependencies beyond 64K tokens remain challenging. Language coverage is primarily English, with translation tasks into Chinese but limited broader multilingual training.
Generalization to unseen document types is therefore not guaranteed. The reported results are strong on scientific documents, but models trained on Doc-750K may require additional domain adaptation for forms-heavy or non-scientific corpora. A plausible implication is that Doc-750K functions most directly as a high-quality substrate for scientific and scholarly document intelligence, rather than as a fully general document benchmark.
Doc-750K, together with code and models, is publicly released at the OpenGVLab Docopilot repository. The paper does not specify license terms, usage restrictions, or privacy filters, and directs readers to consult the repository for current license and compliance details. The paper also does not provide hardware or compute cost details. Future work identified in the same source includes expanding beyond academic domains, strengthening multilingual support, reducing memory and latency for 64K-plus contexts, and adding tasks that stress real-world table and chart operations such as cross-document aggregation and compliance checks.