DocRec1K Benchmark

• DocRec1K is a 1,000-image evaluation dataset designed to benchmark holistic document reconstruction by restoring both physical layout and logical content in strict reading order.
• It integrates four key document elements—Paragraph, Table, Formula, and Figure—using a unified serialized token sequence for comprehensive evaluation.
• The dataset is created using a three-phase annotation pipeline and evaluated with metrics like DSM and NED, highlighting its practical impact on end-to-end reconstruction models.

Searching arXiv for the DREAM paper and closely related dataset context. DocRec1K is a 1,000-image evaluation dataset purpose-built to assess the comprehensive “document reconstruction” task introduced by DREAM, “Document Reconstruction via End-to-end Autoregressive Model” (Li et al., 8 Jul 2025). In that formulation, document reconstruction restores both physical layout and logical content of heterogeneous document elements in strict reading order, enabling end-to-end conversion from document images to a unified, structured sequence representation. DocRec1K was introduced because extant public repositories for document reconstruction do not sufficiently address this task; it therefore serves as a benchmark for holistic reconstruction across text, table, formula, and figure elements while supporting the Document Similarity Metric (DSM), a joint measure of structural and textual fidelity.

1. Task definition and scope

DocRec1K operationalizes a standardized definition of document reconstruction in which the target output is not an isolated subtask prediction, but a full-page sequence that jointly encodes element category, geometry, and content. The task is defined over heterogeneous document elements and requires strict reading order across the page. Physical structure and logical content are evaluated concurrently rather than separately, which distinguishes the benchmark from datasets centered only on layout detection, OCR, table structure recognition, or formula transcription (Li et al., 8 Jul 2025).

This scope addresses two limitations identified in prior document analysis workflows. First, multi-stage systems assemble outputs from distinct document understanding models by heuristic principles, which introduces error propagation. Second, end-to-end generative approaches that recover the logical sequence of plain text, tables, and mathematical expressions do not preserve layout information adequately for reconstruction. DocRec1K is therefore aligned with a task definition in which reading order, element localization, and element-specific transcription are inseparable components of a single reconstruction target.

The benchmark focuses on four element categories: Paragraph, Table, Formula, and Figure. A plausible implication is that DocRec1K is intended less as a generic OCR corpus than as a page-level reconstruction benchmark whose unit of evaluation is the serialized document rather than any individual region.

2. Provenance, corpus composition, and coverage

The dataset comprises 1,000 document images along with their annotations. It is derived from DocLayNet, described as “a large human-annotated dataset for document-layout segmentation” comprising “over 80K manually annotated pages from a variety of data sources, representing a broad range of layout variations” (Li et al., 8 Jul 2025). DocRec1K inherits that diversity of page layouts and sources, while selecting pages that exercise end-to-end reconstruction across Paragraph, Table, Formula, and Figure elements.

Several coverage properties are explicitly delimited. Train/validation/test splits are not reported; DocRec1K is presented as an evaluation benchmark of 1,000 images. Languages are not explicitly specified for DocRec1K. Image resolutions and file formats are also not explicitly specified as dataset properties. The 1024×1024 resizing and padding described in DREAM are model preprocessing choices rather than intrinsic characteristics of the corpus.

DocRec1K fills a gap within document analysis and understanding because existing datasets usually evaluate isolated subtasks. In contrast, DocRec1K targets the end-to-end reconstruction outcome across element types and modalities and evaluates both physical and logical fidelity concurrently. It is therefore positioned as complementary to layout-only, table-only, or formula-only resources rather than as a replacement for them.

3. Construction workflow and annotation process

DocRec1K is constructed from DocLayNet pages through a three-phase pipeline. First, layout elements are ordered using an optimized XY-cut algorithm with a tolerance range to mitigate sensitivity to perturbations in the $y$-coordinate, yielding strict reading order. When overlapping boxes arise from labeling inaccuracies, auxiliary rules group top-to-bottom ranges and then sort left-to-right. This stage defines the page-level serialization order (Li et al., 8 Jul 2025).

Second, unformatted text is associated with layout elements via Intersection-over-Union (IoU). Lines that initially match multiple boxes are consolidated. Third, contents for tables and formulas are transformed using specialized recognizers: SLANet from PaddleOCR detects coordinates of cell boxes and row/column relationships for tables, and CAN transcribes formula images into LaTeX sequences.

The resulting pseudo-labeled contents—specifically the order of elements, table structures, and formula information—are subjected to human verification and correction “to ensure the quality of annotations.” The annotation scope includes Paragraphs, Tables, Formulas, and Figures. Geometric annotations consist of bounding boxes for all elements; for sub-elements, DocRec1K represents per-line coordinates inside Paragraphs and per-cell coordinates inside Tables. The relationship structure is limited to sequence order: the page is serialized in strict layout reading order, but the schema does not define figure–caption links or other explicit cross-element relations beyond that ordering.

4. Unified representation and label semantics

The canonical representation of DocRec1K is a serialized token sequence rather than a separately specified JSON or XML schema. Each element is represented by

$y = \{c, b, t, \langle Sep \rangle\},$

where $c$ is a category token, $b$ is a bounding box encoded by four tokens, $t$ is element-specific transcription content, and $\langle Sep \rangle$ separates consecutive elements. The full page sequence is

$Y = \{y^k\}_{k=1..K},$

where $K$ is the number of elements in strict reading order (Li et al., 8 Jul 2025).

The category token $c$ takes one of four values: $\langle Paragraph \rangle$, $y = \{c, b, t, \langle Sep \rangle\},$0, $y = \{c, b, t, \langle Sep \rangle\},$1, or $y = \{c, b, t, \langle Sep \rangle\},$2. The bounding box token sequence is $y = \{c, b, t, \langle Sep \rangle\},$3. The transcription field $y = \{c, b, t, \langle Sep \rangle\},$4 depends on the element type:

• Paragraph: multiple lines are treated as sub-elements; each line is represented by its content coordinates plus characters, and lines are separated by the token $y = \{c, b, t, \langle Sep \rangle\},$5.
• Table: content is represented in HTML with <tr>, </tr>, <td>, </td>, and attributes rowspan and colspan; per-cell coordinates are embedded into the HTML codes.
• Formula: content is a LaTeX sequence.
• Figure: transcription $y = \{c, b, t, \langle Sep \rangle\},$6 is empty.

This representation couples physical and logical information within a single sequence. A plausible implication is that the serialization format is designed to support both generative decoding and holistic evaluation without requiring separate post hoc alignment of layout and text predictions.

5. Evaluation protocol and Document Similarity Metric

DocRec1K is evaluated primarily with the Document Similarity Metric (DSM), which jointly measures structure and content using edit-distance-like dynamic programming over element sequences (Li et al., 8 Jul 2025). DSM is defined through a location cost, a transcription cost, an element cost, a sequence dynamic program, and a page-level aggregation:

$y = \{c, b, t, \langle Sep \rangle\},$7

$y = \{c, b, t, \langle Sep \rangle\},$8

$y = \{c, b, t, \langle Sep \rangle\},$9

$c$0

$c$1

Here, category mismatch contributes $c$2, category match contributes $c$3, coordinates contribute $c$4, and transcription discrepancy is normalized edit distance. DSM lies in $c$5, and higher is better. At both the element level and the aggregate level, structure and content are balanced with equal $c$6 weighting.

An auxiliary metric, Normalized Edit Distance (NED), is also reported. For NED, reconstruction sequences are converted to markdown-formatted text by removing category and bounding-box information, discarding content coordinates, and replacing $c$7 with two \n tokens. NED provides a text-centric comparison with existing end-to-end methods, whereas DSM remains the principal metric for DocRec1K because it evaluates both physical and logical fidelity together. For reporting on DocRec1K, DSM is applied to raw page sequences in reading order and averaged over the 1,000 images.

6. Reported results and relation to adjacent benchmarks

On the document reconstruction task, DREAM is reported to achieve the strongest results on DocRec1K, with DSM $c$8 and NED $c$9, exceeding the adapted Nougat$b$0 baseline by $b$1 DSM points and $b$2 NED points (Li et al., 8 Jul 2025).

System	DSM	NED
PaddleOCR	65.3	72.8
Pix2Struct$b$3	81.7	86.4
Nougat$b$4	85.2	88.6
DREAM	91.4	92.7

Ablation studies on DocRec1K attribute notable gains to both the feature aggregators (queries) and the parallel decoder. DREAM$b$5 without $b$6 obtains DSM $b$7 and NED $b$8; DREAM$b$9 without $t$0 obtains DSM $t$1 and NED $t$2; full DREAM reaches DSM $t$3 and NED $t$4. The paper states that parallel decoding mitigates repetitive long sequences seen in non-parallel autoregression. In computational complexity, the adapted Nougat$t$5 has $t$6 parameters and $t$7 s/image, whereas DREAM has $t$8 parameters and $t$9 s/image.

DocRec1K also serves as the context for interpreting DREAM’s compatibility with established subtasks on external datasets. Reported highlights include layout analysis on DocBank, PubLayNet, and DocLayNet; text recognition on FUNSD, SROIE, and CORD; table structure recognition on ICDAR-2013, SciTSR, and TableBank; formula recognition on IM2LATEX-100K and CROHME-2014; and reading order detection on ReadingBank. These experiments show that the reconstruction model is competitive and compatible with various tasks, but DocRec1K itself remains a distinct benchmark because it evaluates the unified page reconstruction rather than any single subtask in isolation.

Relative to related datasets, PubLayNet, DocLayNet, and DocBank focus primarily on bounding boxes and categories, i.e., physical structure. ICDAR-2013, SciTSR, and TableBank focus on table detection or structure. IM2LATEX-100K and CROHME-2014 focus on formula transcription. DocRec1K differs by explicitly coupling physical layout with logical content and reading-order serialization in a single unified sequence, evaluated holistically by DSM.

7. Limitations, release status, and open issues

DocRec1K has several explicit limitations. Its scale is 1,000 images, which is relatively small compared with large-scale datasets. Some annotations depend on tool-assisted pseudo labels—specifically ordering, table structures, formula transcription, and IoU-based associations—before human validation and correction, which implies potential residual noise and a continuing need for careful quality control (Li et al., 8 Jul 2025).

Coverage is also bounded. Language coverage is not detailed. Specific document genres are not enumerated. The schema focuses on four categories and reading order, and it does not specify additional element relationships such as explicit figure–caption links. The benchmark is document-centric; the paper notes that DREAM struggles with natural scenes, suggesting that extension to more diverse visual domains would improve robustness.

Release details remain limited. The paper introduces DocRec1K and uses it for evaluation, but does not provide a download link or repository URL. Licensing and usage terms are not specified in the text. The main text and appendix describe the construction and annotation processes, and the paper notes that a more comprehensive elucidation is provided in the supplementary material. For model input, images are rescaled and padded to $\langle Sep \rangle$0, but this is a model setting rather than a dataset property. Formal release details and fuller documentation would facilitate broader community adoption.

In aggregate, DocRec1K is significant because it redefines evaluation around full-page reconstruction rather than isolated subtasks. It provides a page-level sequence format that jointly encodes categories, bounding boxes, and element-specific content in strict reading order, and it introduces an evaluation regime in which structure, content, and order are scored together. This suggests a shift in document understanding benchmarks from modular recognition pipelines toward unified reconstruction objectives.