LED-Dataset: Benchmark for Structural Error Analysis

• LED-Dataset is a synthetic benchmark that diagnoses structural layout errors by injecting empirically grounded errors into document images.
• The dataset supports three hierarchical tasks, including document-level error detection and element-wise error type classification using standardized eight-class taxonomy.
• It employs a deterministic error-injection pipeline, redefining evaluation beyond spatial overlap to focus on semantic and structural diagnostic metrics.

LED-Dataset is a synthetic benchmark for diagnosing structural layout errors in Document Layout Analysis (DLA). It is introduced in the context of the LED benchmark, which was designed to evaluate structural robustness beyond overlap-oriented metrics such as IoU and mAP. The dataset is constructed by injecting empirically grounded structural errors into the TEST split of DocLayNet’s document images, then attaching document-level and element-level error labels for three hierarchical tasks: error existence detection, error type classification, and element-wise error type classification (Heo et al., 31 Jul 2025).

1. Motivation and problem formulation

Recent advances in DLA through LLMs and Large Multimodal Models have improved layout detection, but structural failures remain persistent. The benchmark motivating LED-Dataset emphasizes region merging, splitting, and missing content as critical failure modes that are not adequately captured by conventional evaluation metrics. IoU and mAP primarily measure spatial overlap and class-label correctness, so they cannot distinguish whether two logically separate regions were fused, whether one region was fragmented into several parts, or whether content was omitted altogether (Heo et al., 31 Jul 2025).

LED-Dataset addresses this gap by shifting evaluation from geometric agreement alone to semantic error diagnosis. In the benchmark formulation, correctly diagnosing a Merge error does not require an IoU threshold such as $0.5$; instead, the relevant question is whether the prediction structurally fused two ground-truth regions. This reorients DLA evaluation toward logical consistency of page structure rather than surface-level box alignment (Heo et al., 31 Jul 2025).

2. Dataset construction and corpus composition

LED-Dataset is built by injecting synthetic errors into the TEST split of DocLayNet’s document images in PNG format. The base corpus therefore inherits the layout diversity of DocLayNet, including scientific papers, magazines, forms, and related document genres, as well as element categories such as text blocks, figures, tables, and lists. The resulting dataset contains 4,996 document images and approximately 70,000 total layout elements after injection (Heo et al., 31 Jul 2025).

The error-injection pipeline has three stages. First, the empirical frequency of each error type is estimated by running a state-of-the-art DLA model, maskrcnn_dit_base, on DocLayNet and observing its failure modes. Second, document-level and element-level error types are sampled according to the empirical distribution $P(\text{error\_type})$. Third, each sampled instance is realized through a deterministic geometric or label transformation. No explicit closed-form LaTeX sampling model is given beyond the statement that sampling follows the empirical distribution determined by real-model outputs (Heo et al., 31 Jul 2025).

The empirical distribution used for injection is highly skewed. The reported proportions are Missing 63%, Hallucination 14%, Size Error 11%, Misclassification 8%, and approximately 1% each for Split, Merge, Overlap, and Duplicate. A histogram described in the benchmark shows per-document counts with a long tail for Merge, Split, Overlap, and Duplicate. This distribution makes LED-Dataset structurally diagnostic rather than class-balanced; a plausible implication is that performance on rare error classes is intended to reflect robustness under realistic failure frequencies rather than artificial uniformity (Heo et al., 31 Jul 2025).

3. Standardized error taxonomy

LED defines eight standardized structural error types. The formulation uses ground-truth boxes $B_{\text{gt}}$, predicted boxes $B_{\text{pred}}$, intersection-over-union $\mathrm{IoU}(\cdot,\cdot)$, and box area. Each type is specified by quantitative criteria and paired with an injection rule (Heo et al., 31 Jul 2025).

Error type	Criterion	Injection strategy
Missing	No $B_{\text{pred}}$ with $\mathrm{IoU}(B_{\text{gt}},B_{\text{pred}})\ge 0.1$	Randomly remove $\sim 10\%$ of GT boxes
Hallucination	No $B_{\text{gt}}$ with sufficient overlap	Insert new boxes in empty regions with IoU $\le 0.01$ to any GT
Size Error	$P(\text{error\_type})$0	Scale width/height by a random factor in $P(\text{error\_type})$1
Split	One GT region fragmented into multiple predictions	Horizontally cut one GT box into $P(\text{error\_type})$2 strips
Merge	Multiple nearby same-class GT boxes replaced by one prediction	Use the minimum enclosing rectangle
Overlap	Two predicted boxes satisfy $P(\text{error\_type})$3	Expand one predicted box to force overlap
Duplicate	Multiple predictions correspond to the same GT	Create 2–3 perturbed copies with IoU $P(\text{error\_type})$4
Misclassification	Localized box assigned the wrong class	Randomly reassign the category label

The dataset description provides additional operational detail for each transformation. Hallucinated boxes are sampled in empty regions with sizes drawn uniformly over a realistic range. Size errors are accepted only if the new box’s IoU with neighbors is at most $P(\text{error\_type})$5. Split preserves the original height while allocating widths randomly so that they sum to the original width and are evenly spaced. Merge applies to nearby same-class boxes whose centers lie within $P(\text{error\_type})$6 the average width. Duplicate perturbs the center within $P(\text{error\_type})$7 and scales size by $P(\text{error\_type})$8, while Misclassification changes the label to another valid class (Heo et al., 31 Jul 2025).

These definitions formalize error categories that standard DLA metrics conflate. In particular, Split and Merge encode relational structure among regions, not merely pointwise localization quality. That distinction is central to LED-Dataset’s role as a structural benchmark (Heo et al., 31 Jul 2025).

4. Annotation schema, storage format, and evaluation tasks

Each document is stored as a folder containing four artifacts: the original image in PNG, a ground-truth annotation in COCO-style JSON, a ground-truth visualization in PNG with overlaid boxes and IDs, and an error annotation JSON. The COCO-style annotation stores, for each element, {"bbox": [x,y,w,h], "category_id": c, "id": element_id}. The error annotation stores "has_error": true/false, "error_types": [list of eight binary flags], and "element_errors": list of {"element_id": X, "error_type": E, "bbox": [x,y,w,h]} (Heo et al., 31 Jul 2025).

The metadata operate at two granularities. Per document, LED-Dataset records overall error presence and a multi-label vector of error types. Per element, it records which specific error type, if any, applies, together with original versus injected bounding boxes and original versus injected labels. This dual granularity supports evaluation of both page-level structural reasoning and per-region diagnosis (Heo et al., 31 Jul 2025).

The benchmark defines three hierarchical tasks, all taking as input $P(\text{error\_type})$9. Task $B_{\text{gt}}$0 is document-level error detection, a binary classification problem with output one bit per document and metric

$B_{\text{gt}}$1

Task $B_{\text{gt}}$2 is document-level error type classification. The output is an 8-dimensional binary vector, and the metric is macro-averaged $B_{\text{gt}}$3 over the eight types:

$B_{\text{gt}}$4

Task $B_{\text{gt}}$5 is element-level error type classification. The output is a list of $B_{\text{gt}}$6 pairs, evaluated by macro-averaged $B_{\text{gt}}$7 over the eight types, with Missing included as a class. A predicted element-level diagnosis matches the ground truth only if both element_id and error_type agree exactly (Heo et al., 31 Jul 2025).

5. Benchmark protocol and empirical findings

The evaluation protocol tests eight LMMs under three prompting schemes. $B_{\text{gt}}$8 provides the image together with the JSON of the predicted layout. $B_{\text{gt}}$9 provides the image with overlaid bounding boxes only. $B_{\text{pred}}$0 provides both the image with boxes and the JSON. All models are queried through a unified API with fixed decoding parameters, temperature=1.0 and top-p=1.0. The tested models are GPT-4o, GPT-4o-mini, Google Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.5 Flash Lite, DeepSeek V3, LLaMA 4 Maverick, and LLaMA 4 Scout (Heo et al., 31 Jul 2025).

For the reported overall baseline results under $B_{\text{pred}}$1, Gemini 2.5 Pro is best on all three tasks: $B_{\text{pred}}$2 Accuracy on $B_{\text{pred}}$3, $B_{\text{pred}}$4 $B_{\text{pred}}$5 on $B_{\text{pred}}$6, and $B_{\text{pred}}$7 $B_{\text{pred}}$8 on $B_{\text{pred}}$9. GPT-4o reports $\mathrm{IoU}(\cdot,\cdot)$0, $\mathrm{IoU}(\cdot,\cdot)$1, and $\mathrm{IoU}(\cdot,\cdot)$2 on the same tasks, while GPT-4o-mini reports $\mathrm{IoU}(\cdot,\cdot)$3, $\mathrm{IoU}(\cdot,\cdot)$4, and $\mathrm{IoU}(\cdot,\cdot)$5. The benchmark summary states that Gemini models excel at fine-grained error classification, whereas GPT variants are competitive chiefly on coarse error detection in $\mathrm{IoU}(\cdot,\cdot)$6 (Heo et al., 31 Jul 2025).

The qualitative conclusions are as important as the raw scores. Even the best LMMs struggle with Size, Hallucination, and the low-frequency classes Split, Merge, Overlap, and Duplicate. The benchmark also reports that providing both structured JSON and visual boxes through $\mathrm{IoU}(\cdot,\cdot)$7 generally improves performance, especially for Gemini. These findings indicate that LED-Dataset is not simply measuring OCR-adjacent perception; it is probing multimodal structural reasoning under controlled perturbations (Heo et al., 31 Jul 2025).

6. Limitations and research directions

LED-Dataset is explicitly synthetic. Its errors are injected according to empirical distributions and deterministic perturbation rules, but real model failures may exhibit more complex and correlated patterns. A second limitation is domain concentration: the injection procedure is based on DocLayNet styles, and one benchmark description characterizes this as a single-domain setup tied to scientific-article-like layouts. A third limitation is release structure: the dataset is presented as a diagnostic testbed rather than a conventional supervised-learning corpus, and no separate training, validation, and test splits are released (Heo et al., 18 Mar 2026).

These limitations shape the directions proposed for follow-up work. The benchmark description recommends extending injection to other document domains such as forms, receipts, and presentations; incorporating real model outputs to refine error distributions and co-occurrence modeling; developing end-to-end learning approaches that directly predict element-level error types for self-supervised correction; and combining LED with downstream tasks such as information extraction and question answering to measure end-task impact of structural errors (Heo et al., 18 Mar 2026).

Within those constraints, LED-Dataset occupies a specific methodological niche. It is not designed to replace layout detection datasets or standard localization metrics. Rather, it functions as an error-analysis layer that makes structural failure modes explicit and measurable.

7. Nomenclature and disambiguation

The name “LED” is highly overloaded in the literature. On arXiv it also refers to a vehicle tail light signal dataset and benchmark (Chai et al., 2024), a paired real-world event-camera denoising dataset (Duan et al., 2024), the Chinese legal event detection dataset LEVEN (Yao et al., 2022), a life event extraction dataset from dialogs (Chen et al., 2023), and “Light Enhanced Depth” for nighttime depth estimation (Moreau et al., 2024). The document-analysis LED-Dataset discussed here is specifically the dataset associated with layout error detection in DLA (Heo et al., 31 Jul 2025).

That terminological ambiguity has practical consequences for indexing and citation. In document understanding, “LED-Dataset” denotes the structurally annotated benchmark built on the DocLayNet test split and centered on eight standardized layout error types. In other subfields, the same acronym denotes unrelated resources, tasks, or models. Careful citation by title or arXiv identifier is therefore necessary to avoid conflation.