---
title: 'LED-Dataset: Benchmark for Structural Error Analysis'
url: https://www.emergentmind.com/topics/led-dataset
type: topic
---

# LED-Dataset: Benchmark for Structural Error Analysis

LED-Dataset is a synthetic benchmark for diagnosing structural layout errors in Document Layout Analysis (DLA). It is introduced in the context of the LED benchmark, which was designed to evaluate structural robustness beyond overlap-oriented metrics such as IoU and mAP. The dataset is constructed by injecting empirically grounded structural errors into the TEST split of DocLayNet’s document images, then attaching document-level and element-level error labels for three hierarchical tasks: error existence detection, error type classification, and element-wise error type classification [2507.23295].

## 1. Motivation and problem formulation

Recent advances in DLA through Large Language Models and Large Multimodal Models have improved layout detection, but structural failures remain persistent. The benchmark motivating LED-Dataset emphasizes region merging, splitting, and missing content as critical failure modes that are not adequately captured by conventional evaluation metrics. IoU and mAP primarily measure spatial overlap and class-label correctness, so they cannot distinguish whether two logically separate regions were fused, whether one region was fragmented into several parts, or whether content was omitted altogether [2507.23295].

LED-Dataset addresses this gap by shifting evaluation from geometric agreement alone to semantic error diagnosis. In the benchmark formulation, correctly diagnosing a Merge error does not require an IoU threshold such as $0.5$; instead, the relevant question is whether the prediction structurally fused two ground-truth regions. This reorients DLA evaluation toward logical consistency of page structure rather than surface-level box alignment [2507.23295].

## 2. Dataset construction and corpus composition

LED-Dataset is built by injecting synthetic errors into the TEST split of DocLayNet’s document images in PNG format. The base corpus therefore inherits the layout diversity of DocLayNet, including scientific papers, magazines, forms, and related document genres, as well as element categories such as text blocks, figures, tables, and lists. The resulting dataset contains 4,996 document images and approximately 70,000 total layout elements after injection [2507.23295].

The error-injection pipeline has three stages. First, the empirical frequency of each error type is estimated by running a state-of-the-art DLA model, `maskrcnn_dit_base`, on DocLayNet and observing its failure modes. Second, document-level and element-level error types are sampled according to the empirical distribution $P(\text{error\_type})$. Third, each sampled instance is realized through a deterministic geometric or label transformation. No explicit closed-form LaTeX sampling model is given beyond the statement that sampling follows the empirical distribution determined by real-model outputs [2507.23295].

The empirical distribution used for injection is highly skewed. The reported proportions are Missing 63%, Hallucination 14%, Size Error 11%, Misclassification 8%, and approximately 1% each for Split, Merge, Overlap, and Duplicate. A histogram described in the benchmark shows per-document counts with a long tail for Merge, Split, Overlap, and Duplicate. This distribution makes LED-Dataset structurally diagnostic rather than class-balanced; a plausible implication is that performance on rare error classes is intended to reflect robustness under realistic failure frequencies rather than artificial uniformity [2507.23295].

## 3. Standardized error taxonomy

LED defines eight standardized structural error types. The formulation uses ground-truth boxes $B_{\text{gt}}$, predicted boxes $B_{\text{pred}}$, intersection-over-union $\mathrm{IoU}(\cdot,\cdot)$, and box area. Each type is specified by quantitative criteria and paired with an injection rule [2507.23295].

| Error type | Criterion | Injection strategy |
|---|---|---|
| Missing | No $B_{\text{pred}}$ with $\mathrm{IoU}(B_{\text{gt}},B_{\text{pred}})\ge 0.1$ | Randomly remove $\sim 10\%$ of GT boxes |
| Hallucination | No $B_{\text{gt}}$ with sufficient overlap | Insert new boxes in empty regions with IoU $\le 0.01$ to any GT |
| Size Error | $\mathrm{area}(B_{\text{pred}})/\mathrm{area}(B_{\text{gt}})\notin[0.6,1.4]$ | Scale width/height by a random factor in $\{0.7\ldots1.3\}$ |
| Split | One GT region fragmented into multiple predictions | Horizontally cut one GT box into $N\ge2$ strips |
| Merge | Multiple nearby same-class GT boxes replaced by one prediction | Use the minimum enclosing rectangle |
| Overlap | Two predicted boxes satisfy $\mathrm{IoU}\ge 0.1$ | Expand one predicted box to force overlap |
| Duplicate | Multiple predictions correspond to the same GT | Create 2–3 perturbed copies with IoU $\ge 0.9$ |
| Misclassification | Localized box assigned the wrong class | Randomly reassign the category label |

The dataset description provides additional operational detail for each transformation. Hallucinated boxes are sampled in empty regions with sizes drawn uniformly over a realistic range. Size errors are accepted only if the new box’s IoU with neighbors is at most $0.01$. Split preserves the original height while allocating widths randomly so that they sum to the original width and are evenly spaced. Merge applies to nearby same-class boxes whose centers lie within $1.5\times$ the average width. Duplicate perturbs the center within $\pm 10\%$ and scales size by $\pm 10\%$, while Misclassification changes the label to another valid class [2507.23295].

These definitions formalize error categories that standard DLA metrics conflate. In particular, Split and Merge encode relational structure among regions, not merely pointwise localization quality. That distinction is central to LED-Dataset’s role as a structural benchmark [2507.23295].

## 4. Annotation schema, storage format, and evaluation tasks

Each document is stored as a folder containing four artifacts: the original image in PNG, a ground-truth annotation in COCO-style JSON, a ground-truth visualization in PNG with overlaid boxes and IDs, and an error annotation JSON. The COCO-style annotation stores, for each element, `{"bbox": [x,y,w,h], "category_id": c, "id": element_id}`. The error annotation stores `"has_error": true/false`, `"error_types": [list of eight binary flags]`, and `"element_errors": list of {"element_id": X, "error_type": E, "bbox": [x,y,w,h]}` [2507.23295].

The metadata operate at two granularities. Per document, LED-Dataset records overall error presence and a multi-label vector of error types. Per element, it records which specific error type, if any, applies, together with original versus injected bounding boxes and original versus injected labels. This dual granularity supports evaluation of both page-level structural reasoning and per-region diagnosis [2507.23295].

The benchmark defines three hierarchical tasks, all taking as input $(\text{Image} + \text{ModelPrediction})$. Task $T_1$ is document-level error detection, a binary classification problem with output one bit per document and metric
$$
\mathrm{Accuracy}=\frac{\#\text{documents correctly classified}}{\text{Total documents}}.
$$

Task $T_2$ is document-level error type classification. The output is an 8-dimensional binary vector, and the metric is macro-averaged $F_1$ over the eight types:
$$
F_{1,i}=\frac{2\cdot TP_i}{2\cdot TP_i+FP_i+FN_i}, \qquad
F_1=\frac{1}{8}\sum_{i=1}^{8}F_{1,i}.
$$

Task $T_3$ is element-level error type classification. The output is a list of $(\text{element\_id},\text{predicted\_error\_type})$ pairs, evaluated by macro-averaged $F_1$ over the eight types, with Missing included as a class. A predicted element-level diagnosis matches the ground truth only if both `element_id` and `error_type` agree exactly [2507.23295].

## 5. Benchmark protocol and empirical findings

The evaluation protocol tests eight LMMs under three prompting schemes. $P_1$ provides the image together with the JSON of the predicted layout. $P_2$ provides the image with overlaid bounding boxes only. $P_3$ provides both the image with boxes and the JSON. All models are queried through a unified API with fixed decoding parameters, `temperature=1.0` and `top-p=1.0`. The tested models are GPT-4o, GPT-4o-mini, Google Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.5 Flash Lite, DeepSeek V3, LLaMA 4 Maverick, and LLaMA 4 Scout [2507.23295].

For the reported overall baseline results under $P_1$, Gemini 2.5 Pro is best on all three tasks: $0.636$ Accuracy on $T_1$, $0.598$ $F_1$ on $T_2$, and $0.443$ $F_1$ on $T_3$. GPT-4o reports $0.597$, $0.287$, and $0.066$ on the same tasks, while GPT-4o-mini reports $0.538$, $0.323$, and $0.159$. The benchmark summary states that Gemini models excel at fine-grained error classification, whereas GPT variants are competitive chiefly on coarse error detection in $T_1$ [2507.23295].

The qualitative conclusions are as important as the raw scores. Even the best LMMs struggle with Size, Hallucination, and the low-frequency classes Split, Merge, Overlap, and Duplicate. The benchmark also reports that providing both structured JSON and visual boxes through $P_3$ generally improves performance, especially for Gemini. These findings indicate that LED-Dataset is not simply measuring OCR-adjacent perception; it is probing multimodal structural reasoning under controlled perturbations [2507.23295].

## 6. Limitations and research directions

LED-Dataset is explicitly synthetic. Its errors are injected according to empirical distributions and deterministic perturbation rules, but real model failures may exhibit more complex and correlated patterns. A second limitation is domain concentration: the injection procedure is based on DocLayNet styles, and one benchmark description characterizes this as a single-domain setup tied to scientific-article-like layouts. A third limitation is release structure: the dataset is presented as a diagnostic testbed rather than a conventional supervised-learning corpus, and no separate training, validation, and test splits are released [2603.17265].

These limitations shape the directions proposed for follow-up work. The benchmark description recommends extending injection to other document domains such as forms, receipts, and presentations; incorporating real model outputs to refine error distributions and co-occurrence modeling; developing end-to-end learning approaches that directly predict element-level error types for self-supervised correction; and combining LED with downstream tasks such as information extraction and question answering to measure end-task impact of structural errors [2603.17265].

Within those constraints, LED-Dataset occupies a specific methodological niche. It is not designed to replace layout detection datasets or standard localization metrics. Rather, it functions as an error-analysis layer that makes structural failure modes explicit and measurable.

## 7. Nomenclature and disambiguation

The name “LED” is highly overloaded in the literature. On arXiv it also refers to a vehicle tail light signal dataset and benchmark [2409.02508], a paired real-world event-camera denoising dataset [2405.19718], the Chinese legal event detection dataset LEVEN [2203.08556], a life event extraction dataset from dialogs [2304.08327], and “Light Enhanced Depth” for nighttime depth estimation [2409.08031]. The document-analysis LED-Dataset discussed here is specifically the dataset associated with layout error detection in DLA [2507.23295].

That terminological ambiguity has practical consequences for indexing and citation. In document understanding, “LED-Dataset” denotes the structurally annotated benchmark built on the DocLayNet test split and centered on eight standardized layout error types. In other subfields, the same acronym denotes unrelated resources, tasks, or models. Careful citation by title or arXiv identifier is therefore necessary to avoid conflation.

Source: https://www.emergentmind.com/topics/led-dataset