RT-DocLayout: Real-Time Document Layout Analysis

• RT-DocLayout is a suite of real-time document layout analysis models that unify detection, segmentation, and reading order reasoning in a single pass.
• It leverages transformer-based architectures with multi-layer decoders and attention mechanisms to achieve state-of-the-art accuracy and competitive speed.
• The framework integrates robust post-processing and layout-aware prompting for LLMs, ensuring effective parsing under diverse document distortions.

RT-DocLayout refers to a family of real-time, end-to-end document layout analysis models and pipelines that unify detection, segmentation, and reading order reasoning, providing robust, high-speed front-ends for document parsing and OCR-based understanding tasks. The RT-DocLayout framework encapsulates a broad range of model implementations, notably including the single-architecture RT-DocLayout multi-task Transformer (Cui et al., 22 Jun 2026), the suite of Docling detector variants (Livathinos et al., 15 Sep 2025), and their practical integration with layout-aware prompting for LLMs (Lamott et al., 2024). These approaches are characterized by their ability to segment heterogeneous layout elements, handle geometric distortions, and predict logical reading order within diverse, real-world document structures, all with state-of-the-art accuracy and competitive runtimes.

1. Core Architectures and Design Principles

RT-DocLayout systems are unified by attention-based, transformer-style architectures built on RT-DETR backbones. The signature RT-DocLayout model (Cui et al., 22 Jun 2026) employs a lightweight convolutional backbone with Feature Pyramid Network (FPN) to extract multiscale features, followed by a multi-layer Transformer decoder maintaining $N=300$ object queries $\{q_i\}_{i=1}^N$.

Each query interacts via cross-attention with visual features and via self-attention with other queries. After decoding, outputs fan out into four heads:

• Classification head: Per-query MLP using focal loss to handle class imbalance.
• Bounding box regression head: Per-query MLP predicting $(x, y, w, h)$, optimized with L1 and GIoU losses.
• Mask head: Per-query Transformer followed by upsampling, producing a binary segmentation mask (MaskDINO-inspired).
• Reading-order relational head: Per-query scorer producing an anti-symmetric $N\times N$ matrix encoding pairwise order relations.

Across variants (Livathinos et al., 15 Sep 2025), different detector families (RT-DETR, RT-DETRv2, D-FINE) are instantiated with architectural modifications:

Model	Backbone	Params	Decoder Layers	Mask/RO Output	Speed (A100)
RT-DocLayout	RT-DETR (ResNet)	33M	6	Yes	132.1 FPS
heron-101 (RT-DETRv2)	ResNet-101-vd	76.7M	8	No	28 ms/img
egret-m (D-FINE-med)	HGNet-V2 medium	19.5M	6	No	24 ms/img

Critical innovations include a unified multi-task objective, deep supervision, optimized backbone/pretraining, and fine-grained post-processing for clustering and assignment (Livathinos et al., 15 Sep 2025).

2. Unified Multi-Task Optimization

The RT-DocLayout formulation integrates detection, segmentation, and reading-order prediction in a single forward pass, with the following loss structure (Cui et al., 22 Jun 2026):

$$\mathcal{L}_{\mathrm{total}} = \lambda_{\mathrm{cls}} \mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{bbox}} \mathcal{L}_{\mathrm{bbox}} + \lambda_{\mathrm{giou}} \mathcal{L}_{\mathrm{giou}} + \lambda_{\mathrm{mask}} \mathcal{L}_{\mathrm{mask}} + \lambda_{\mathrm{dice}} \mathcal{L}_{\mathrm{dice}} + \lambda_{\mathrm{order}} \mathcal{L}_{\mathrm{order}}$$

• Classification: Focal loss ($\alpha=0.25$, $\gamma=2.0$) per query
• Box regression: L1 and GIoU between predicted and ground-truth boxes
• Segmentation: Binary cross-entropy and Dice loss on sampled mask points
• Reading order: Generalized cross-entropy on an anti-symmetric pairwise ordering matrix, with higher weight ($\lambda_{\mathrm{order}}=50$) to counteract gradient vanishing

Losses except $\mathcal{L}_{\mathrm{order}}$ are applied to all decoder layers (deep supervision), while the order loss is applied only at the final layer.

The reading order is inferred through deterministic, non-autoregressive ranking based on pairwise scores $S_{i,j}$, replacing previous pointer network or GNN-based cascades.

3. Datasets, Training Protocols, and Augmentation

RT-DocLayout models are trained and evaluated on a diverse sampling of document images incorporating various formats and degradations:

• In-house training set: 38,000 real documents, 25 fine-grained categories with reading order (Cui et al., 22 Jun 2026); 150,000 pages and 2.3M annotations for Docling models (Livathinos et al., 15 Sep 2025)
• Benchmarks: OmniDocBench v1.5 (PDF-style), Real5-OmniDocBench (mobile, warped, illuminated), DocLayNet and DocLayNet-v2 (for detectors), SROIE and custom KIE tasks (for LLM-based pipelines) (Cui et al., 22 Jun 2026, Livathinos et al., 15 Sep 2025, Lamott et al., 2024)

Physical spatial-aware data augmentation is essential for robustness. This introduces intrinsic surface deformation (ISD) and extrinsic projective transformations (EPT) simulating warping, skew, and illumination effects. Mask and box annotations are correspondingly transformed (Cui et al., 22 Jun 2026).

For detector variants, heavy random scale, flip, color, and zoom augmentations dominate, with attention to rare-class sampling and noise filtering in constructing canonical training sets (Livathinos et al., 15 Sep 2025).

4. Evaluation Metrics and Empirical Performance

Standard COCO-style metrics are used for detectors: mAP@[.50:.05:.95], [email protected], [email protected], and AP by size. Downstream evaluation uses “Edit,” “CDM,” and TEDS metrics for text, formula, and table recognition as well as reading-order edit distance.

Key experimental results:

• RT-DocLayout (Cui et al., 22 Jun 2026): 94.5% Overall score on OmniDocBench v1.5, outperforms heavier models (MinerU2.5, Dolphin-v2) and previous SOTA PP-DocLayoutV2. Real-time 132.1 FPS on NVIDIA A100.
• Docling detectors (Livathinos et al., 15 Sep 2025): “heron-101” achieves 78% mAP on canonical DocLayNet with 28 ms/img on A100, improving absolute mAP by 23.9% over the baseline.
• Ablations: Mask-based localization and coupled order prediction yield measurable gains ($\{q_i\}_{i=1}^N$0Warp: +4.57%, $\{q_i\}_{i=1}^N$1Skew: +14.11%) in challenging conditions.

RT-DocLayout demonstrates significant performance gains on document parsing tasks under real-world, distorted, and mobile-capture scenarios, with qualitative examples demonstrating robust region isolation and background suppression (Cui et al., 22 Jun 2026).

5. Post-Processing, Deployment, and Best Practices

Post-processing in RT-DocLayout pipelines (Livathinos et al., 15 Sep 2025) includes confidence thresholding, IoU-based matching to OCR/PDF text “cells,” wrapper-object expansion, and rule-based pruning. Both wrapper and regular elements are post-processed to remove implausible clusters and erroneous overlaps. Deterministic clustering promotes spatial and categorical consistency post-detection.

For deployment, key recommendations and practices include:

• Model quantization (ONNX/INT8 or PyTorch quantization) can yield 30–50% inference speedups with minor (<1% mAP) accuracy degradation.
• Pruning (10–20% decoder heads) and knowledge distillation (heron-101 to heron) reduce latency up to 20% at ≈1% mAP cost.
• For high-throughput inference, saturate A100 GPUs with large batches (≥200) and multi-threaded data loading; select lighter backbones for Apple/CPU environments.
• Monitor known failure modes: small-object confusion, over-merged wrappers, dense layout fragmentation.

6. Integration with LLMs and Layout-Aware Prompting

When paired with LLMs, RT-DocLayout outputs provide layout-aware enrichment via strategies such as LAPDoc (Lamott et al., 2024). The recommended pipeline is:

1. OCR or detector-based extraction of blocks with geometry and reading order.
2. Normalization of coordinates to [0,1] scale.
3. Sequencing of blocks, optionally correcting OCR order (NEAREST strategy).
4. Verbalization of block-level layout via formats such as SpatialFormat or BoundingBox Markup.
5. Prompt assembly: insert verbalized layout into a template for downstream LLM reasoning (QA, KIE, etc.).
6. LLM call in deterministic (temperature=0, JSON mode) configuration.
7. Strict parse and downstream answer extraction.

This enrichment closes much of the gap to specialized multi-modal models and offers up to 15% relative accuracy gains on infographics, KIE, and VQA tasks compared to plain-text LLM prompts. Robustness is maintained under OCR reordering and up to ±20px coordinate jitter (Lamott et al., 2024).

7. Limitations, Failure Modes, and Future Directions

Known limitations include sensitivity to OCR errors (e.g., merged bounding boxes), failure with extremely large/complex grids, and challenges in non-rectilinear multi-page or missing-layout inputs (Lamott et al., 2024). Object detectors may still confuse geometrically similar classes (Form/Table) and struggle with tiny elements (Checkboxes). Overlap post-processing can merge adjacent tables, necessitating tuned heuristics.

A plausible implication is that further integration of small-object-aware heads and advanced post-processing, as well as the extension to multi-page and multilingual settings, are critical near-term research directions. As code, weights, and pipelines are open-source and HuggingFace-hosted, RT-DocLayout provides a scalable, reproducible foundation for research and industry deployment in document intelligence (Livathinos et al., 15 Sep 2025, Cui et al., 22 Jun 2026, Lamott et al., 2024).