RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild
Abstract: Accurate document layout analysis remains a critical bottleneck for document parsing systems, due to the intricate coupling among heterogeneous document layout elements, geometric distortions (\eg, paper warping and bending, perspective variations), and reading order within diverse layout structures. Existing approaches typically rely on fragmented multi-stage pipelines or computationally heavy generative Transformer architectures, leading to error propagation and limited efficiency. In this paper, we present RT-DocLayout, a highly efficient end-to-end framework for document layout analysis, designed as a front-end for document parsing tasks. The proposed model unifies classification, detection, pixel-level segmentation, and reading order prediction for layout elements within a single 33M-parameter architecture. Built upon the RT-DETR, our key contribution is a unified multi-task formulation within a single query-based decoder that simultaneously classifies, regresses bounding box, generates masks, and constructs relationship to reason reading order. By jointly learning geometric and structural representations, RT-DocLayout introduces multi-task optimization that substantially improves robustness under real-world document distortions. Extensive experiments on public benchmarks demonstrate state-of-the-art performance in document layout analysis while maintaining real-time inference speed(132.1 FPS). When coupled with downstream OCR engines, RT-DocLayout significantly improves full-document reconstruction quality, providing a scalable and practical foundation for real-world document intelligence systems.
Summary
- The paper presents a unified Transformer that integrates pixel-level mask detection, bounding box regression, and reading order prediction for efficient document layout analysis.
- It introduces a physical spatial-aware data augmentation pipeline that simulates realistic distortions like skew, warping, and illumination changes.
- Empirical results on OmniDocBench and Real5-OmniDocBench demonstrate superior accuracy and speed (132.1 FPS) compared to larger vision-language models.
RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild
Introduction
RT-DocLayout addresses the persistent challenges in document layout analysis (DLA), specifically the accurate identification and structured parsing of layout elements in physically degraded documents. DLA underpins all automated document understanding systems, yet prior approaches—often multi-stage pipelines or monolithic generative transformers—suffer from either accumulated error propagation, inefficiency, or insufficient geometric fidelity. RT-DocLayout presents a paradigm shift: it unifies classification, pixel-level detection, mask-based segmentation, and reading order prediction in a single, highly-efficient 33M-parameter Transformer-based architecture built atop RT-DETR. This non-autoregressive, query-based model, coupled with a physical spatial-aware data augmentation schema, yields SOTA performance in both accuracy and speed across synthetic and real-world benchmarks, particularly under severe physical distortions (e.g., skew, warping, illumination, screen photography).
Architecture and Methodological Advances
RT-DocLayout's core is a unified query-based Transformer decoder, which, in each forward pass, outputs element class, bounding box, segmentation mask, and intrinsic reading order relationships. The multi-head architecture incorporates parallel streams for spatial segmentation and logical sequencing.
Figure 1: The unified architecture of RT-DocLayout, featuring parallel heads for segmentation and reading order prediction.
The innovation in reading order prediction is the direct coupling with the detection queries. Each object query is projected into a relational space; an antisymmetric scoring function computes pairwise precedence, and a voting-based strategy globally ranks elements. Unlike pointer-networks or post-hoc models, this integrated mechanism enables direct joint optimization of geometric and logical structure. The loss, heavily weighted for the order component to mitigate the gradient dilution, is computed over all valid pairwise relations at the last decoder layer, while auxiliary losses provide supervision to the classification, regression, and segmentation branches.
The mask-centric detection head departs from prior bounding-box limited systems, producing pixel-accurate regions—which is especially advantageous in non-rigid physical distortions. The spatial augmentation pipeline dynamically simulates physically realistic deformations by cascading mesh-based intrinsic warping (e.g., page curls) with extrinsic projective transformations (camera perspective), ensuring the model trains on high-fidelity synthetic geometric artifacts.
Figure 2: The Physical Spatial Aware Data Augmentation Pipeline.
Evaluation and Empirical Results
Experiments utilize an extensive 38k-instance labeled dataset spanning diverse document genres, with comprehensive annotations for both structure and reading order, and are evaluated primarily on OmniDocBench v1.5 and its real-world subset Real5-OmniDocBench.
Performance Analysis
RT-DocLayout achieves an overall score of 94.50% on OmniDocBench and 92.05% on Real5-OmniDocBench when paired with the PaddleOCR-VL-1.5-0.9B recognizer. The architecture not only consistently outperforms larger vision-LLMs (e.g., 1.2B-param MinerU2.5, 3B-param Dolphin-v2) in both accuracy and speed (132.1 FPS), but also demonstrates significant robustness in the most challenging physically-degraded scenarios (e.g., Skew: 91.66%, Warp: 91.25%).
Qualitative analysis highlights mask-guided predictions enabling precise extraction of text and structure, especially in non-uniform illumination or severe warping, surpassing previous SOTA methods like PP-DocLayoutV2, which still suffer from background interference and missed regions under such conditions.
Figure 3: Qualitative comparison on four real-world scenarios. Each group shows, left to right: Origin, PP-DocLayoutV2, Ours. (a) Skew. (b) Page warping. (c) Non-uniform illumination. (d) Screen photography.
Ablation Studies
Ablation studies confirm the criticality of each architectural component:
- Mask-based localization consistently improves all accuracy and order metrics over box-only supervision, especially in warped and skewed cases.
- Joint (coupled) reading order learning dramatically outperforms both post-hoc (XYCut) and decoupled models (RO drops from 0.189 and 0.074 to 0.041).
- Physical spatial-aware data augmentation yields large robustness gains (e.g., Skew: +14.11% overall, RO: 0.146 → 0.061).
Implications and Future Directions
RT-DocLayout's unified inference for both geometric and logical document structure, combined with high-fidelity pixel-level masks and an efficient, deterministic decoding paradigm, establishes it as an optimal front-end for scalable document intelligence systems. Its robust performance under severe distortion directly addresses industrial applications where uncontrolled image capture (e.g., handheld photography, camera scans) is prevalent. Moreover, the mask-centric approach yields purified inputs for downstream OCR/VLM stages, which is critical for pipeline composability and global document parsing quality.
The joint multi-task paradigm foreshadows future directions in document intelligence where geometric, visual, and logical structures are consistently fused in compact, end-to-end architectures. Incorporation of multilingual, multi-modal, or retrieval-augmented capacities could further extend the real-world applicability of such systems.
Conclusion
RT-DocLayout sets a new empirical upper bound in efficient document layout analysis, unifying segmentation and reading order prediction under a single query-based Transformer framework. The model's mask-level geometric reasoning, robust physical data augmentation, and high-throughput inference establish both the practical and theoretical foundation for the next generation of document parsing architectures. Its contributions push the field towards structurally aware, robust, and real-time document intelligence solutions suited to in-the-wild deployment.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about
This paper introduces RT-DocLayout, a fast and compact AI model that can look at a document page (like a PDF or a phone photo of a page) and figure out:
- what each part is (text, title, table, image, etc.),
- exactly where each part is (down to the precise pixels, not just a rough box),
- and in what order you should read those parts.
The goal is to help computers “see” and understand documents the way people do, even when pages are bent, skewed, photographed at an angle, or poorly lit.
What questions the researchers asked
The team focused on three simple questions:
- Can one small, fast model do four jobs at once—find items, name them, outline their exact shape, and decide the reading order?
- Will using exact shapes (masks) instead of simple rectangles make things work better on messy, real-world pages (like phone photos)?
- Can we make the model strong and reliable by training it with realistic “bent and tilted page” examples?
How the model works (in everyday language)
Think of analyzing a page like organizing a messy desk:
- Older systems would first find items (like sticky notes), then run separate steps to sort or read them. Errors in step one often messed up the next steps.
- RT-DocLayout acts like a single organized helper that does everything in one go.
Here’s how RT-DocLayout thinks about a page:
- It uses a modern “Transformer” system (a popular AI design) that asks the image a set of smart “questions” (called queries): “Is there a title here?”, “Is that a table?”, “Where exactly are its pixels?”
- Instead of drawing rough rectangles, it “colors in” the exact shape of each element (called a mask). This is like carefully tracing the outline of an object instead of putting a big sticky note over it. That way, the text inside is clean and there’s less background noise for OCR tools.
- It also learns reading order by comparing pairs of elements and deciding which one comes first, like deciding the order of comic panels. It then turns those pairwise decisions into a complete reading sequence.
To make it tough and reliable in real life, they trained it with “physical” data tricks that imitate:
- pages that curl or warp (like a page in a book that doesn’t lie flat),
- camera angles that tilt or skew the page,
- other effects like uneven lighting or screen glare. These tricks help the model handle phone photos and scanned documents, not just perfect PDFs.
Behind the scenes, the model:
- Is built on a speedy detector called RT-DETR.
- Is small (about 33 million parameters—compact for this kind of model).
- Trains all tasks together (classification, boxes, masks, reading order), so the parts learn to help each other.
What they found and why it matters
Main results:
- Accuracy + speed: RT-DocLayout reached state-of-the-art accuracy on public tests while running in real time (about 132 pages per second on a powerful GPU).
- Exact shapes help: Using pixel-level masks made the model more accurate and robust than using rectangles, especially when pages were warped, tilted, or photographed from a screen.
- Better reading flow: Predicting reading order inside the same model (instead of as a separate step) significantly improved how well the model reconstructed the correct sequence of content.
- Works well with OCR: When connected to different OCR systems, RT-DocLayout consistently boosted overall document understanding (text, formulas, tables, and reading order).
- Robust in the wild: It performed especially well on tough scenarios like warping and skew—common issues in mobile photos—thanks to the realistic training distortions.
Why this matters:
- Clean inputs mean better OCR: Because the model cuts out exact shapes, OCR tools get less background clutter and make fewer mistakes.
- Fewer moving parts: Doing everything in one pass reduces errors that can pile up in long, multi-step pipelines.
- Practical: It’s fast and small enough for real-world use at scale.
What this could change
- Better document apps: From scanning homework to processing invoices, apps can now more accurately pull out text, tables, formulas, and images in the right order—even from quick phone photos.
- Faster pipelines: Companies that process large numbers of documents can get both speed and quality without huge hardware.
- A base for smarter systems: Because it produces clean, structured layouts with reading order, it’s a strong front-end for more advanced systems (like AI that understands the meaning of a report).
In short, RT-DocLayout shows that one compact, real-time model can accurately “see” a document’s structure and reading flow—even when the page isn’t perfect—making digital document understanding more reliable and practical.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what remains uncertain, missing, or unexplored in the paper, intended to guide follow-up research and engineering work.
- Dataset transparency and generalization: The 38k in-house training set (25 categories) is not publicly available; generalization to unseen domains, document genres, or annotation styles cannot be independently verified.
- Coverage across languages and scripts: Robustness of reading order and segmentation for RTL scripts (e.g., Arabic/Hebrew), CJK vertical layouts, mixed-orientation pages, and multilingual documents is not evaluated.
- Public benchmarks beyond OmniDocBench: No results on widely used public datasets (e.g., PubLayNet, DocBank, PRIMA, ICDAR tasks), limiting comparability and external validation.
- Multi-page sequencing: The model operates at page level; cross-page reading order and document-level sequencing are unaddressed.
- Hierarchical structure: Only linear reading order is predicted; hierarchical relations (section–subsection), figure–caption linkage, footnote–anchor associations, and other semantic graph relations are not modeled.
- Pairwise order consistency: The anti-symmetric pairwise scorer does not enforce transitivity; cycles/inconsistencies may arise. The vote-based ranking (Borda-like) lacks a formal tie-breaking strategy and guarantees for global consistency.
- Scaling with dense layouts: Pairwise order inference is O(N2). Runtime/memory scaling with large N (dense pages) and its effect on FPS/latency are not reported.
- Order-loss design choices: Order supervision is applied only at the final decoder layer; the impact of intermediate-layer supervision, alternative listwise losses, or differentiable sorting is not explored.
- Loss-weight sensitivity: The large order loss weight (λ_order = 50) is heuristic; no sensitivity analysis or principled loss-balancing (e.g., GradNorm, DWA, PCGrad) is provided.
- Task interference: Sharing a single decoder for detection, segmentation, and order may cause gradient conflicts; no analysis or mitigation of multi-task interference is given.
- Handling missing/spurious elements in order: It is unclear how the ranking procedure deals with false positives/negatives and how order evaluation accounts for unmatched elements.
- Granularity of reading order: The approach targets block-level ordering; line-level and token-level ordering, and the interplay between block and line order, are not studied.
- Mask resolution and thin structures: The effect of mask resolution on thin elements (e.g., table rules, separators, superscripts) and OCR-friendliness is not analyzed.
- Segmentation specifics: The mask head architecture, mask resolution, NMS/overlap handling, and their influence on downstream OCR are insufficiently detailed for reproduction and optimization.
- Augmentation realism and validation: The proposed intrinsic surface deformation (ISD) and extrinsic projective transformation (EPT) are synthetic; no quantitative alignment to real capture statistics, parameter distributions, or validation against real warped datasets is provided.
- Photometric degradations: Robustness to blur, compression artifacts, glare/ghosting, shadows, color cast, and noise is not systematically measured beyond the “illumination” dimension reported.
- Ablations on augmentation components: Contributions of ISD vs. EPT vs. standard augmentations are not isolated; hyperparameters (α, β, distributions) and code for reproducibility are not fully specified.
- OCR coupling and domain dependence: Gains are shown with a few recognizers; the extent to which improvements persist with different OCR systems, languages, or low-resource recognizers remains unclear.
- Purely visual order modeling: Reading order is inferred without textual or linguistic cues; the potential benefits of integrating text content (e.g., line breaks, numbering, punctuation) are unexplored.
- Standard detection/segmentation metrics: The paper relies on downstream parsing metrics; standard mAP/AP/PQ for detection/segmentation are not reported, making it harder to compare geometric quality directly.
- Fairness of baselines: Comparisons to VLM-based parsers and YOLO+XYCut may be mismatched in objectives/capabilities; head-to-head against unified order-aware detectors (e.g., DLAFormer) on common benchmarks is missing.
- Runtime reporting: Reported FPS is on A100 with batch size 32; single-image latency, memory footprint, and performance on CPUs/edge/mobile hardware (and under varying N) are not reported.
- High-resolution inputs and small text: Input preprocessing (scaling, tiling) is not detailed; impact on tiny text, dense equations, or extreme aspect ratios is unreported.
- Annotation ambiguity and noise: Reading-order labeling can be subjective; beyond adopting GCE, inter-annotator agreement, ambiguity handling, and sensitivity to noisy order labels are not analyzed.
- Domain breadth: Handwritten content, stamps/seals, signatures, marginalia, and non-document imagery within pages are not separately evaluated; failure modes for these cases are not documented.
- Robustness across page types: Magazines, comics, posters, forms with multi-path reading orders, and non-grid layouts are not specifically tested.
- Table and formula structure: While downstream metrics (TEDS, CDM) are reported, the layout model itself does not output structural representations (e.g., table cell graph, formula tree); integration with structure parsers is left open.
- Post-processing to OCR: The cropping/masking pipeline and its parameters (e.g., dilation, padding, binarization) are not described; how such choices affect OCR quality is unknown.
- Reproducibility and variability: Random seed variability, confidence intervals, and multi-run stability are not reported; some hyperparameters and exact training recipes are insufficiently specified for exact replication.
- Licensing and data confidentiality: Use of an in-house labeled corpus raises questions about licensing, privacy, and whether comparable public substitutes can achieve the same performance.
Practical Applications
Immediate Applications
Below are concrete use cases that can be deployed now by leveraging the paper’s released model (PP-DocLayoutV3) and methods, especially the unified real-time layout+mask+reading-order pipeline and the distortion-aware augmentation.
- Enterprise document automation (Finance, Insurance, Manufacturing)
- What: Higher-accuracy extraction from invoices, receipts, purchase orders, contracts, shipment docs by feeding OCR with pixel-level masks and correct reading order.
- Tools/workflows: Replace/augment current layout front-ends (e.g., YOLO/box-only) in RPA and IDP stacks; integrate with PaddleOCR/Tesseract; export structured JSON/HTML with sequence-preserving text blocks.
- Assumptions/dependencies: Domain-specific class mapping and light fine-tuning may be needed; throughput depends on available GPU/CPU; compliance/security requirements may require on-prem deployment.
- Accessible PDF tagging and reading-order repair (Government, Publishing, Education)
- What: Auto-tag PDFs for PDF/UA compliance; fix broken reading order to improve screen-reader experiences.
- Tools/workflows: Acrobat/LibreOffice plugins or CLI to convert RT-DocLayout outputs (masks + order) into PDF structure tags and reflowed HTML/EPUB.
- Assumptions/dependencies: Accurate mapping from predicted categories to PDF tagging schema; multilingual/vertical-text coverage validated on target corpora.
- Healthcare forms and clinical document ingestion
- What: Robust parsing of intake forms, lab reports, discharge summaries captured by mobile scanners; mask-aware redaction of PHI.
- Tools/workflows: EHR ingestion microservice that crops with masks before OCR; rule/ML-based field extractors operating on cleaned regions; redaction pipelines using masks to ensure complete coverage.
- Assumptions/dependencies: HIPAA/GDPR compliance; on-prem inference or VPC isolation; template diversity may require fine-tuning.
- Archival digitization and library pipelines (Academia, Cultural Heritage)
- What: Enhanced OCR on warped pages from bound volumes; reliable reading order for multi-column and mixed layouts.
- Tools/workflows: Batch processing pipeline that uses masks to isolate text/figures/tables; export TEI/ALTO/IIIF with correct order; confidence-based human-in-the-loop QA.
- Assumptions/dependencies: Historical fonts/handwriting may need complementary models; strict preservation policies may constrain compute and data movement.
- Scientific PDF parsing and math/table isolation (Academia, Software)
- What: Better formula/table segmentation and sequence-preserving text for PDF-to-HTML/LaTeX conversion and literature mining.
- Tools/workflows: Preprocessor for tools like GROBID, Science Parse, arXiv scrapers; mask-guided formula OCR (e.g., LaTeX recognition) and TEDS-improved table extraction.
- Assumptions/dependencies: Downstream formula/table recognizers still required; licensing of target corpora; handling of vector PDFs vs scanned images.
- Mobile scanning SDKs and office apps (Daily life, SMBs)
- What: Clearer receipts/notes scans and correct text flow from angled or warped captures.
- Tools/workflows: iOS/Android SDK using quantized RT-DocLayout; on-device cropping with masks; reading-order-aware reflow for share/export.
- Assumptions/dependencies: Model optimization (INT8/FP16) and hardware acceleration (NNAPI, Core ML, GPU) needed; battery/latency constraints.
- Retrieval-augmented generation (RAG) and AI indexing (Software)
- What: Higher-quality chunks (mask-cropped, order-correct) for embeddings and QA, reducing hallucinations from mixed content.
- Tools/workflows: Pre-ingestion step in ETL: segment → crop via masks → OCR → chunk respecting reading order → index with metadata (region type, page, order).
- Assumptions/dependencies: Integration with vector DBs; re-indexing costs for existing corpora; evaluate task-specific gains.
- Synthetic data generation via physical spatial aware augmentation (ML Ops)
- What: Use the paper’s warping + perspective augmentation to harden any document model (detectors, OCR, table/Math recognizers) to real-world distortions.
- Tools/workflows: Training-time augmentation library or standalone dataset synthesizer; curriculum schedules (mild→strong distortion).
- Assumptions/dependencies: Parameter tuning to match target capture conditions; ensure label consistency with masks; compute for large-scale augmentation.
Long-Term Applications
These use cases are feasible with further research, scaling, or engineering—e.g., broader datasets, on-device optimization, tighter integration with generative models, or standardization.
- On-device real-time reading assistance and AR overlays (Accessibility, Education)
- What: Live TTS that follows correct reading order, highlights segments, and filters background using masks for low-vision users or language learners.
- Tools/products: AR glasses/phone apps combining RT-DocLayout with TTS/translation; interactive reflow.
- Dependencies: Efficient quantization/distillation to meet mobile latency; privacy-first on-device inference; robust multilingual/vertical-text support.
- OCR-free or hybrid document understanding with VLMs
- What: Replace heavy generative doc parsers by coupling mask+order outputs with lightweight captioners or VLMs for key-value, table, and figure understanding.
- Tools/products: Pipeline where RT-DocLayout structures the page, and a small VLM handles localized semantics per region.
- Dependencies: Training data for multimodal alignment; avoiding 1–3B+ parameter VLM costs; latency/throughput targets for production.
- Autonomous “document agents” for end-to-end workflows (Finance, Gov, Enterprise)
- What: Intake → structure → extract → validate → reconcile → file, with confidence-based exception handling.
- Tools/products: Orchestrations using RT-DocLayout for structure, field extractors, rule engines, and LLM-based validators.
- Dependencies: Strong guarantees on accuracy and auditability; integration with ERP/EHR/CRM; human-in-the-loop design patterns.
- Standardized reading-order annotations and PDF/UA interoperability (Policy, Standards)
- What: Create community benchmarks and converters so engines can output interoperable reading-order graphs and masks mapped to standardized tags.
- Tools/products: Open schema and reference converters (e.g., PDF tags, EPUB); certification tests.
- Dependencies: Cross-vendor adoption; multilingual and script-specific conventions; governance for evolving standards.
- Certified redaction with verifiable coverage (Legal, Compliance)
- What: Mask-driven redaction that provably covers sensitive regions (PII/PHI), with audit logs and reproducible policies.
- Tools/products: Redaction service that stores mask provenance and hashes; differential previews for auditors.
- Dependencies: Third-party certification; handling edge cases (bleed-through, watermarks, handwriting); legal acceptance.
- Kiosk/mailroom/robotic sorting for incoming documents (Logistics, Government)
- What: Real-time triage of forms, letters, and shipping docs under uncontrolled lighting/angles.
- Tools/products: Edge gateways with cameras; RT-DocLayout powering layout-aware routing and barcode/ID capture.
- Dependencies: Ruggedized hardware; deterministic latency requirements; secure edge compute and maintenance.
- Low-supervision expansion to new domains and scripts
- What: Use the paper’s augmentation and weak supervision to adapt layout+order models to niche domains (e.g., energy field forms, maritime logs) and low-resource scripts.
- Tools/products: Active learning loops; synthetic-to-real adaptation with domain-specific distortion profiles.
- Dependencies: Seed annotations for new class taxonomies; evaluation datasets; script/handwriting coverage.
- Web-scale corpus structuring for AI assistants (Search, Knowledge Graphs)
- What: Convert PDFs at scale to structured, order-preserving HTML/JSON for trustworthy citation and downstream analytics.
- Tools/products: Distributed pipelines (Spark/Ray) using RT-DocLayout; provenance tracking; downstream KG construction.
- Dependencies: Compute and storage budgets; licensing/robots handling; de-duplication and quality control.
- Privacy-preserving federated learning for document parsing (Healthcare, Finance)
- What: Improve models across organizations without sharing raw documents by exchanging gradients or distilled features.
- Tools/products: Federated training with distortion-aware augmentations; secure aggregation.
- Dependencies: Differential privacy budgets; heterogeneous data distributions; governance agreements.
- Explainability and quality assurance dashboards (Ops, Risk)
- What: Visualize masks, reading-order graphs, and confidence to debug failures and guide retraining.
- Tools/products: UI that overlays predictions; drift monitors for capture conditions (warp, skew, illumination).
- Dependencies: Telemetry pipelines; standardized metrics (e.g., Real5-like scores) for production monitoring.
Notes on general dependencies across applications:
- Performance: Reported 132.1 FPS was on an A100 with batch size 32; per-page latency varies by hardware and batch size. Edge/mobile deployments require optimization (e.g., quantization, pruning).
- Downstream stack: Gains depend on OCR/recognizer quality and correct use of masks and reading order. Table/math/handwriting still need specialized recognizers.
- Data/labels: New domains may need taxonomy mapping and light fine-tuning; reading-order conventions can be domain/language-specific.
- Licensing and openness: Model weights are available as PP-DocLayoutV3 via PaddleOCR/Hugging Face; ensure license compatibility and attribution.
- Security/compliance: Sensitive domains (healthcare/finance/gov) require on-prem or secure cloud, audit trails, and adherence to applicable regulations.
Glossary
- AdamW optimizer: A variant of Adam that decouples weight decay from gradient updates to improve generalization. "We utilize the AdamW optimizer with a weight decay of 0.0001."
- Anti-symmetric (relation matrix): A matrix S where S_{i,j} = −S_{j,i}, used here to encode pairwise precedence relations. "The resulting relation matrix is inherently anti-symmetric ()"
- Autoregressive Transformer: A model that generates outputs sequentially, each step conditioned on previous outputs. "LayoutReader~\cite{wang2021layoutreaderpretrainingtextlayout} pre-trains an autoregressive Transformer on the ReadingBank benchmark"
- Axis-aligned bounding boxes: Rectangular boxes with edges parallel to the axes, used to localize objects but limited under distortions. "all methods remain confined to axis-aligned bounding boxes"
- BCE (Binary Cross-Entropy): A loss function for binary classification or mask prediction tasks. "Pixel-level mask supervision comprises binary cross-entropy and Dice loss"
- Bipartite matching: Matching elements from two sets optimally, often used to align predictions with ground truth. "We employ the Hungarian algorithm to establish optimal bipartite matching between predictions and ground-truth annotations"
- Cascaded pipelines: Multi-stage systems where outputs of one stage feed into the next, prone to error propagation. "cascaded pipelines attempt to reconstruct reading order by stacking a layout detector with secondary relationship models"
- CDM: A formula recognition accuracy metric (context-specific to benchmarks). "Formula\textsuperscript{CDM}"
- DETR: A Transformer-based detection framework that formulates detection as set prediction. "end-to-end DETR-based framework"
- Deep supervision: Applying auxiliary losses at intermediate layers to improve optimization and convergence. "Auxiliary losses from intermediate decoder layers provide deep supervision for classification, detection, and segmentation"
- DiT: A document Transformer pre-trained in a self-supervised manner for document understanding tasks. "DiT~\cite{li2022ditselfsupervisedpretrainingdocument} introduced self-supervised pre-training for document Transformers"
- Dice loss: A segmentation loss that measures overlap between predicted and ground-truth masks, improving class imbalance robustness. "Pixel-level mask supervision comprises binary cross-entropy and Dice loss"
- DLAFormer: A DETR-based model that jointly predicts layout regions and reading order via a unified label space. "DLAFormer~\cite{wang2024dlaformerendtoendtransformerdocument} jointly predicts layout regions and reading order via a unified label space in an end-to-end DETR-based framework"
- Donut: An OCR-free encoder–decoder model that generates structured document markup. "Donut~\cite{kim2022ocrfreedocumentunderstandingtransformer} and Nougat~\cite{blecher2023nougatneuralopticalunderstanding} generate structured markup via OCR-free encoder--decoder models"
- Extrinsic Projective Transformation (EPT): A simulated camera-induced geometric distortion modeled via a homography. "Stage 2: Extrinsic Projective Transformation (EPT)."
- Focal loss: A classification loss that down-weights easy examples to focus learning on hard, imbalanced cases. "For category prediction, we adopt focal loss with and "
- GCE (Generalized Cross Entropy): A robust loss function that interpolates between MAE and cross-entropy to handle noisy labels. "we employ Generalized Cross Entropy (GCE) loss with label smoothing ()"
- GIoU (Generalized Intersection over Union): A bounding-box regression loss that improves over IoU by providing useful gradients when boxes do not overlap. "Bounding box regression combines loss and GIoU loss:"
- GNNs (Graph Neural Networks): Neural architectures that operate on graph-structured data to model relationships. "GNNs~\cite{lee2021ropereadingorderequivariant}"
- Homography: A 3×3 projective transformation matrix modeling perspective mappings between planes. "We formulate this process as a composite homography "
- Hungarian algorithm: An algorithm for solving assignment problems optimally in polynomial time. "We employ the Hungarian algorithm to establish optimal bipartite matching"
- Importance weighting: Adjusting sample contributions to the loss based on their significance, often to emphasize uncertain regions. "we employ point-based sampling with importance weighting to compute mask loss efficiently"
- Intrinsic Surface Deformation (ISD): A model of non-rigid page deformations (e.g., curl) applied before camera projection. "Stage 1: Intrinsic Surface Deformation (ISD)."
- Label smoothing: A regularization technique that softens target labels to improve generalization and calibration. "we employ Generalized Cross Entropy (GCE) loss with label smoothing ()"
- LayoutLMv3: A multimodal Transformer that jointly pre-trains on text, layout, and visual signals for document AI. "LayoutLMv3~\cite{huang2022layoutlmv3pretrainingdocumentai}"
- MaskDINO: A unified Transformer-based framework for detection and segmentation that inspires mask training strategies. "Following MaskDINO, we employ point-based sampling with importance weighting"
- Mesh-based warping operator: A deformation operator using smooth periodic displacement fields to simulate non-rigid page warping. "We propose a mesh-based warping operator."
- Non-autoregressive decoding: Producing outputs in parallel without conditioning each step on previous outputs. "replacing both autoregressive generation and cascaded pipelines with deterministic, non-autoregressive decoding"
- OCR-free: Approaches that bypass traditional text recognition, directly generating structured outputs from images. "generate structured markup via OCR-free encoder--decoder models"
- Object queries: Learnable embeddings in Transformer decoders that represent potential objects/elements for prediction. "the decoder iteratively refines object queries"
- OmniDocBench v1.5: A benchmark evaluating document parsing (layout, formulas, tables, reading order) under diverse conditions. "OmniDocBench v1.5"
- Pairwise reading order supervision: Training signals defined over element pairs to learn precedence relations for reading order. "The key contribution of our loss design is the pairwise reading order supervision."
- Perspective foreshortening: Apparent shrinking of objects with distance in a perspective projection, modeled via projective parameters. " introduces perspective foreshortening via its projective parameters and "
- Pointer-network: A neural architecture that outputs sequences of indices pointing to input elements, often for ordering tasks. "the decoupled pointer-network design used in prior systems"
- Point-based sampling: Selecting a subset of pixels/points for efficient mask loss computation during segmentation training. "we employ point-based sampling with importance weighting to compute mask loss efficiently"
- Query-based Transformer decoder: A decoder that processes a fixed set of learned queries to predict objects/regions. "within a single query-based Transformer decoder"
- Real5-OmniDocBench: A real-world benchmark focusing on five physical distortion scenarios (Scanning, Warping, Screen-Photography, Illumination, Skew). "Real5-OmniDocBench"
- Relational embedding space: A learned space where query embeddings are projected to compute relationships (e.g., precedence scores). "map queries to a relational embedding space"
- Relative reading order logits matrix: The matrix of pairwise logits indicating which element precedes another. "Relative reading order logits matrix is calculated from the refined query embeddings"
- RT-DETR: A real-time DETR variant serving as the base detector for the proposed model. "Built upon the RT-DETR"
- TEDS: A table structure similarity metric based on tree edit distance used for table evaluation. "Table\textsuperscript{TEDS}"
- VLMs (Vision-LLMs): Large models jointly trained on visual and textual data for multimodal understanding. "VLMs (\eg, LayoutLMv3~\cite{...}, DiT~\cite{...}, MinerU~\cite{...}, Dolphin~\cite{...})"
- Voting-based ranking strategy: Aggregating pairwise precedence probabilities to obtain a globally consistent reading sequence. "a voting-based ranking strategy converts pairwise relations into a globally consistent sequence"
- XYCut: A heuristic rule-based algorithm that recursively partitions the page to infer reading order. "Rule-based XYCut~\cite{ha1995recursive} is brittle on complex layouts."
Collections
Sign up for free to add this paper to one or more collections.
