Visually-Rich Document Understanding (VRDU)

• VRDU is the process of automatically extracting, structuring, and interpreting multimodal content from documents using text, graphics, and layout cues.
• It integrates computer vision, natural language processing, and document analysis to perform key information extraction, classification, and visual question answering.
• Research explores OCR-dependent and OCR-free models that emphasize robust reading order, template generalization, and domain adaptation in diverse document formats.

Visually-Rich Document Understanding (VRDU) denotes the automatic extraction, structuring, and interpretation of information from documents whose semantics are conveyed jointly by text, graphics, and complex layout, including forms, invoices, academic papers, and reports. The field lies at the confluence of Natural Language Processing, Computer Vision, and Document Analysis, and its standard task spectrum includes key information extraction, document visual question answering, entity linking or relation extraction, document classification, and structure analysis (Ding et al., 14 Jul 2025, Ding et al., 2024). Contemporary VRDU research spans OCR-dependent pipelines, OCR-free end-to-end models, and markup-native methods for HTML or XML documents, while increasingly emphasizing reading order, template variation, hierarchical entities, long-context reasoning, and robustness under domain shift (Barboule et al., 4 Jan 2025, Li et al., 2021).

1. Problem formulations and document assumptions

VRDU is usually posed over documents in which textual content alone is insufficient for interpretation. In business forms, receipts, filings, and similar artifacts, field identity often depends on two-dimensional arrangement, table structure, headers, key–value alignment, repetition patterns, or visual salience rather than on lexical content alone (Wang et al., 2022). This is why benchmark design in VRDU explicitly targets rich schema, layout-rich documents, diverse templates, high-quality OCR, and token-level annotation (Wang et al., 2022).

A central formal distinction concerns what structural supervision is assumed. In traditional VrDU, the input can include OCR-extracted words $W=\{w_l\}_{l=1}^L$, global 1D positions $O=\{o_l\}_{l=1}^L$, word-wise bounding boxes $B=\{b_l\}_{l=1}^L$ in $\mathbb{R}^4$, and manually annotated semantic groups $S'=\{s'_k\}_{k=1}^K$, where each semantic group contains semantically relevant but not obviously grouped words (Jiang et al., 2024). The ReVrDU formulation removes those manual semantic groups and instead restricts inputs to OCR-supplied words, positions, boxes, and text segments $S=\{s_k\}_{k=1}^K$, where each segment is a spatially contiguous line or region (Jiang et al., 2024). That formulation exposes a practical gap: off-the-shelf OCR provides text segments, not oracle semantic grouping.

The broader literature identifies three persistent problem drivers. First, heterogeneous layouts introduce multi-column text, tables, images, charts, headers, and footers with highly variable spatial organization. Second, domain shifts degrade performance when pretrained models move from general corpora to specialized domains such as scientific articles, legal briefs, or handwritten forms. Third, high-quality token- and entity-level annotation remains expensive, especially when manual grouping, entity boxes, and question–answer supervision are required (Ding et al., 2024). A related misconception is that strong performance on small fixed-template datasets implies robust document understanding; benchmark studies show that generalization to unseen templates remains very challenging, and hierarchical fields such as line-items are especially difficult (Wang et al., 2022).

2. Modalities, inputs, and representational primitives

VRDU systems are commonly divided into OCR-dependent and OCR-free frameworks. OCR-dependent systems first apply OCR or PDF parsing to obtain text and bounding-box information, then combine these signals—often with low- or mid-resolution page images—inside a multimodal encoder or an LLM-driven pipeline. Their advantages include mature OCR engines and reduced visual token load, while their disadvantages include OCR error accumulation, limited end-to-end learning, and weaker performance on noisy or handwritten text (Ding et al., 14 Jul 2025). OCR-free systems bypass explicit text extraction, take raw images as input, and rely on a vision encoder plus a language decoder or multimodal LLM to learn text recognition and layout reasoning jointly; these approaches avoid an explicit OCR bottleneck but incur high computational cost and require aggressive visual compression for long sequences (Ding et al., 14 Jul 2025).

Across these settings, representation learning is usually organized around three modalities. Textual features are derived from OCR tokens or higher-level units such as lines, fields, or paragraphs; visual features come from RoI-pooled CNN features or patch embeddings from a vision transformer; layout features are encoded through absolute or relative two-dimensional geometry, typically bounding-box coordinates $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$ or pairwise spatial biases (Ding et al., 2024). In encoder-style fusion, an early-fusion form is often written as

$h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$

while the underlying attention computation follows

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$

or a layout-biased variant thereof (Ding et al., 2024, Ding et al., 14 Jul 2025).

Layout is not limited to coordinate embeddings. Some systems verbalize layout as prompt content by inserting bounding boxes into textual input; others integrate layout through self-supervised tasks such as block masking, layout-guided infilling, table reconstruction, or visual grounding (Ding et al., 14 Jul 2025). Reading order is a particularly important latent variable. ERNIE-Layout argues that simple raster-scan serialization often violates human reading order on multi-column text, tables, and forms, and therefore introduces reading-order-aware serialization and a Reading Order Prediction objective (Peng et al., 2022). XYLayoutLM makes a similar claim, replacing noisy OCR token order with Augmented XY Cut and adding Dilated Conditional Position Encoding to capture local one-dimensional and two-dimensional neighborhood structure (Gu et al., 2022).

There is also a distinct branch for born-digital documents with markup structure rather than fixed raster layout. MarkupLM represents each token as

$$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$

where XPath embeddings encode DOM ancestry and sibling structure in HTML or XML (Li et al., 2021). This branch addresses documents whose layout is dynamically rendered rather than fixed in a scanned page.

3. Model families and pre-training paradigms

A major line of work uses encoder-only multimodal Transformers that jointly model text, layout, and image. LayoutLMv2 employs a two-stream multimodal Transformer with spatial-aware self-attention and three pre-training objectives: Masked Visual-Language Modeling, Text-Image Alignment, and Text-Image Matching (Xu et al., 2020). On standard VrDU benchmarks it reports improvements from LayoutLM to LayoutLMv2 on FUNSD $O=\{o_l\}_{l=1}^L$0, CORD $O=\{o_l\}_{l=1}^L$1, SROIE $O=\{o_l\}_{l=1}^L$2, Kleister-NDA $O=\{o_l\}_{l=1}^L$3, RVL-CDIP $O=\{o_l\}_{l=1}^L$4, and DocVQA $O=\{o_l\}_{l=1}^L$5 (Xu et al., 2020). Bi-VLDoc retains the multimodal Transformer setting but shifts emphasis to bidirectional vision-language supervision through hybrid-attention and auxiliary tasks such as Text-Image Position Awareness, Region-Wise Text Prediction, and Bidirectional Text-Image Alignment; it reports gains on FUNSD from $O=\{o_l\}_{l=1}^L$6 to $O=\{o_l\}_{l=1}^L$7, on CORD from $O=\{o_l\}_{l=1}^L$8 to $O=\{o_l\}_{l=1}^L$9, on RVL-CDIP from $B=\{b_l\}_{l=1}^L$0 to $B=\{b_l\}_{l=1}^L$1, and on DocVQA from $B=\{b_l\}_{l=1}^L$2 to $B=\{b_l\}_{l=1}^L$3 for previous single-model methods (Luo et al., 2022).

Another line targets layout structure more explicitly. ERNIE-Layout combines reading-order-aware serialization, Reading Order Prediction, Replaced Region Prediction, and Spatial-Aware Disentangled Attention; it reports entity F1 of $B=\{b_l\}_{l=1}^L$4 on FUNSD, $B=\{b_l\}_{l=1}^L$5 on CORD, $B=\{b_l\}_{l=1}^L$6 on SROIE, and $B=\{b_l\}_{l=1}^L$7 on Kleister-NDA, as well as RVL-CDIP accuracy $B=\{b_l\}_{l=1}^L$8 (Peng et al., 2022). GraphLayoutLM augments LayoutLMv3 with a document layout structure graph, graph reordering, and layout-aware self-attention, reaching FUNSD-Base $B=\{b_l\}_{l=1}^L$9 versus LayoutLMv3 $\mathbb{R}^4$0, CORD-Base $\mathbb{R}^4$1 versus $\mathbb{R}^4$2, and XFUND-Chinese-Base $\mathbb{R}^4$3 versus $\mathbb{R}^4$4 (Li et al., 2023). XYLayoutLM instead uses Augmented XY Cut and DCPE to improve multilingual semantic entity recognition and relation extraction over LayoutXLM, with XFUN averages of $\mathbb{R}^4$5 for SER and $\mathbb{R}^4$6 for RE (Gu et al., 2022).

Realistic OCR settings motivate a further reformulation. ReLayout introduces ReVrDU and replaces dependence on manually annotated semantic groups with three pre-training heads: MLM, 1D Local Order Prediction, and 2D Text Segment Clustering (Jiang et al., 2024). Its combined loss is

$\mathbb{R}^4$7

with $\mathbb{R}^4$8 and $\mathbb{R}^4$9, and $S'=\{s'_k\}_{k=1}^K$0 applied only in the final epoch (Jiang et al., 2024). In the ReVrDU setting, ReLayout_Base reports FUNSD F1 $S'=\{s'_k\}_{k=1}^K$1, CORD F1 $S'=\{s'_k\}_{k=1}^K$2, and DocVQA ANLS $S'=\{s'_k\}_{k=1}^K$3, outperforming OCR-only LayoutLMv3 and LayoutMASK variants; its ablation study attributes the largest single gain to 1-LOP, with $S'=\{s'_k\}_{k=1}^K$4 F1 on FUNSD-P (Jiang et al., 2024).

End-to-end pixel-based models provide an alternative to OCR dependence. DUBLIN is a seq2seq system with roughly $S'=\{s'_k\}_{k=1}^K$5 million trainable parameters, pre-trained with Masked Document Text Generation, Bounding Box, and Rendered Question Answering tasks on rendered webpages and QA corpora (Aggarwal et al., 2023). It reports DocVQA ANLS $S'=\{s'_k\}_{k=1}^K$6, CORD F1 $S'=\{s'_k\}_{k=1}^K$7, WebSRC $S'=\{s'_k\}_{k=1}^K$8 EM/F1, and strong results on infographics, AI2D, OCR-VQA, and table QA (Aggarwal et al., 2023). MarkupLM occupies a different niche: it is render-free, HTML-native, and uses token-, node-, and page-level objectives over text plus DOM structure, reporting WebSRC dev EM/F1/POS $S'=\{s'_k\}_{k=1}^K$9 for MarkupLM_BASE and $S=\{s_k\}_{k=1}^K$0 for MarkupLM_LARGE (Li et al., 2021).

4. Benchmarks, tasks, and evaluation

The dedicated VRDU benchmark introduced in 2022 was designed to address overly simple schemas, minimal layout complexity, homogeneous templates, OCR inconsistency, and the lack of token-level ground truth in earlier document information extraction datasets (Wang et al., 2022). It contains two datasets drawn from U.S. government filings: Registration Forms with $S=\{s_k\}_{k=1}^K$1 scanned PDF pages and Ad-Buy Forms with $S=\{s_k\}_{k=1}^K$2 scanned pages (Wang et al., 2022). The benchmark defines three settings—Single Template Learning, Mixed Template Learning, and Unseen Template Learning—with a fixed test set of $S=\{s_k\}_{k=1}^K$3 documents and training sizes of $S=\{s_k\}_{k=1}^K$4, $S=\{s_k\}_{k=1}^K$5, $S=\{s_k\}_{k=1}^K$6, and $S=\{s_k\}_{k=1}^K$7 (Wang et al., 2022). Its type-aware matching normalizes numeric values, parses dates, and uses punctuation-normalized comparison for free text; hierarchical line-items are matched only if all components align in order (Wang et al., 2022).

Widely used downstream benchmarks remain central to comparative evaluation. FUNSD and CORD are standard for token- or entity-level extraction with F1; DocVQA is standard for extractive QA with ANLS; RVL-CDIP is standard for document classification with accuracy; Form-NLU has become important for entity retrieval and multi-format form understanding; and multilingual variants such as XFUND probe language transfer (Xu et al., 2020, Ding et al., 2024). More recent benchmarks substantially broaden the stress conditions. MosaicDoc contains $S=\{s_k\}_{k=1}^K$8 pages from newspapers and magazines in English and Chinese, over $S=\{s_k\}_{k=1}^K$9 QA pairs, $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$0 publishers, and annotations for OCR, DocVQA, reading order, and localization, explicitly targeting dense multi-column and non-Manhattan layouts (Chen et al., 13 Nov 2025).

Benchmark or dataset	Document/task profile	Primary metrics
VRDU benchmark	Registration Forms and Ad-Buy Forms; STL, MTL, UTL	micro-F$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$1, macro-F$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$2
FUNSD / CORD / SROIE	form and receipt key information extraction	entity- or field-level F1
DocVQA	document visual question answering	ANLS
RVL-CDIP	document image classification	accuracy
Form-NLU / VRD-IU	entity retrieval and end-to-end localization on multi-format forms	F1, mAP
MosaicDoc	OCR, DocVQA, reading order, content-aware localization	ANLSL, CRR, OCRR, F1, IoU-based scores

Metric design in VRDU reflects task heterogeneity. F1 remains standard for extraction and relation tasks; ANLS and ANLSL are used for document QA; mAP appears in localization settings; and OCR or grounding tasks use character recognition or IoU-based measures (Wang et al., 2022, Chen et al., 13 Nov 2025). Benchmark evidence repeatedly shows that template generalization is unresolved: in the VRDU benchmark, moving from mixed-template to unseen-template evaluation costs roughly $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$3–$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$4 points in micro-F$(x_{\min}, y_{\min}, x_{\max}, y_{\max})$5 on Registration Forms at $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$6 training pages, while line-item extraction remains near zero in very low-data regimes (Wang et al., 2022). MosaicDoc extends this diagnosis to modern VLMs, reporting that all models break on multi-span reasoning, dense layouts, and cross-column reading, with paragraph-level reading order far weaker than line-level ordering (Chen et al., 13 Nov 2025).

5. Robustness, domain adaptation, and deployment constraints

Real-world VRDU is constrained by annotation scarcity, OCR noise, handwriting, and domain-specific vocabularies. DAViD addresses this through synthetic annotation generation plus joint-grained representation learning over fine-grained tokens and coarse-grained entities (Ding et al., 2024). Its domain adaptation stage applies Structural Domain Shifting, Synthetic Sequence Tagging, and Synthetic Instructed Tuning; on CORD with only a small guidance set, the joint-grained backbone achieves $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$7 F1 versus $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$8 for UDOP, and adding L2V, SDS, and SST raises F1 to $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$9 (Ding et al., 2024). On printed Form-NLU, entity-level accuracy rises from $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$0 with LXMERT to $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$1 with SDS+SIT, and on handwritten forms from $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$2 to $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$3 (Ding et al., 2024).

Few-shot and efficiency-oriented adaptation are addressed by FS-DAG, a modular graph model with $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$4 million parameters and a frozen language backbone plus a light visual encoder and two GNN layers (Agarwal et al., 22 May 2025). In a 5-shot regime it reports average F1 $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$5 without OCR errors and $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$6 with OCR errors, a drop of $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$7, whereas LayoutLMv3 drops from $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$8 to $h_i = W_f\bigl[x_i^T; v_i; pos_i\bigr] + b_f,$9 (Agarwal et al., 22 May 2025). On WildReceipt, FS-DAG reaches $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$0 versus LayoutLMv3 $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$1 and LayoutLMv2 $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$2, while inference latency is $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$3 ms/document versus $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$4 s for LayoutLMv3 (Agarwal et al., 22 May 2025).

Handwriting and localization remain a separate deployment difficulty. A document-specific augmentation pipeline for VRDIU Track B creates pseudo-handwritten variants with InkBleed, Letterpress, low-ink lines, JPEG artifacts, DirtyScreen, and mild rotation, then trains Faster R-CNN or Mask R-CNN on the union of original and augmented images (Dai, 10 Feb 2025). The reported effect is higher handwritten mAP for several backbones, including ResNet-101 + Faster R-CNN $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$5, DiT + Mask R-CNN $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$6, and LayoutLMv3 + Mask R-CNN $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$7, with small digital-set drops (Dai, 10 Feb 2025). In the VRD-IU competition itself, Track A entity-based retrieval reaches $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$8 public and private F1 for the top team, whereas Track B end-to-end localization peaks at $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Bigl(\frac{QK^\top}{\sqrt{d_k}}\Bigr)V$$9 public mAP but only $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$0 private mAP, indicating that localization is materially harder under realistic shift (Ding et al., 2 Jun 2025).

Privacy-sensitive and low-resource domains motivate retrieval-guided synthetic supervision. Docs2Synth processes raw scans, generates and verifies synthetic QA pairs, tunes a lightweight visual retriever over OCR entities, and then uses an iterative retrieval–generation loop with an MLLM (Ding et al., 18 Jan 2026). It reports retriever top-1 recall above $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$1 on Form-NLU, CORD, and Ephoie, and end-to-end gains of $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$2 points on printed Form-NLU, $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$3 on CORD, and $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$4 on Ephoie over the best zero-shot MLLM baselines, while reducing hallucination errors by over $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$5 in a manual audit of $$E_i = E_i^{text} + E_i^{pos1D} + E_i^{seg} + E_i^{xpath},$$6 samples (Ding et al., 18 Jan 2026). This suggests that grounded retrieval has become a practical complement to generative VRDU rather than a competing paradigm.

6. Open problems and emerging directions

Several unresolved issues now organize the research frontier. The first is robustness to real-world structure. ReVrDU shows that methods depending on perfect semantic grouping can deteriorate sharply when only OCR segments are available, while the original VRDU benchmark shows that unseen-template generalization and hierarchical field extraction remain difficult even for strong multimodal models (Jiang et al., 2024, Wang et al., 2022). A common misconception is therefore that high scores on FUNSD or CORD alone establish broad document competence.

The second is the OCR-dependent versus OCR-free trade-off. OCR-dependent pipelines benefit from mature OCR and lower visual token counts, but they inherit OCR errors and two-stage failure cascades; OCR-free systems can learn recognition and reasoning jointly, but they require high-resolution encoding, compression, and large-scale pretraining (Ding et al., 14 Jul 2025). DUBLIN, Donut-style models, and newer MLLM pipelines indicate that OCR-free modeling is viable, yet ReLayout and related work argue that realistic OCR settings still deserve dedicated modeling rather than being treated as a solved preprocessing step (Aggarwal et al., 2023, Jiang et al., 2024).

The third is long-document and dense-layout reasoning. QA surveys identify cross-page reasoning, sparse global-local attention, retrieval-augmented generation, and recurrent memory as active directions for multi-page inputs (Barboule et al., 4 Jan 2025). MosaicDoc adds evidence that multi-span QA, paragraph-level reading order, and localization in dense newspapers remain weak for current VLMs, even when line-level ordering is relatively strong (Chen et al., 13 Nov 2025). This suggests that future progress depends not only on larger backbones but also on explicit modeling of reading sequences, block relations, and answer grounding.

Recent surveys on MLLM-based VRDU emphasize several converging directions: modular, agent-based architectures; efficient adaptation via LoRA, projectors, visual token pruning, and correlation-guided compressors; enhanced layout reasoning through disentangled spatial attention, Group Positional Embeddings, and Layout-CoT; high-fidelity synthetic data with LLM-based validation and human-in-the-loop correction; retrieval-augmented and tool-augmented generation; and domain-specific scaling analyses (Ding et al., 14 Jul 2025). Parallel work on markup-native models, joint-grained distillation, and relation extraction suggests that VRDU is broadening from page-level field tagging into a more general document intelligence problem encompassing DOM structure, hierarchical grouping, graph relations, and evidence-grounded reasoning (Li et al., 2021, Ding et al., 2024, Adnan et al., 2024).