---
title: Format Graph Construction
url: https://www.emergentmind.com/topics/format-graph-construction
type: topic
---

# Format Graph Construction

Format graph construction refers to the class of algorithms and system architectures that recover the full hierarchical, document-level structure (typically represented as a tree or forest) from raw, semi-structured, or unstructured digital formats such as PDF, HTML, or document images. The resulting format graph (more commonly called a document tree or DocTree in recent literature) encodes not only the reading sequence but also the parent–child and left–right ordering relations that organize headings, paragraphs, lists, tables, figures, and other semantic units. Format graph construction underpins downstream document understanding, passage retrieval, information extraction, and reflow for generative AI models.

## 1. Formal Definitions and Core Taxonomy

In contemporary research, format graph construction is equivalently termed DocTree, Document Structured Extraction (DSE), hierarchical document structure analysis (HDSA), or Table of Contents (ToC) extraction [2401.11874][2409.05137][2212.02896]. The canonical output is a rooted, ordered tree \( T = (V, E) \):

- **Nodes** (\(V\)): Semantic units such as headings of various levels, paragraphs or text blocks, tables, figures, lists, formulas, and code blocks.
- **Edges** (\(E\)): Directed parent–child relations, encoding nesting (e.g., section–subsection), and sibling order, plus sometimes supplementary or layout-specific edges for headers, footers, and sidebars.

In formal terms, given an input document \( D \) (e.g., raw PDF, HTML, or OCR fragment set), format graph construction is learning a function \( f : D \to T \), where \(T\) captures both the correct reading order and the hierarchical, nested structure of the document [2409.05137][2401.11874][2602.23061].

Node type taxonomies typically include five core categories: Heading, PlainText, Formula, Table, CodeBlock [2409.05137], sometimes extended to figures, images, lists, captions, footers, and supplementary blocks [2602.23061][2401.11874][2212.02896][1911.01702].

## 2. Methodological Paradigms

Three principal paradigms dominate format graph construction:

### a. Rule-based and Heuristic Tree Construction

Methods such as those in [1210.6113][2111.12317][2310.18073][2603.00122] construct initial trees using geometric and heuristics:

- **HTML DOM-based:** Model the document as a DOM tree \( (N, E) \), compute per-node statistics (e.g., chars-nodes ratio, CNR), prune nodes (e.g., scripts, styles), and aggregate high-CNR nodes into blocks via lowest-common-ancestor merging [1210.6113].
- **Reading-Order and Font-size Trees:** Text blocks are sorted in reading order (left-to-right, top-to-bottom); each block is made a child of the nearest prior block with larger font size, building an initial tree (CMM, [2310.18073]).
- **Spatial Clustering/Assignment:** Page elements are assigned to layout regions (columns, multi-column, row groups) by geometric clustering, then grouped into segments or subtrees (NovaLAD, [2603.00122]).

These approaches rely on hand-crafted rules (e.g., for parent assignment or block merging), spatial/visual cues (font size, bounding-box position), and limited statistical thresholds.

### b. Sequential or Pairwise Insertion with Learned Models

Frameworks such as HELD [2105.09297] and tree-decoder architectures [2212.02896][2401.11874] treat tree construction as a sequence of insertion or link-prediction decisions:

- **Sequential Insertion (HELD):** Given a list of detected objects, each is inserted at the optimal position among the current tree’s legal candidate slots (down the rightmost path), using a “put-or-skip” binary classifier (e.g., BiLSTM-based). Traversal and candidate scoring strategies control trade-offs between inference quality and computational cost.
- **Pairwise Link Prediction:** For each candidate node, predict its parent and/or sibling via neural scoring functions, often using multimodal encoders (textual, layout, vision features) and dependency parsing analogs [2401.11874][2212.02896].
- **Tree-Decoder Models:** Heading entities are encoded multimodally, then a sequential decoder with (hard/soft) attention predicts parent/sibling/identity links, building the tree in document order [2212.02896].

### c. Hybrid and Multimodal Methods

Recent systems leverage deep multimodal fusion:

- **Multimodal Feature Encoding:** Jointly encode semantic (BERT or similar), layout (2D geometry, font), and visual (CNN) representations for each region or OCR fragment, often with gated fusion [2010.11685][2212.02896].
- **Graph Neural Networks:** Local node-centric subtrees are modeled with BiGRU (for reading order context) and Graph Attention Networks (GATs) to integrate local and long-range context, supporting scalable refinement actions (keep/delete/move) [2310.18073].
- **Relation Prediction and Dependency Parsing:** Tree construction formulated as parallel parent/sibling prediction (Construct phase) with cross-entropy losses over possible edges, often optimized end-to-end [2401.11874].

Many frameworks partition the task into modular phases: object detection, classification (heading detection), reading order, and hierarchical tree construction. Single-pass or pipeline approaches dominate, with downstream coreference and entity linking sometimes included.

## 3. Evaluation Protocols, Benchmarks, and Metrics

Evaluation of format graph construction emphasizes both structural and content correctness:

- **Tree Edit Distance Similarity (TEDS):**
  \[
  \text{TEDS}(T_1, T_2) = 1 - \frac{\text{TED}(T_1,T_2)}{\max(|T_1|,|T_2|)}
  \]
  where TED is the tree-edit distance, and \(|T|\) is the number of nodes [2212.02896][2409.05137][2401.11874][2310.18073].
- **Edit Distance Similarity (EDS):** Normalized Levenshtein distance over text spans or tables [2409.05137].
- **Kendall’s Tau Distance Similarity (KTDS):** Measures reading-order or sequence correlation between predicted and gold [2409.05137].
- **Vocabulary-level F1:** Word-level overlap of predicted and gold text tokens [2409.05137].
- **Downstream Metrics:** Passage retrieval accuracy (mAP, recall@k), entity detection mAP (COCO-style), hierarchical-reconstruction micro/macro STEDS [2401.11874].

Public benchmarks include READoc (PDF→Markdown, 2,233 documents) [2409.05137], Comp-HRDoc (1,500 documents, complex layouts, hierarchical ToC, [2401.11874]), HierDoc (650 arXiv-style papers, [2212.02896]), ESGDoc (annual reports, 1,093 docs, [2310.18073]), and DP-Bench (structured JSON, Markdown, NID, and TEDS metrics, [2603.00122]).

## 4. Scalability and Complexity Analysis

Different approaches yield distinct computational and memory profiles:

- **Rule-based methods** (e.g., font-size trees, spatial clustering) typically run in near \( O(N) \) time, where \( N \) is the number of layout elements [2310.18073][1210.6113][2603.00122]. The CMM pipeline is designed for \( O(N) \) document-level complexity by constraining node-centric subtrees.
- **Pairwise or exhaustive models** (e.g., MTD, pairwise insertion) require \(O(N^2)\) runtime and memory for all possible parent–child pairs, which can be prohibitive for multi-hundred-page documents [2212.02896][2310.18073]. CMM and similar strategies avoid this via local subtree extraction.
- **End-to-end deep models** (Detect-Order-Construct, [2401.11874]) parallelize detection, ordering, and construction; practical GPU-based training handles thousands of documents with up to 50 pages efficiently.
- **CPU-optimized pipelines** (NovaLAD) exploit thread pools for parallel YOLO detection, image classification, and layout grouping, maintaining inference time per document (e.g., 8.5s for DP-Bench) [2603.00122].

Most modern systems incorporate scalable data structures (e.g., R-trees for candidate lookup), early pruning strategies, and model ablation to balance accuracy with throughput on large-scale corpora.

## 5. Empirical Findings and Comparative Results

Rigorous comparisons on public and proprietary benchmarks demonstrate:

- **MTD achieves TEDS=87.2% and F1=88.1% on scientific ToC extraction (HierDoc),** and heading-detection F1 of 96.1% [2212.02896].
- **CMM yields 88.1% TEDS on HierDoc, but greatly outperforms MTD on ESGDoc full-length reports (TEDS 33.2% vs. 12.8%) and scales efficiently to 500+ page documents** [2310.18073].
- **HELD sets new state-of-the-art on variable-depth logical hierarchy extraction, with accuracies of 97.3%, 73.0%, 95.8% (Chinese, English financial, arXiv)** [2105.09297].
- **Detect-Order-Construct achieves micro-STEDS up to 86.05% on Comp-HRDoc and 95.04% on simpler HRDoc instances**, outperforming prior pipelines and jointly optimizing detection, reading order, and hierarchy [2401.11874].
- **NovaLAD attains 96.49% TEDS and 98.51% NID on DP-Bench**, surpassing both commercial and open-source baselines in CPU-only settings [2603.00122].
- **Weak supervision, multimodal fusion, and bottom-up structure refinement substantially improve relation parsing F1 and detection (DocParser, DocStruct)** [1911.01702][2010.11685].
- **Ablation studies show multimodal fusion and graph-based context integration yield large TEDS and F1 gains over pure text or layout features** [2212.02896][2310.18073][2010.11685].

A summary of TEDS scores from selected systems is provided below:

| System                | Dataset        | TEDS (%)        |
|-----------------------|---------------|-----------------|
| MTD                   | HierDoc       | 87.2            |
| CMM                   | HierDoc       | 88.1            |
| CMM                   | ESGDoc (full) | 33.2            |
| HELD                  | arXiv         | 95.8            |
| DOC-ORDER-CON         | Comp-HRDoc    | 86.1            |
| NovaLAD               | DP-Bench      | 96.5            |

## 6. Advanced Topics and Limitations

Despite significant progress, several open problems remain:

- **Cross-Page Hierarchy**: Most systems process pages in isolation; few treat the document as a single structured object, hindering robust parent–child linking across page boundaries [2409.05137].
- **Multi-Modality & Visual Semantics**: Highly variable table, formula, float, and sidebar layouts demand tight integration of vision, layout, and text modules.
- **Deep Layout Complexity**: Multi-column, float-heavy, or highly decorative formats cause degradation in recall and tree similarity, especially as ToC depth increases [2409.05137].
- **Error Propagation**: Page object detection and reading-order prediction errors propagate to hierarchy construction; robust end-to-end joint models remain scarce [2401.11874].
- **Non-Tree Structures**: Directed acyclic graph conventions (e.g., cross-reference links, multi-parent nodes) are not supported; most systems assume strict tree structure.
- **Resource Efficiency**: Vision-language model (VLM) approaches have high memory/compute costs; CPU-optimized pipelines (NovaLAD) demonstrate speed but may underperform on ambiguous structure detection.

Unified loss functions (e.g., differentiable tree-edit distance objectives) and graph-structured logic representations (beyond trees) are targets for future research [2409.05137][2401.11874].

## 7. Applications and Impact

Format graph construction enables a suite of downstream tasks:

- **Passage-level Retrieval & Information Extraction**: Accurate tree paths and passage metadata enable substantial mAP and recall@k improvements in passage retrieval, supporting semantic search and QA [2105.09297][2409.05137].
- **Summarization and Generative AI**: Hierarchical segmentation and node metadata allow efficient chunking for large language models and retrieval-augmented generation [2603.00122].
- **Knowledge Base Construction**: Structured, layout- and semantics-aware graphs can be directly converted to knowledge graphs, JSON, Markdown, and other structured outputs [2603.00122][2409.05137].
- **Small-Screen, Accessibility, and Reflow**: Accurate region- and block-level hierarchy extraction is central to document reflow for small-screen or accessible reading [1210.6113].
- **Document Understanding in Finance, Government, and Enterprise**: Organization chart extraction, form understanding, and legal/financial directory parsing (e.g., directory blocks, key–value relations) depend on robust tree-structured recovery [2010.11685][2111.12317].

Recent unified benchmarks (READoc, Comp-HRDoc, DP-Bench) and code releases (NovaLAD, MoDora, MTD) catalyze ongoing improvement and foster rigorous standardization across the field [2603.00122][2409.05137][2212.02896][2602.23061].

Source: https://www.emergentmind.com/topics/format-graph-construction