---
title: Hierarchical PDF Segmentation
url: https://www.emergentmind.com/topics/hierarchical-pdf-segmentation
type: topic
---

# Hierarchical PDF Segmentation

Hierarchical PDF segmentation is the automated extraction of the nested, multi-level logical structure of documents encoded in the PDF format, producing a representation where document elements (e.g., sections, paragraphs, tables, fields) are organized into a hierarchy that reflects their containment and reading order relations. This task encompasses segmentation of both visually complex scanned documents and digitally typeset files, requiring systems to operate with minimal external metadata to infer rich, arbitrarily deep structural trees. Solutions combine semantic segmentation, instance detection, layout analysis, and relation modeling, leveraging both classic computer vision and state-of-the-art machine learning techniques, including CNNs, instance-segmentation backbones, weak supervision pipelines, and, most recently, LLM-enabled structure inference.

## 1. Hierarchical Structure Representations

All hierarchical PDF segmentation approaches output representations that encode both entities and their relations in a tree or tree-like structure. DocParser [1911.01702] formalizes each document page as a pair $T = (E, R)$, where $E = \{E_j\}$ is the set of detected entities, each with a semantic type $c_j$, bounding box $B_j$, and confidence $P_j$, and $R = \{ (E_i, E_j, \Psi_k ) \}$ is a set of typed binary relations. The core structural relations include $\mathit{parent\_of}$, signifying containment/nesting, and $\mathit{followed\_by}$, signifying reading order among siblings. The hierarchy is thus an ordered rooted tree, constrained so every non-top-level entity has one parent and siblings are totally ordered.

In form-centric pipelines such as that of Sarkar et al. [1911.12170], document structure is decoded into four explicit levels—TextRuns/Widgets (leaves), TextBlocks/ChoiceGroup Titles (blocks), TextFields/ChoiceFields (fields), and ChoiceGroups (containers)—each corresponding to a CNN segmentation head. The hierarchy is made explicit during annotation via per-instance instance masks and containment links, and is enforced in the model via multi-head supervision.

In textbook and legal document contexts [2509.00909], the output is a nested segmentation over text spans, with hierarchy levels tied to sectioning structure (e.g., chapters, sections, subsections), and boundaries inferred from headings detected via TOC, layout, or LLMs.

## 2. Algorithmic Approaches

The three dominant families of algorithms for hierarchical PDF segmentation are:

| Approach                          | Key Characteristics                                           | Exemplars                        |
|------------------------------------|--------------------------------------------------------------|----------------------------------|
| Hierarchical Semantic Segmentation | Strip-wise, high-res, multi-level semantic masks, CNN-based  | Sarkar et al. [1911.12170]       |
| Instance Segmentation + Heuristics | Per-entity detection (boxes/masks) + rule-based tree assembly| DocParser [1911.01702]           |
| Headline Detection & Parsing       | Heading detectors via TOC, layout, or LLM, then tree induction| HiPS [2509.00909]                |

**Hierarchical Semantic Segmentation:** Sarkar et al. employ a deep CNN operating on high-resolution overlapping image strips, where the segmentation mask of one strip acts as a prior for the next. The model jointly outputs semantic segmentation masks for each hierarchy level, enforcing level consistency and enabling fine-grained compositional parsing [1911.12170]. This strategy addresses challenges of continuity for structures spanning multiple image tiles and resolves both visual segmentation and containment simultaneously.

**Instance Segmentation & Heuristics:** DocParser detects all entities (tables, figures, headings, etc.) as instances using a Mask R-CNN backbone, then applies a grammar-aware cascade of deterministic heuristic rules to infer nesting (parent–child) and sibling order relations based on geometric containment, domain grammars, and layout [1911.01702]. Weak supervision is leveraged during training using noisy annotations from reverse-LaTeX rendering, ensuring scalability even with limited ground-truth data.

**Section Headline Detection & Parsing:** The HiPS system [2509.00909] approaches hierarchical segmentation for complex books by first detecting section titles via TOC extraction, document layout parsing, or LLM-based refinement (sometimes with OCR features). It assigns hierarchy levels to each detected heading and infers section boundaries by matching headings to text locations via normalized, substring, or fuzzy criteria. Tree structure is then induced by the heading sequence and assigned levels.

## 3. Model Architectures and System Design

**Strip-based Hierarchical Segmentation ([1911.12170]):**

- **Input:** Overlapping horizontal strips of grayscale image $S_h \times w$, concatenated with binary prior mask ($C$ channels).
- **Image Encoder:** Stacked 2D convolutions, ReLU activations, and downsampling. Multiple skip connections allow fine detail recovery.
- **Context Encoder:** Four blocks of bi-directional 1D dilated convolutions applied vertically and horizontally, capturing long-range dependencies critical for hierarchical parsing of forms.
- **Decoder:** U-Net–style upsampling with skip connections. Each upsampled representation passes through $K$ independent 1x1 conv "heads" yielding a $C_k$-class softmax mask per hierarchy level.
- **Prior Propagation:** At inference, the predicted mask’s bottom overlap region is used as the prior for the next strip, maintaining structure continuity across segments.
- **Loss:** Multi-head cross-entropy, optionally augmented by prior-consistency MSE across overlap; empirically, supervision with cross-entropy suffices ($\lambda_p = 0$).
- **Post-processing:** Convex hull smoothing for instance crispness; DOM-tree assembly via containment of polygons.

**DocParser Instance Approach ([1911.01702]):**

- **Backbone:** ResNet-110 with Feature Pyramid Network (FPN).
- **Entity Detection:** Mask R-CNN paradigm, producing bounding boxes, classes, and masks.
- **Hierarchy Rules:** No learned relation module—instead, four levels of containment/orientation heuristics (geometric overlap, domain grammar, directness filtering, max-IoU parent selection) yield $parent\_of$ and $followed\_by$ links.
- **Supervision:** Loss sums classification, box regression, and mask segmentation; used for both strong and weak supervision.
- **Weak Supervision:** LaTeX SyncTeX–derived boxes as noisy labels.

**Headline-Driven Segmentation (HiPS, [2509.00909]):**

- **TOC-Based:** Extracts hierarchy directly from PDF outline metadata; matching to text by normalized/fuzzy string search.
- **Layout-Based:** Font-size, spacing, and margin heuristics (e.g., PDFstructure, Marker).
- **LLM-Based:** XML and OCR-based candidate lines are filtered and ranked by a prompting strategy using instruction-tuned LLMs (GPT-5, Llama 3), assigning level and purging non-headings.
- **Section Boundary Detection:** Algorithmic scan, creating segments upon matching heading; level assignment either direct from TOC or via LLM.

## 4. Datasets and Evaluation Protocols

**Document Forms Dataset ([1911.12170]):**

- **Scale:** 52,490 human-annotated forms, ∼1800×1000px.
- **Annotations:** Hierarchical mask and bounding box labels for TextRuns, Widgets, TextBlocks, ChoiceGroups, etc.; explicit containment relations.

**DocParser Datasets ([1911.01702]):**

- **\datasetmanual/:** 362 arXiv papers (up to 30 pages each), 23 entity categories, complete nest/order annotation.
- **\datasetauto/:** 127,472 arXiv papers, 14 category weak labels from SyncTeX.

**HiPS Evaluation ([2509.00909]):**

- Application to legal textbooks and complex academic books; public data and code.

**Metrics Used:**

| Metric                         | Formalization/Reference                        | Context                              |
|--------------------------------|------------------------------------------------|--------------------------------------|
| Pixel mean Intersection-over-Union (MIoU) | $\text{MIoU} = \frac{1}{C}\sum_{c=1}^C \frac{\mathrm{TP}_c}{\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c}$ | Semantic segmentation [1911.12170]   |
| Object-level precision/recall  |  $\mathrm{TP}$ at $\text{IoU} \geq \tau$      | Instance mask match [1911.12170]     |
| mAP (IoU=0.5, 0.65)            | mean Average Precision                        | Entity detection [1911.01702]        |
| Relation F1                    | Triple matching on $(E_i, E_j, \Psi)$         | Relation structure [1911.01702]      |
| Tree Edit Distance             | $d_{\mathrm{ETD}}(T_{\text{pred}}, T_{\text{gt}})$ | Hierarchy match [2509.00909]         |
| $P_k$, WindowDiff              | Beeferman/Pevzner–Hearst segmentation metrics | Section boundary detection [2509.00909]|
| Edit-distance relaxed $P_{ED}, R_{ED}$ | Tolerant match allowing $ED \le 2$           | Title detection [2509.00909]         |

## 5. Quantitative Performance and Comparative Results

State-of-the-art results are reported by each system in its target domain.

- **Sarkar et al.** ([1911.12170]): On forms, Highresnet yields MIoU of 92.7% (TextRun), 83.0% (ChoiceGroup), substantially outperforming DeepLabV3+ and MFCN baselines. Object-level F1 for TextRuns improves with increased model complexity (Lowres→Highres: 62.6→73.2). On generalization sets, table MIoU and F1 match or surpass literature benchmarks.
- **DocParser** ([1911.01702]): Weak-supervision improves mAP from 49.9% (no WS) to 69.4% (full WS+fine-tune), and relation F1 from 0.453 to 0.615 (+35.8%). On ICDAR-2013, DocParser reaches F1=0.9292 (prior art: 0.9144).
- **HiPS** ([2509.00909]): TOC-based has precision $P_{ED} > 0.98$, recall $R_{ED}$ 0.6–0.95 depending on metadata; XML+OCR+GPT-5 achieves $P_{ED} \approx 0.95$, $R_{ED} \approx 0.90$, significantly outperforming structure-based tools on deep hierarchies and boundary accuracy as measured by $P_k$, WindowDiff, and tree edit distance.

## 6. Practical Implementation and Deployment Considerations

**Preprocessing** steps crucially impact performance. For semantic segmentation, precise rasterization to high DPI and zero-padding/cropping ensure uniform input dimensions ([1911.12170]). Strip slicing (e.g., $S_h = 600, O_h = 200$) with overlap enables globally consistent segmentation. In heading-driven systems, XML layout features and OCR on rendered pages substantially reduce false positives and enhance boundary/level assignment ([2509.00909]).

**Postprocessing** includes instance mask smoothing (convex hulls), DOM-tree assembly from masks or entities, and emission of HTML or TEI for downstream use ([1911.12170], [2509.00909]).

**System selection** depends on document properties. Where detailed TOC metadata is available and trustworthy, TOC-based headline parsing offers unmatched speed and depth. For complex, poorly structured, or handwritten/scanned pages, LLM-refined methods with signal from both OCR and PDF structure provide state-of-the-art hierarchical extraction at increased computational expense [2509.00909].

## 7. Strengths, Limitations, and Recommendations

| Approach                | Strengths                                                               | Limitations                                                |
|-------------------------|------------------------------------------------------------------------|------------------------------------------------------------|
| TOC-Based PageParser    | Near-perfect precision, supports deep hierarchies when metadata is full | Recall zero for missing headings, brittle to TOC errors    |
| Structural Parsers      | Fast, metadata-independent, strong for 2–3 levels                      | Sharp degradation on deeper or non-academic layouts        |
| LLM-Refined PageParser  | Discovers missing headings, deep and flexible hierarchy, high recall    | Higher compute, prompt/LLM dependence, risk of hallucination |

For large-scale document ingestion, structural parsers like Marker or PDFstructure suffice for shallow segmentation. For accuracy-critical or deep hierarchical extraction, especially with incomplete metadata, structure-aware LLM-based refinement is currently preferred ([2509.00909]) with OCR and XML features. High-resolution semantic segmentation (as in [1911.12170]) delivers robust results for visually-structured forms and scanned documents.

## References

- "Document Structure Extraction using Prior based High Resolution Hierarchical Semantic Segmentation" [1911.12170]
- "DocParser: Hierarchical Structure Parsing of Document Renderings" [1911.01702]
- "HiPS: Hierarchical PDF Segmentation of Textbooks" [2509.00909]

Source: https://www.emergentmind.com/topics/hierarchical-pdf-segmentation