---
title: Diagram Parse Graphs (DPGs)
url: https://www.emergentmind.com/topics/diagram-parse-graphs-dpgs
type: topic
---

# Diagram Parse Graphs (DPGs)

A Diagram Parse Graph (DPG) is a structured, typed graph representation of the constituent elements and relationships present within a diagram. DPGs serve as an explicit, symbolic abstraction bridging the gap between raw diagram imagery and higher-level reasoning tasks such as question answering, knowledge extraction, or automated theorem proving. By encoding both the entities (e.g., blobs, geometric primitives, text) and their interrelations (e.g., labeling, geometric predicates, causality), DPGs provide a modality-agnostic interface suitable for multi-modal diagram understanding in both generic and domain-specific contexts [1603.07396] [1711.09528] [2205.09947] [2205.09363].

## 1. Formal Definition and Structural Variants

A DPG is formally defined as a (typed, attributed) graph:
$$
G = (V, E)
$$
where $V$ denotes the set of nodes (diagram constituents) and $E$ the set of typed edges (relations among constituents).

**Node Taxonomy:**
- **Generic diagrams:** Nodes correspond to blobs (illustrative objects), text boxes (OCR regions), arrow heads/tails, and domain-specific entities [1603.07396] [1711.09528].
- **Plane geometry diagrams:** Nodes represent geometric primitives (points—including intersection, tangent, endpoint, and independent types; lines—solid, dashed, mixed; circles and arcs) and non-geometric primitives (symbols—perpendicular marks, bars, angle marks; text—labels, measures, etc.) [2205.09947] [2205.09363].

**Edge Taxonomy:**  
Edges are directed and typed, encoding semantic or geometric relations:
- **Generic diagram types:** Labeling (text→blob), linkage (arrow, connector), title/caption association, region labeling, etc. [1603.07396].
- **Plane geometry types:** Incidence (on-line, on-circle), center-of-circle, parallelism, perpendicularity, symbol→geometry (e.g., bar→equal-length), text→geometry (e.g., angle value→angle), text→symbol [2205.09947] [2205.09363].

Each edge $e\in E$ is typed and, when appropriate, may be represented as a tuple $(\text{subj},\,r,\,\text{[obj]})$, supporting relations of arity $>2$ in certain implementations [2205.09947].

Attributes ($A_V$, $A_E$) supplement node and edge sets with class, bounding-box/pixel mask, class confidence, OCR string, and geometric parameters (position, radius, etc.) [1603.07396] [2205.09363].

## 2. Syntactic Parsing and DPG Construction Algorithms

Inferring a DPG from a raw diagram is a structured prediction problem requiring identification of diagram constituents, proposal of candidate relationships, and selection of the optimum relational structure matching the ground truth.

**Generic Diagrams:**
- *Object detection:* SSD-style networks detect constituents, each with class, bounding box, and confidence [1711.09528].
- *Relationship proposal:* Candidate (typically all) pairs or higher-arity groups are generated for edge classification [1603.07396].
- *DPG Construction:*
  - Dynamic Graph Generation Network (DGGN) processes candidate pairs using a gated recurrent unit (GRU) architecture with global context and a dynamic adjacency tensor memory [1711.09528]. Each pair $(i,j)$ is updated iteratively, allowing probabilistic edge formation.
  - DSDP-Net (LSTM-based) consumes a sequence of relationship proposals, maintaining memory across previously accepted or rejected edges and producing a structured parse of viable edges [1603.07396].
- *Training:* Losses integrate object detection and edge prediction (softmax/CE); proposals are matched to ground truth via IoU thresholds and sampling for positive and negative relationships [1711.09528].

**Geometry Diagrams:**
- *Primitive extraction:* Segmentation and detection modules (e.g., instance segmentation for points, lines, circles; FCOS/Mask-RCNN for symbols/text) localize and classify primitives [2205.09947] [2205.09363].
- *Feature extraction:* Primitive-level features include visual (FPN/ROI-aligned), location (coordinate/radius), and class semantic embeddings [2205.09363].
- *Relation parsing:* Edge proposals filtered by geometric priors (e.g., only allow point–line incidence, line–line parallelism/perpendicularity). GNNs apply attention-based message passing, producing relation existence probabilities [2205.09363].
- *Multi-task end-to-end training:* Detection, segmentation, and relational reasoning losses are balanced to improve joint performance and minimize error propagation [2205.09363].

## 3. Edge and Node Typologies Across Domains

The permitted primitive and relation types—and their compositional rules—are domain-dependent.

| Domain                 | Primitives (Nodes)                                                          | Relations (Edges)                             |
|------------------------|------------------------------------------------------------------------------|-----------------------------------------------|
| Generic/Science        | Blob, text box, arrow tail, arrow head                                      | Labeling, intra/inter-object linkage, arrow assignment, region, caption, title [1603.07396] |
| Plane Geometry         | Points (subtypes), lines (subtypes), circles/arcs, symbols (16+), text (6+) | On-line, on-circle, center, parallel, perpendicular, bar, angle, text/label-to-primitive, symbol-to-primitive [2205.09947][2205.09363] |

In geometric DPGs, edge types are curated to align with formal geometric relations, supporting downstream automated reasoning and proposition generation in languages such as the Geometric Description Language (GDL) [2205.09947].

## 4. Annotation, Datasets, and Evaluation

Several major datasets support DPG research, providing annotated diagrams with primitive-level and relation-level ground truth.

**AI2 Diagrams dataset** [1603.07396]:
- 5,000 diagrams (train/val/test split), 118k nodes, 53k relation instances, 15k associated multiple-choice QA pairs. Comprehensive annotation covers blobs, text regions, arrow heads/tails, and ten relation types.

**PGDP5K** [2205.09947]:
- 5,000 plane geometry diagrams, 16 shape types, 5 positional relations, 22 symbol types, 6 text types. Fine-grained primitive and relation annotation at the pixel/instance level. Annotation pipeline combines semi-automatic extraction and manual correction for robust labeling.

**IMP-Geometry3K** (used in benchmarking) [2205.09947]:
- Existing geometry parsing dataset; for comparison, PGDP5K is larger and more finely annotated.

*Metrics:* Precision, recall, F1 (at 15px match for primitives), string-matching for generated propositions. Performance results indicate F1 ≈ 86% for circles, ≈76–77% for points and lines, and overall GDL-proposition generation F1 = 66.07% on PGDP5K—substantially lower than on less challenging testbeds [2205.09947].

**Bottlenecks and Challenges:**
- Long-tailed distributions for rare classes/relations.
- Complex layouts, overlapping or low-contrast primitives.
- Subtle variations in symbol rendering.
- Hidden or ambiguous relationships (e.g., tangent points, multi-arc geometries) [2205.09947].

## 5. Reasoning Applications and Semantic Leveraging

DPGs are foundational for diagram-centric reasoning in both the vision and mathematical domains.

- **QA over science diagrams:** DPG-based attention enables alignment of textual questions/answers with diagram fact strings via embedding similarity, with question–answer pairs mapped to DPG relations [1603.07396].
- **Plane geometry theorem proving:** The explicit, fine-grained nature of geometric DPGs permits conversion to proposition templates (PointLiesOnLine, Perpendicular, etc.). These can be fed into downstream provers or symbolic reasoning modules [2205.09947].
- **Semantic integration:** DPGs unify multi-modal evidence (visual, textual, symbolic) within a coherent symbolic intermediate, facilitating downstream applications and enabling modularity in complex reasoning pipelines [1711.09528].

## 6. Model Families, Implementation Details, and End-to-End Examples

**Generic DPG Parsing:**
- UDPNet: SSD backbone for detection; DGGN (GRU+dynamic memory) for edge prediction; joint loss over detection and relation existence [1711.09528].
- DSDP-Net: Two-layer LSTM parses sequential candidate relations; end-to-end cross-entropy optimization [1603.07396].

**Plane Geometry Diagram Parsing:**
- PGDPNet: MobileNetV2+FPN backbone, FCOS detection for text/symbols, instance segmentation for geometry, EGAT-GNN for relation parsing, multi-branch loss aggregation for robust end-to-end supervision [2205.09363].

**Step-wise Example** (*summarized from [1711.09528])*:  
A pulley diagram is processed by SSD to detect constituent blobs and text. Each node is paired, features are extracted, and DGGN iteratively proposes edge existence, resulting in a DPG where, e.g., (blob@pulley → text@Pulley) is formed if above-threshold, and similarly for other constituent links.

## 7. Impact, Limitations, and Future Directions

DPGs formalize diagram understanding as a symbolic structure prediction task, supporting advances in vision-language reasoning, scientific illustration parsing, and geometric theorem-proving. The move from brittle rule-based systems to deep-learning-based, jointly optimized DPG extractors enables better generalization, scalability, and applicability in complex real-world diagrams [1711.09528] [2205.09947].

Future research directions include:
- Improving geometric primitive recognition, especially under occlusion and low contrast.
- Relation parsing under long-tail class distributions and symbol variability.
- Seamless integration of DPGs with downstream reasoning (e.g., theorem provers, multi-hop QA).
- Development of larger and even more diverse annotated datasets.
- Cross-domain DPG transfer and adaptation.

DPGs provide a critical structured interface for diagram-centric scientific AI, forming the backbone of multi-modal visual reasoning frameworks [1603.07396] [1711.09528] [2205.09947] [2205.09363].

Source: https://www.emergentmind.com/topics/diagram-parse-graphs-dpgs