---
title: Multimodal Tree Decoder (MTD) Overview
url: https://www.emergentmind.com/topics/multimodal-tree-decoder-mtd
type: topic
---

# Multimodal Tree Decoder (MTD) Overview

Searching arXiv for the cited MTD-related papers to ground the article in the current literature.
Searching arXiv for "Multimodal Tree Decoder" and closely related decoder architectures.
Multimodal Tree Decoder (MTD) denotes, in its narrowest and explicit sense, the end-to-end model introduced for table of contents extraction from document images, in which multimodal entity representations are decoded into a hierarchical tree of headings through explicit structural prediction [2212.02896]. In a broader conceptual sense suggested by adjacent work, the term also refers to decoder architectures that combine multimodal representations with sparse, iterative, or explicitly tree-structured prediction over image regions, document entities, symbolic layouts, program syntax trees, or entailment trees [2209.13959]. Across these settings, the common objective is not merely sequence generation but structured decoding: selecting nodes, relations, or evidence subsets so that the output preserves hierarchy, locality, and task-specific constraints.

## 1. Definition and conceptual scope

The explicit MTD formulation appears in table-of-contents extraction, where the model is “mainly composed of three parts, namely encoder, classifier, and decoder,” and the decoder reconstructs a tree over heading entities rather than predicting flat labels or bounded-depth levels [2212.02896]. The corresponding task is defined over document entities of the form $(content, position, heading, id)$, with the output tree encoding parent–child structure among headings.

The broader literature suggests a family resemblance rather than a single canonical implementation. In visual grounding, Dynamic MDETR decouples multimodal fusion from localization and performs “iterative refinement process” through sparse, text-guided exploration of 2D image features; the supplied description states that this is “exactly the sort of dynamic, hierarchical exploration pattern expected from a Multimodal Tree Decoder” [2209.13959]. In multimodal multi-hop question answering, entailment trees are generated over textualized image, table, and text facts, with leaves corresponding to multimodal evidence and intermediate nodes corresponding to composed reasoning steps [2412.05821]. In handwritten mathematical expression recognition, tree decoders predict symbol-layout structure through parent–child relations and relation labels such as Above, Below, Sup, Sub, Inside, and Right, again shifting the output space from strings to explicit trees [2303.07077]. Bidirectional tree decoding further extends this paradigm by introducing Mirror-Flipped Symbol Layout Trees and Bidirectional Asynchronous Training for tree-structured decoding [2401.00435].

A plausible implication is that “MTD” is best understood as a design pattern with two levels of use. The narrow use names a specific multimodal document-understanding model [2212.02896]. The broader use denotes decoder architectures that treat multimodal inference as structured tree growth, tree-like search, or entailment-tree construction, typically with explicit mechanisms for node selection, relation prediction, sparsity control, or grammar-like constraints [2209.13959].

## 2. Canonical formulation in document table-of-contents extraction

The canonical MTD instantiation is designed for ToC extraction from document images and PDFs. Its input space is a sequence of text-line entities, each annotated as $(content, position, heading, id)$, where `position` is a bounding box, `heading` indicates whether the entity is a heading or part of a heading, and `id` is a symbolic hierarchical identifier such as `2.2.1` [2212.02896]. The associated HierDoc benchmark contains **650** English scientific papers from arXiv, with **350** documents for training and **300** for testing, and includes both document-level ToC trees and entity-level annotations.

The encoder fuses three modalities for each entity: visual features, textual features, and layout features. Visual features are obtained from a frozen ResNet-34 + FPN backbone with RoIAlign pooling of size $3 \times 3$; textual features are obtained from frozen BERT followed by two linear layers with ReLU; layout features encode normalized box coordinates, normalized width and height, and local vertical gaps [2212.02896]. Fusion is performed by a gated multimodal unit:
\[
\begin{aligned}
z_t &= \sigma\left( \mathbf{W}_z [f_t^v, f_t^s, f_t^p] \right) \\
f_t &= z_t * f_t^v + (1 - z_t) * f_t^s + \mathbf{E}_z f_t^p
\end{aligned}
\]
This formulation makes the relative contribution of vision and text adaptive, while layout enters through a learned projection.

The classifier operates over the fused entity sequence in reading order, defined as top-to-bottom and then left-to-right. A BiGRU contextualizes entity features and a softmax layer predicts heading versus normal, trained with focal loss to address class imbalance [2212.02896]. Only predicted heading entities are forwarded to the decoder.

The decoder is tree-structured. It first applies a Transformer encoder over the heading sequence to model global interactions, then processes headings sequentially. For each heading, it selects a **reference entity** among prior headings and predicts one of three relations: **parent**, **sibling**, or **identity**. The relation semantics are operational: `parent` attaches the current heading as a child of the reference, `sibling` attaches it as a child of the reference’s parent, and `identity` merges multi-line headings into a single logical heading [2212.02896]. This choice avoids direct prediction of absolute depth and therefore does not impose a fixed upper bound on hierarchy depth.

The decoder uses hard attention to select a single reference entity. Its attention score is Bahdanau-style and incorporates accumulated past alignments:
\[
\hat{e}_{si} = v^T \tanh(W_h \hat{h}_s + W_m m_i + W_d d_i)
\]
The selected reference induces a context vector, and relation prediction is performed by a feed-forward network over the context and hidden state. The overall training objective is
\[
O = \alpha_1 L_{cls} + \alpha_2 L_{ref} + \alpha_3 L_{re}
\]
with $\alpha_1 = \alpha_2 = \alpha_3 = 1$ in experiments [2212.02896].

## 3. Decoder mechanisms across multimodal and tree-like variants

A central mechanism in broader MTD-like systems is **explicit structural decomposition**. In the ToC model, the decomposition is reference-entity selection plus relation prediction over headings [2212.02896]. In handwritten mathematical expression recognition, SS-TD decomposes decoding into prediction of triples $(y_t^c, y_t^p, y_t^{rel})$, where the child symbol, parent node, and spatial relation are predicted separately, and relation labels are drawn from a fixed set $\{\text{Above}, \text{Below}, \text{Sup}, \text{Sub}, \text{Inside}, \text{Right}\}$ [2303.07077]. In automated program repair, KNOD further refines this structural factorization into a three-stage tree decoder: **parent decoder**, **edge decoder**, and **node decoder**, generating ASTs “incrementally, edge-by-edge and node-by-node” [2302.01857]. Although KNOD is not presented as a multimodal model, the supplied discussion proposes it as a foundation for multimodal tree decoding.

A second recurring mechanism is **localized or sparse expansion**. Dynamic MDETR implements this through a 2D adaptive sampling module and a text-guided decoding module, stacked alternately across layers [2209.13959]. At decoder layer $i$, average-pooled language features are fused with a sampling query, offsets are predicted relative to a reference point, and only $P$ sampled image positions are decoded:
\[
f_s^i = \mathrm{MLP}([f_s^{i-1}; f_l^i]), \qquad
\{(\Delta x_j^i, \Delta y_j^i)\}_{j=1}^{P} = \mathrm{Linear}(f_s^i)
\]
followed by bilinear interpolation on the 2D feature map. The decoder then performs self-attention among sampled visual tokens and cross-attention with language as queries. The supplied description explicitly states that this is “not a literal tree structure in the data structure sense,” but that it is “functionally” a tree-like search over the 2D feature map [2209.13959].

A third mechanism is **tree construction over multimodal fact bases**. In multimodal multi-hop QA, the entailment-tree approach converts images to captions with LLaVA-1.5 and tables to natural-language linearizations, yielding a unified textual fact base derived from images, tables, and text [2412.05821]. An entailment tree is defined as
\[
T = (h, L, E, S)
\]
where $L$ are leaf facts, $E$ are intermediate nodes, and $S$ are entailment steps. GPT-3.5 generates initial tree structure and intermediate nodes, while a T5-based Mixture-of-Experts model refines leaf selection through a Fact Retrieval Generation decoder and predicts answers through a QA decoder. This system treats tree generation and answer prediction as coupled decoding problems over multimodal evidence.

A fourth mechanism is **constraint-aware decoding**. SS-TD applies syntax masks to relation logits so that symbol-specific relation capabilities and non-repetition constraints directly modulate the decoder distribution [2303.07077]. KNOD applies first-order-logic syntax and semantic rules to define sets of must, might, and invalid actions, producing teacher distributions over parents, edge labels, and AST nodes during both training and inference [2302.01857]. BAT-TDv2, by contrast, enriches tree decoding with bidirectional context by introducing Mirror-Flipped Symbol Layout Trees and Hidden-state Attention over a right-to-left branch, allowing future structural context to influence left-to-right decoding [2401.00435].

## 4. Learning objectives, constraints, and evaluation

The explicit ToC MTD is trained jointly for heading classification, reference selection, and relation prediction. All three losses are focal losses, and evaluation combines **F1-Measure** with **tree-edit-distance similarity (TEDS)**. TEDS is defined as
\[
\operatorname{TEDS}\left(T_a, T_b\right) = 1 - \frac{\operatorname{EditDist}(T_a, T_b)}{\max(|T_a|, |T_b|)}
\]
which measures similarity between predicted and ground-truth ToC trees by normalized tree edit distance [2212.02896]. This metric is structurally appropriate because it evaluates content and hierarchy jointly.

Dynamic MDETR uses a different objective because its output is a single grounding box rather than a document tree. The regression head predicts
\[
\hat{b} = (\hat{x}, \hat{y}, \hat{w}, \hat{h})
\]
from pooled language features, and training uses
\[
\mathcal{L} = \mathcal{L}_{L1}(b, \hat{b}) + \mathcal{L}_{\mathrm{GIoU}}
\]
with no bipartite matching and no classification head [2209.13959]. This separation between multimodal fusion and sparse localization is one of the principal departures from encoder-only grounding frameworks.

SS-TD defines a multi-term objective aligned with its triple decoder:
\[
L = \lambda_1 L_c + \lambda_2 L_{\text{pos}} + \lambda_3 L_{\text{rel}} + \lambda_4 L_{\alpha}
\]
with $\lambda_1 = \lambda_2 = \lambda_3 = 1$ and $\lambda_4 = 0.1$ [2303.07077]. Here $L_c$ is child prediction loss, $L_{\text{pos}}$ is parent prediction loss, $L_{\text{rel}}$ is relation prediction loss, and $L_{\alpha}$ is an attention self-regularization term. Syntax masks are computed from a static relation-capability matrix and a dynamic matrix tracking already used relations.

In multimodal QA, the small MoE model is trained by multi-task learning:
\[
L = L_{frg} + L_{qa}
\]
where $L_{frg}$ supervises fact-index decoding and $L_{qa}$ supervises token-level answer generation [2412.05821]. Because WebQA and MultimodalQA do not provide gold entailment trees, supervision is weakly induced from evidence labels and answers rather than from fully annotated tree structures.

KNOD augments cross-entropy with KL distillation from rule-derived teacher distributions over parent, edge, and node choices [2302.01857]. A plausible implication is that future MTDs may combine multimodal evidence with similarly explicit rule distillation whenever the output domain admits syntactic or semantic constraints.

## 5. Representative results and application domains

The empirical record associated with MTD-style decoding spans document understanding, visual grounding, multimodal question answering, handwritten mathematical expression recognition, and structured code generation. Only some of these systems are explicitly named MTD, but the supplied literature treats them as closely related instantiations or design blueprints.

| Setting | Representative result | Source |
|---|---|---|
| ToC extraction on HierDoc | Average TEDS **87.2%** and average F1-Measure **88.1%** | [2212.02896] |
| Visual grounding on RefCOCOg-umd | **69.20%** vs **67.66%** with **2.23** vs **3.96 GFLOPs** for 3 encoders + 3 dynamic decoders, $P=36$ | [2209.13959] |
| WebQA | Retrieval **0.89**, QA-FL **0.68**, QA-Acc **0.73**, QA **0.54** | [2412.05821] |
| MultimodalQA | All EM **68.2**, All F1 **70.9** | [2412.05821] |
| HMER with SS-TD | Exp.Rate\(_{latex}\) **52.48 / 51.29 / 54.32** on CROHME 2014/2016/2019 | [2303.07077] |
| HMER with BAT-TDv2 | **60.34 / 60.50 / 60.80** on CROHME 2014/2016/2019 and **68.66** on HME100K | [2401.00435] |
| Automated program repair | **72** bugs on Defects4J v1.2, **25** on QuixBugs, **50** on additional Defects4J v2.0 | [2302.01857] |

Within the explicit MTD paper, the document-domain ablations are particularly informative. Removing text drops ToC extraction TEDS from **87.2** to **63.7**, removing layout drops it to **80.4**, and removing vision drops it to **86.5**, indicating that text is the most critical modality while layout contributes substantially to structure recovery [2212.02896]. The same study reports that the tree decoder substantially outperforms a depth-classification baseline: **87.2** TEDS for MTD versus **72.1** for a C-class depth classifier.

Dynamic MDETR shows that sparse, text-guided decoding can improve both accuracy and efficiency. On RefCOCOg-umd, using only **9%** feature points in the decoder reduces approximately **44%** GFLOPs while improving Top-1 accuracy from **67.66%** to **69.20%** [2209.13959]. The CLIP-based version further reports **85.97** on RefCOCO val, **74.83** on RefCOCO+ val, and **74.14** on RefCOCOg val-u.

In multimodal QA, the entailment-tree approach achieves the first place on the official WebQA leaderboard “since April 10, 2024,” and its ablations show that removing entailment tree generation, FRG, MMOE, or the iterative feedback mechanism degrades multi-modal exact match [2412.05821]. In HMER, SS-TD and BAT-TDv2 show that tree decoding benefits from both explicit relation constraints and bidirectional context, while BAT-TDv2’s gains over TDv2 are reported under greedy decoding rather than beam search [2303.07077].

## 6. Misconceptions, limitations, and research directions

A common misconception is that every MTD must be a literal tree data structure. The supplied material explicitly rejects that equation in the case of Dynamic MDETR: it is “not a literal tree structure in the data structure sense,” but behaves like tree search through iterative, sparse, text-guided exploration of image space [2209.13959]. The more general point is that MTD-like systems may be literal tree decoders, as in ToC extraction or AST generation, or tree-like decoders, as in sparse localization.

A second misconception is that MTD is confined to document understanding. The explicit name belongs to ToC extraction [2212.02896], but the supplied literature suggests a much wider structural pattern: tree generation over multimodal evidence in QA [2412.05821], symbol-layout decoding in HMER [2303.07077; 2401.00435], and tree-centric constrained generation in program repair [2302.01857]. This suggests a broader research category, though that broader categorization is interpretive rather than terminologically uniform across all papers.

The limitations are domain-specific. The ToC MTD depends on reliable entity segmentation and on accurate PDF parsing; the authors note that errors in this stage propagate, and that the dataset is drawn from scientific papers from arXiv, so generalization to legal, financial, or magazine documents may differ [2212.02896]. Dynamic MDETR depends on informative sparse sampling; the reported ablations show that increasing the number of sampled points beyond approximately 100 can hurt accuracy because spatial redundancy is reintroduced [2209.13959]. The multimodal QA approach remains dependent on LLM quality, lacks gold entailment tree annotations on WebQA and MultimodalQA, and incurs substantial computational cost because it combines multiple LLM calls with MoE training and iterative feedback [2412.05821]. In HMER, the benefit of Shared Language Modeling grows with larger datasets and is modest on smaller datasets such as CROHME [2401.00435]. In KNOD, the current system is limited by fault localization, focuses on Java, and struggles with multi-hunk bugs [2302.01857].

The research trajectory implied by these works is relatively clear. The document MTD paper suggests extension to more document types and languages, end-to-end fine-tuning of multimodal encoders, and integration with broader document-understanding pipelines [2212.02896]. Dynamic MDETR suggests that MTD-style decoders may serve as lightweight task heads for large multimodal foundation encoders [2209.13959]. The multimodal QA work suggests end-to-end trainable tree decoders, more explicit multimodal entailment-tree supervision, and tree-aware expert routing [2412.05821]. KNOD suggests that multimodal rule distillation could combine syntax, semantics, dynamic traces, and natural-language constraints in a single structured decoder [2302.01857]. Taken together, these works position MTD not as a single architecture, but as an increasingly coherent research program in multimodal structured decoding.

Source: https://www.emergentmind.com/topics/multimodal-tree-decoder-mtd