Multimodal Tree Decoder (MTD) Overview

• The paper demonstrates that integrating a tree decoder with multimodal fusion significantly improves table-of-contents extraction, achieving 87.2% TEDS and 88.1% F1-Measure.
• MTD is a design pattern that decodes structured hierarchies from multimodal inputs and extends beyond document understanding to tasks like visual grounding and program repair.
• MTD decoders employ explicit structural decomposition and localized expansion to maintain hierarchy and enforce task-specific constraints during inference.

Searching arXiv for the cited MTD-related papers to ground the article in the current literature. Searching arXiv for "Multimodal Tree Decoder" and closely related decoder architectures. Multimodal Tree Decoder (MTD) denotes, in its narrowest and explicit sense, the end-to-end model introduced for table of contents extraction from document images, in which multimodal entity representations are decoded into a hierarchical tree of headings through explicit structural prediction (Hu et al., 2022). In a broader conceptual sense suggested by adjacent work, the term also refers to decoder architectures that combine multimodal representations with sparse, iterative, or explicitly tree-structured prediction over image regions, document entities, symbolic layouts, program syntax trees, or entailment trees (Shi et al., 2022). Across these settings, the common objective is not merely sequence generation but structured decoding: selecting nodes, relations, or evidence subsets so that the output preserves hierarchy, locality, and task-specific constraints.

1. Definition and conceptual scope

The explicit MTD formulation appears in table-of-contents extraction, where the model is “mainly composed of three parts, namely encoder, classifier, and decoder,” and the decoder reconstructs a tree over heading entities rather than predicting flat labels or bounded-depth levels (Hu et al., 2022). The corresponding task is defined over document entities of the form $(content, position, heading, id)$, with the output tree encoding parent–child structure among headings.

The broader literature suggests a family resemblance rather than a single canonical implementation. In visual grounding, Dynamic MDETR decouples multimodal fusion from localization and performs “iterative refinement process” through sparse, text-guided exploration of 2D image features; the supplied description states that this is “exactly the sort of dynamic, hierarchical exploration pattern expected from a Multimodal Tree Decoder” (Shi et al., 2022). In multimodal multi-hop question answering, entailment trees are generated over textualized image, table, and text facts, with leaves corresponding to multimodal evidence and intermediate nodes corresponding to composed reasoning steps (Zhang et al., 2024). In handwritten mathematical expression recognition, tree decoders predict symbol-layout structure through parent–child relations and relation labels such as Above, Below, Sup, Sub, Inside, and Right, again shifting the output space from strings to explicit trees (Lin et al., 2023). Bidirectional tree decoding further extends this paradigm by introducing Mirror-Flipped Symbol Layout Trees and Bidirectional Asynchronous Training for tree-structured decoding (Cheng et al., 2023).

A plausible implication is that “MTD” is best understood as a design pattern with two levels of use. The narrow use names a specific multimodal document-understanding model (Hu et al., 2022). The broader use denotes decoder architectures that treat multimodal inference as structured tree growth, tree-like search, or entailment-tree construction, typically with explicit mechanisms for node selection, relation prediction, sparsity control, or grammar-like constraints (Shi et al., 2022).

2. Canonical formulation in document table-of-contents extraction

The canonical MTD instantiation is designed for ToC extraction from document images and PDFs. Its input space is a sequence of text-line entities, each annotated as $(content, position, heading, id)$, where position is a bounding box, heading indicates whether the entity is a heading or part of a heading, and id is a symbolic hierarchical identifier such as 2.2.1 (Hu et al., 2022). The associated HierDoc benchmark contains 650 English scientific papers from arXiv, with 350 documents for training and 300 for testing, and includes both document-level ToC trees and entity-level annotations.

The encoder fuses three modalities for each entity: visual features, textual features, and layout features. Visual features are obtained from a frozen ResNet-34 + FPN backbone with RoIAlign pooling of size $3 \times 3$; textual features are obtained from frozen BERT followed by two linear layers with ReLU; layout features encode normalized box coordinates, normalized width and height, and local vertical gaps (Hu et al., 2022). Fusion is performed by a gated multimodal unit: $$\begin{aligned} z_t &= \sigma\left( \mathbf{W}_z [f_t^v, f_t^s, f_t^p] \right) \ f_t &= z_t * f_t^v + (1 - z_t) * f_t^s + \mathbf{E}_z f_t^p \end{aligned}$$ This formulation makes the relative contribution of vision and text adaptive, while layout enters through a learned projection.

The classifier operates over the fused entity sequence in reading order, defined as top-to-bottom and then left-to-right. A BiGRU contextualizes entity features and a softmax layer predicts heading versus normal, trained with focal loss to address class imbalance (Hu et al., 2022). Only predicted heading entities are forwarded to the decoder.

The decoder is tree-structured. It first applies a Transformer encoder over the heading sequence to model global interactions, then processes headings sequentially. For each heading, it selects a reference entity among prior headings and predicts one of three relations: parent, sibling, or identity. The relation semantics are operational: parent attaches the current heading as a child of the reference, sibling attaches it as a child of the reference’s parent, and identity merges multi-line headings into a single logical heading (Hu et al., 2022). This choice avoids direct prediction of absolute depth and therefore does not impose a fixed upper bound on hierarchy depth.

The decoder uses hard attention to select a single reference entity. Its attention score is Bahdanau-style and incorporates accumulated past alignments: $$\hat{e}_{si} = v^T \tanh(W_h \hat{h}_s + W_m m_i + W_d d_i)$$ The selected reference induces a context vector, and relation prediction is performed by a feed-forward network over the context and hidden state. The overall training objective is

$$O = \alpha_1 L_{cls} + \alpha_2 L_{ref} + \alpha_3 L_{re}$$

with $\alpha_1 = \alpha_2 = \alpha_3 = 1$ in experiments (Hu et al., 2022).

3. Decoder mechanisms across multimodal and tree-like variants

A central mechanism in broader MTD-like systems is explicit structural decomposition. In the ToC model, the decomposition is reference-entity selection plus relation prediction over headings (Hu et al., 2022). In handwritten mathematical expression recognition, SS-TD decomposes decoding into prediction of triples $(y_t^c, y_t^p, y_t^{rel})$, where the child symbol, parent node, and spatial relation are predicted separately, and relation labels are drawn from a fixed set $$\{\text{Above}, \text{Below}, \text{Sup}, \text{Sub}, \text{Inside}, \text{Right}\}$$ (Lin et al., 2023). In automated program repair, KNOD further refines this structural factorization into a three-stage tree decoder: parent decoder, edge decoder, and node decoder, generating ASTs “incrementally, edge-by-edge and node-by-node” (Jiang et al., 2023). Although KNOD is not presented as a multimodal model, the supplied discussion proposes it as a foundation for multimodal tree decoding.

A second recurring mechanism is localized or sparse expansion. Dynamic MDETR implements this through a 2D adaptive sampling module and a text-guided decoding module, stacked alternately across layers (Shi et al., 2022). At decoder layer $i$, average-pooled language features are fused with a sampling query, offsets are predicted relative to a reference point, and only $(content, position, heading, id)$0 sampled image positions are decoded: $(content, position, heading, id)$1 followed by bilinear interpolation on the 2D feature map. The decoder then performs self-attention among sampled visual tokens and cross-attention with language as queries. The supplied description explicitly states that this is “not a literal tree structure in the data structure sense,” but that it is “functionally” a tree-like search over the 2D feature map (Shi et al., 2022).

A third mechanism is tree construction over multimodal fact bases. In multimodal multi-hop QA, the entailment-tree approach converts images to captions with LLaVA-1.5 and tables to natural-language linearizations, yielding a unified textual fact base derived from images, tables, and text (Zhang et al., 2024). An entailment tree is defined as

$(content, position, heading, id)$2

where $(content, position, heading, id)$3 are leaf facts, $(content, position, heading, id)$4 are intermediate nodes, and $(content, position, heading, id)$5 are entailment steps. GPT-3.5 generates initial tree structure and intermediate nodes, while a T5-based Mixture-of-Experts model refines leaf selection through a Fact Retrieval Generation decoder and predicts answers through a QA decoder. This system treats tree generation and answer prediction as coupled decoding problems over multimodal evidence.

A fourth mechanism is constraint-aware decoding. SS-TD applies syntax masks to relation logits so that symbol-specific relation capabilities and non-repetition constraints directly modulate the decoder distribution (Lin et al., 2023). KNOD applies first-order-logic syntax and semantic rules to define sets of must, might, and invalid actions, producing teacher distributions over parents, edge labels, and AST nodes during both training and inference (Jiang et al., 2023). BAT-TDv2, by contrast, enriches tree decoding with bidirectional context by introducing Mirror-Flipped Symbol Layout Trees and Hidden-state Attention over a right-to-left branch, allowing future structural context to influence left-to-right decoding (Cheng et al., 2023).

4. Learning objectives, constraints, and evaluation

The explicit ToC MTD is trained jointly for heading classification, reference selection, and relation prediction. All three losses are focal losses, and evaluation combines F1-Measure with tree-edit-distance similarity (TEDS). TEDS is defined as

$(content, position, heading, id)$6

which measures similarity between predicted and ground-truth ToC trees by normalized tree edit distance (Hu et al., 2022). This metric is structurally appropriate because it evaluates content and hierarchy jointly.

Dynamic MDETR uses a different objective because its output is a single grounding box rather than a document tree. The regression head predicts

$(content, position, heading, id)$7

from pooled language features, and training uses

$(content, position, heading, id)$8

with no bipartite matching and no classification head (Shi et al., 2022). This separation between multimodal fusion and sparse localization is one of the principal departures from encoder-only grounding frameworks.

SS-TD defines a multi-term objective aligned with its triple decoder: $(content, position, heading, id)$9 with $3 \times 3$0 and $3 \times 3$1 (Lin et al., 2023). Here $3 \times 3$2 is child prediction loss, $3 \times 3$3 is parent prediction loss, $3 \times 3$4 is relation prediction loss, and $3 \times 3$5 is an attention self-regularization term. Syntax masks are computed from a static relation-capability matrix and a dynamic matrix tracking already used relations.

In multimodal QA, the small MoE model is trained by multi-task learning: $3 \times 3$6 where $3 \times 3$7 supervises fact-index decoding and $3 \times 3$8 supervises token-level answer generation (Zhang et al., 2024). Because WebQA and MultimodalQA do not provide gold entailment trees, supervision is weakly induced from evidence labels and answers rather than from fully annotated tree structures.

KNOD augments cross-entropy with KL distillation from rule-derived teacher distributions over parent, edge, and node choices (Jiang et al., 2023). A plausible implication is that future MTDs may combine multimodal evidence with similarly explicit rule distillation whenever the output domain admits syntactic or semantic constraints.

5. Representative results and application domains

The empirical record associated with MTD-style decoding spans document understanding, visual grounding, multimodal question answering, handwritten mathematical expression recognition, and structured code generation. Only some of these systems are explicitly named MTD, but the supplied literature treats them as closely related instantiations or design blueprints.

Setting	Representative result	Source
ToC extraction on HierDoc	Average TEDS 87.2% and average F1-Measure 88.1%	(Hu et al., 2022)
Visual grounding on RefCOCOg-umd	69.20% vs 67.66% with 2.23 vs 3.96 GFLOPs for 3 encoders + 3 dynamic decoders, $3 \times 3$9	(Shi et al., 2022)
WebQA	Retrieval 0.89, QA-FL 0.68, QA-Acc 0.73, QA 0.54	(Zhang et al., 2024)
MultimodalQA	All EM 68.2, All F1 70.9	(Zhang et al., 2024)
HMER with SS-TD	Exp.Rate$$\begin{aligned} z_t &= \sigma\left( \mathbf{W}_z [f_t^v, f_t^s, f_t^p] \right) \ f_t &= z_t * f_t^v + (1 - z_t) * f_t^s + \mathbf{E}_z f_t^p \end{aligned}$$0 52.48 / 51.29 / 54.32 on CROHME 2014/2016/2019	(Lin et al., 2023)
HMER with BAT-TDv2	60.34 / 60.50 / 60.80 on CROHME 2014/2016/2019 and 68.66 on HME100K	(Cheng et al., 2023)
Automated program repair	72 bugs on Defects4J v1.2, 25 on QuixBugs, 50 on additional Defects4J v2.0	(Jiang et al., 2023)

Within the explicit MTD paper, the document-domain ablations are particularly informative. Removing text drops ToC extraction TEDS from 87.2 to 63.7, removing layout drops it to 80.4, and removing vision drops it to 86.5, indicating that text is the most critical modality while layout contributes substantially to structure recovery (Hu et al., 2022). The same study reports that the tree decoder substantially outperforms a depth-classification baseline: 87.2 TEDS for MTD versus 72.1 for a C-class depth classifier.

Dynamic MDETR shows that sparse, text-guided decoding can improve both accuracy and efficiency. On RefCOCOg-umd, using only 9% feature points in the decoder reduces approximately 44% GFLOPs while improving Top-1 accuracy from 67.66% to 69.20% (Shi et al., 2022). The CLIP-based version further reports 85.97 on RefCOCO val, 74.83 on RefCOCO+ val, and 74.14 on RefCOCOg val-u.

In multimodal QA, the entailment-tree approach achieves the first place on the official WebQA leaderboard “since April 10, 2024,” and its ablations show that removing entailment tree generation, FRG, MMOE, or the iterative feedback mechanism degrades multi-modal exact match (Zhang et al., 2024). In HMER, SS-TD and BAT-TDv2 show that tree decoding benefits from both explicit relation constraints and bidirectional context, while BAT-TDv2’s gains over TDv2 are reported under greedy decoding rather than beam search (Lin et al., 2023).

6. Misconceptions, limitations, and research directions

A common misconception is that every MTD must be a literal tree data structure. The supplied material explicitly rejects that equation in the case of Dynamic MDETR: it is “not a literal tree structure in the data structure sense,” but behaves like tree search through iterative, sparse, text-guided exploration of image space (Shi et al., 2022). The more general point is that MTD-like systems may be literal tree decoders, as in ToC extraction or AST generation, or tree-like decoders, as in sparse localization.

A second misconception is that MTD is confined to document understanding. The explicit name belongs to ToC extraction (Hu et al., 2022), but the supplied literature suggests a much wider structural pattern: tree generation over multimodal evidence in QA (Zhang et al., 2024), symbol-layout decoding in HMER (Lin et al., 2023, Cheng et al., 2023), and tree-centric constrained generation in program repair (Jiang et al., 2023). This suggests a broader research category, though that broader categorization is interpretive rather than terminologically uniform across all papers.

The limitations are domain-specific. The ToC MTD depends on reliable entity segmentation and on accurate PDF parsing; the authors note that errors in this stage propagate, and that the dataset is drawn from scientific papers from arXiv, so generalization to legal, financial, or magazine documents may differ (Hu et al., 2022). Dynamic MDETR depends on informative sparse sampling; the reported ablations show that increasing the number of sampled points beyond approximately 100 can hurt accuracy because spatial redundancy is reintroduced (Shi et al., 2022). The multimodal QA approach remains dependent on LLM quality, lacks gold entailment tree annotations on WebQA and MultimodalQA, and incurs substantial computational cost because it combines multiple LLM calls with MoE training and iterative feedback (Zhang et al., 2024). In HMER, the benefit of Shared Language Modeling grows with larger datasets and is modest on smaller datasets such as CROHME (Cheng et al., 2023). In KNOD, the current system is limited by fault localization, focuses on Java, and struggles with multi-hunk bugs (Jiang et al., 2023).

The research trajectory implied by these works is relatively clear. The document MTD paper suggests extension to more document types and languages, end-to-end fine-tuning of multimodal encoders, and integration with broader document-understanding pipelines (Hu et al., 2022). Dynamic MDETR suggests that MTD-style decoders may serve as lightweight task heads for large multimodal foundation encoders (Shi et al., 2022). The multimodal QA work suggests end-to-end trainable tree decoders, more explicit multimodal entailment-tree supervision, and tree-aware expert routing (Zhang et al., 2024). KNOD suggests that multimodal rule distillation could combine syntax, semantics, dynamic traces, and natural-language constraints in a single structured decoder (Jiang et al., 2023). Taken together, these works position MTD not as a single architecture, but as an increasingly coherent research program in multimodal structured decoding.