---
title: 'TableDART: Adaptive Multimodal Table Understanding'
url: https://www.emergentmind.com/topics/tabledart
type: topic
---

# TableDART: Adaptive Multimodal Table Understanding

TableDART is a framework for table understanding that treats tables as multimodal objects—text plus layout and vision—but does so in a dynamic, adaptive, and training-efficient way. Rather than fine-tuning a large multimodal LLM that always consumes both text and image for every table-query pair, TableDART learns a lightweight routing policy that decides, per instance, whether to use a **Text-only** expert, an **Image-only** expert, or a **Fusion** agent that reasons over both. In the reported implementation, only a **2.59M-parameter MLP gating network** is trained, while the large pretrained experts remain frozen; on seven benchmarks, the framework establishes new state-of-the-art performance among open-source models and surpasses the strongest baseline by an average of **4.02%** [2509.14671].

## 1. Problem setting and motivation

Table understanding requires modeling both **semantic** information and **structural** information. The semantic side includes cell contents, headers, and accompanying textual context; the structural side includes rows, columns, hierarchical headers, merged cells, and layout. TableDART is formulated for two task families: **Table Question Answering (TQA)**, where a table $\mathcal{T}$ and a natural language question $q$ are mapped to an answer $y$, and **Table Fact Verification (TFV)**, where a table $\mathcal{T}$ and a statement $q$ are mapped to a verification label $y$ [2509.14671].

The framework is motivated by limitations in three established paradigms. **Table-as-Text** approaches serialize a table into a linear string and feed it to a text model; these approaches are strong on textual reasoning but lose true layout and can become sensitive to serialization choices and row order. **Table-as-Image** approaches preserve spatial layout and structural cues but struggle with fine-grained semantics and complex numerical reasoning. **Table-as-Multimodality** approaches combine both views inside a multimodal LLM, but the paper identifies two persistent issues: both modalities are processed for every query-table pair, which can introduce redundancy and conflicts, and the approach depends on costly fine-tuning of large multimodal models [2509.14671].

This design target places TableDART in the broader line of multimodal table reasoning systems, but with a distinctive emphasis on selective modality use. Related work on table-tool integration, such as TART, addresses different weaknesses of large language models by adding specialized tools for table operations and explanation generation [2409.11724]. TableDART instead focuses on routing across pretrained single-modality experts and an output-level fusion agent, with no full multimodal fine-tuning [2509.14671].

## 2. Architectural organization

TableDART is organized into three stages: **Multimodal Encoding**, **Gating Network (Policy Training)**, and **Dynamic Inference Pathways** [2509.14671]. The system integrates five components: a **Table-as-Text expert** $\mathcal{M}_t$, a **Table-as-Image expert** $\mathcal{M}_v$, a **Query embedding model** $\mathcal{E}_q$, a **gating network** $\mathcal{G}$, and a **Fusion agent** [2509.14671].

In the reported experiments, the **Table-as-Text expert** is **TableGPT2-7B**, the **Table-as-Image expert** is **Ovis2-8B**, the query embedder is **Sentence-BERT (all-MiniLM-L6-v2)**, and the Fusion agent is implemented with **Google Gemini 2.0 Flash** [2509.14671]. The text expert accepts serialized table text plus query; the image expert accepts table screenshots plus textual prompt. The query model produces a compact semantic representation of the question or statement [2509.14671].

The multimodal representation is formed by computing three embeddings:
$$
\begin{aligned}
\mathbf{e}_{t} &= \mathcal{E}_{t}(\text{Serialize}(\mathcal{T})) \\
\mathbf{e}_{v} &= \mathcal{E}_{v}(\text{Screenshot}(\mathcal{T})) \\
\mathbf{e}_{q} &= \mathcal{E}_{q}(q) \\
\mathbf{x} &= [\mathbf{e}_{q}, \mathbf{e}_{t}, \mathbf{e}_{v}]
\end{aligned}
$$
In the implementation described in the paper, $\mathbf{e}_t \in \mathbb{R}^{3584}$ is obtained by attention-masked mean pooling of TableGPT2-7B input embeddings, $\mathbf{e}_v \in \mathbb{R}^{6144}$ is obtained by spatial-temporal mean pooling of Ovis2-8B visual encoder outputs, and $\mathbf{e}_q \in \mathbb{R}^{384}$ is produced by MiniLM, giving a total input dimension of $\mathbf{x} \in \mathbb{R}^{10112}$ [2509.14671].

A central design choice is that only the embedding layers of the experts are used to construct $\mathbf{e}_t$ and $\mathbf{e}_v$ during routing. The rest of the expert models are activated only after a path is chosen. This is one of the mechanisms by which the framework remains training-efficient [2509.14671].

## 3. Gating network and routing objective

The gating network is a **2-layer MLP** that takes the concatenated representation $\mathbf{x}$ and outputs logits for three routing options: **Text-only**, **Image-only**, and **Fusion** [2509.14671]. Its forward computation is:
$$
\begin{aligned}
\mathbf{h} &= \text{ReLU}(\mathbf{W}_1 \mathbf{x} + \mathbf{b}_1) \\
\mathbf{h}' &= \text{Dropout}(\mathbf{h}, p=0.1) \\
\mathbf{z} &= \mathbf{W}_2 \mathbf{h}' + \mathbf{b}_2
\end{aligned}
$$
with $\mathbf{W}_1 \in \mathbb{R}^{256 \times 10112}$ and $\mathbf{W}_2 \in \mathbb{R}^{3 \times 256}$ [2509.14671]. The total parameter count is reported as **2,589,699**, or approximately **2.59M** [2509.14671].

At inference time, the routing distribution is computed as
$$
p = \text{softmax}(\mathbf{z}),
$$
and the selected path is
$$
\hat{k} = \argmax_{k \in \{ \text{text}, \text{image}, \text{fusion} \}} p_k.
$$
The route is chosen per table-query pair, not per dataset or per model configuration [2509.14671].

The training objective combines a task-alignment term and a resource term:
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{resource}}.
$$
The task loss is defined by a KL divergence between a target distribution over paths and the gate’s predicted distribution:
$$
\mathcal{L}_{\text{task}} = \text{KL}\big(\text{softmax}(\mathbf{s} / \tau) \parallel \text{softmax}(\mathbf{z} / \tau_g)\big),
$$
where $\mathbf{s}$ is a vector of path scores, $\tau$ is a temperature for the target distribution, and $\tau_g$ is a temperature for the gate distribution [2509.14671]. The resource loss penalizes expected computational cost:
$$
\mathcal{L}_{\text{resource}} = \text{softmax}(\mathbf{z} / \tau_g)^T \mathbf{c},
$$
where the empirically measured cost vector is **0.73** for Text-only, **0.81** for Image-only, and **0.96** for Fusion [2509.14671].

The paper reports that the gate is trained on a **10,000-sample** mixture consisting of **2,000** samples from each of **WTQ**, **TABMWP**, **TAT-QA**, **TabFact**, and **InfoTabs**, with a **1,500-sample** validation set [2509.14671]. **HiTab** and **FeTaQA** are not used in training and are evaluated zero-shot [2509.14671]. Hyperparameters include a learning rate of $1 \times 10^{-4}$, cosine schedule with **5% warmup**, batch size **8**, gradient accumulation **4**, weight decay **0.01**, gradient clipping **1.0**, hidden dimension **256**, dropout **0.1**, temperatures $\tau = 0.3$ and $\tau_g = 1.0$, and resource weight $\lambda = 0.15$ [2509.14671].

## 4. Dynamic inference pathways and the fusion agent

After routing, TableDART follows one of three execution pathways. In the **Text-only** path, the system runs the table-text expert $\mathcal{M}_t$ to produce an answer. In the **Image-only** path, it runs the image expert $\mathcal{M}_v$ to produce an answer. In the **Fusion** path, it runs both experts and then passes their outputs, together with the original table and question, to the Fusion agent [2509.14671].

The Fusion agent receives the original question $q$, the full table $\mathcal{T}$ in markdown, the text expert’s answer and explanation $(r_t, a_t)$, and the image expert’s answer and explanation $(r_v, a_v)$, along with dataset-specific formatting instructions [2509.14671]. Its role is not limited to choosing one of the two answers. The paper explicitly describes two operating modes. As **Arbitrator**, it selects the better answer when the two experts disagree and one is clearly supported by the table. As **Rescuer**, it synthesizes a new answer when both experts are uncertain or partially wrong [2509.14671].

The paper provides qualitative examples of both modes. In a **TABMWP** case, the text model miscomputes a probability while the image model computes the correct value; the Fusion agent analyzes the table and the two rationales and selects the image model’s answer. In a **HiTab** case, neither expert is fully correct, but each returns a partially correct occupation; the Fusion agent combines the correct parts and produces the correct answer [2509.14671]. This indicates that TableDART’s fusion operates at the level of output reasoning rather than simple feature concatenation.

A plausible implication is that the framework treats multimodality as a contingent resource rather than a fixed architectural obligation. The paper’s own analysis supports this by showing that many instances can be solved correctly by a single modality, while a smaller subset benefits from cross-modal rescue or arbitration [2509.14671].

## 5. Benchmarks, results, and efficiency

TableDART is evaluated on **seven** benchmarks: **WTQ**, **TABMWP**, **TAT-QA**, **HiTab**, **FeTaQA**, **TabFact**, and **InfoTabs** [2509.14671]. The first five are TQA benchmarks, with **FeTaQA** evaluated using **BLEU**; **TabFact** and **InfoTabs** are TFV benchmarks evaluated using **accuracy** [2509.14671]. The reported test set sizes are **4,344** for WTQ, **7,686** for TABMWP, **772** for TAT-QA, **1,586** for HiTab, **2,003** for FeTaQA, **6,845** for TabFact, and **5,400** for InfoTabs [2509.14671].

The main reported results are as follows.

| Method | WTQ | TABMWP | TAT-QA |
|---|---:|---:|---:|
| TableGPT2-7B | 61.42 | 83.87 | 50.39 |
| Ovis2-8B | 58.76 | 87.00 | 47.67 |
| HIPPO-8B | 55.77 | 87.50 | 60.75 |
| TableDART | **70.58** | 84.54 | **62.05** |

| Method | HiTab | FeTaQA (BLEU) | TabFact | InfoTabs | Average Acc |
|---|---:|---:|---:|---:|---:|
| TableGPT2-7B | 70.27 | 28.97 | 77.80 | 71.07 | 69.14 |
| Ovis2-8B | 68.59 | 34.70 | 80.80 | 74.11 | 69.49 |
| HIPPO-8B | 63.00 | 33.18 | 82.27 | 75.74 | 70.84 |
| TableDART | **74.37** | **36.11** | 81.37 | **76.22** | **74.86** |

Across these benchmarks, TableDART surpasses both constituent experts and outperforms **HIPPO-8B** on **5/7** benchmarks, with an **average improvement of 4.02% accuracy and +2.93 BLEU points** [2509.14671]. The paper further reports that TableDART achieves **74.95%** average accuracy on seen datasets and **74.37%** on unseen datasets, whereas HIPPO-8B drops from **72.41%** to **63.00%** on unseen datasets; the corresponding improvement is reported as **+18.05% accuracy and +2.93 BLEU** on unseen evaluation [2509.14671].

Efficiency is analyzed by comparing **Dynamic Routing** with **Non-Adaptive Fusion**, where Fusion is applied to every instance. The reported average latency is **2.20 s** for Dynamic Routing and **2.92 s** for Non-Adaptive Fusion, while throughput is **17.77** tokens per second for Dynamic Routing and **3.85** for Non-Adaptive Fusion [2509.14671]. The paper summarizes this as a latency reduction of about **24.5%** and a throughput improvement of about **4.6×** [2509.14671].

The ablation on the resource weight $\lambda$ reports average accuracies of **66.73** for $\lambda=1.0$, **69.32** for $\lambda=0.15$, **67.64** for $\lambda=0.10$, **69.23** for $\lambda=0.05$, and **69.27** for $\lambda=0.00$ [2509.14671]. This indicates that a moderate resource penalty yields the best average performance in the reported configuration.

## 6. Interpretation, complementarity, and limitations

The paper’s path-contribution analysis divides the evaluation space into three broad regimes. In **58.7%** of cases, both unimodal experts are correct, so multimodal fusion is unnecessary. In about **24%** of cases, only one unimodal expert is correct, with **17.2%** solved only by the image path and **6.8%** solved only by the text path. In about **17.3%** of cases, both unimodal experts fail, and Fusion rescues about **2.4%** of these hard cases [2509.14671]. This analysis is central to TableDART’s rationale: text and image are complementary, but their utility is highly instance-dependent.

The framework should also be understood relative to adjacent strands of table research. TART targets table reasoning by integrating a table formatter, a tool maker, and an explanation generator, with a focus on precise table operations and explainability [2409.11724]. By contrast, TableDART targets adaptive multimodal routing over pretrained experts, aiming to avoid static processing of both text and image for every example [2509.14671]. In document table recognition, systems such as TDATR address a different problem—end-to-end recovery of HTML structure and cell-level alignment from table images—rather than question answering or fact verification over already given tables [2603.22819].

Several limitations are stated or implied in the reported evaluation. TableDART depends on the strength of its base experts; if both the text expert and the image expert are weak in a target domain, the framework cannot correct that without changing the experts themselves [2509.14671]. The Image-only path and the Fusion path depend on table screenshots, so poor rendering or very large tables may degrade performance [2509.14671]. The Fusion path also incurs higher latency because it requires running both experts and the Fusion agent [2509.14671]. In addition, the Fusion agent relies on dataset-specific prompt instructions for output formatting [2509.14671].

Within the scope of the reported experiments, however, TableDART demonstrates that table multimodality need not be processed statically. The framework’s defining claim is that selective routing across pretrained single-modality experts, combined with an output-level fusion agent, can improve accuracy, generalization, and efficiency simultaneously [2509.14671].

Source: https://www.emergentmind.com/topics/tabledart