TableDART: Adaptive Multimodal Table Understanding

• The paper introduces TableDART, a framework that adaptively routes table queries using a lightweight 2.59M-parameter gating network.
• It leverages pretrained text and image experts along with a fusion agent to address both semantic and structural challenges in table understanding.
• Experiments across seven benchmarks demonstrate a 4.02% average accuracy improvement and reduced latency compared to non-adaptive fusion methods.

TableDART is a framework for table understanding that treats tables as multimodal objects—text plus layout and vision—but does so in a dynamic, adaptive, and training-efficient way. Rather than fine-tuning a large multimodal LLM that always consumes both text and image for every table-query pair, TableDART learns a lightweight routing policy that decides, per instance, whether to use a Text-only expert, an Image-only expert, or a Fusion agent that reasons over both. In the reported implementation, only a 2.59M-parameter MLP gating network is trained, while the large pretrained experts remain frozen; on seven benchmarks, the framework establishes new state-of-the-art performance among open-source models and surpasses the strongest baseline by an average of 4.02% (Xing et al., 18 Sep 2025).

1. Problem setting and motivation

Table understanding requires modeling both semantic information and structural information. The semantic side includes cell contents, headers, and accompanying textual context; the structural side includes rows, columns, hierarchical headers, merged cells, and layout. TableDART is formulated for two task families: Table Question Answering (TQA), where a table $\mathcal{T}$ and a natural language question $q$ are mapped to an answer $y$, and Table Fact Verification (TFV), where a table $\mathcal{T}$ and a statement $q$ are mapped to a verification label $y$ (Xing et al., 18 Sep 2025).

The framework is motivated by limitations in three established paradigms. Table-as-Text approaches serialize a table into a linear string and feed it to a text model; these approaches are strong on textual reasoning but lose true layout and can become sensitive to serialization choices and row order. Table-as-Image approaches preserve spatial layout and structural cues but struggle with fine-grained semantics and complex numerical reasoning. Table-as-Multimodality approaches combine both views inside a multimodal LLM, but the paper identifies two persistent issues: both modalities are processed for every query-table pair, which can introduce redundancy and conflicts, and the approach depends on costly fine-tuning of large multimodal models (Xing et al., 18 Sep 2025).

This design target places TableDART in the broader line of multimodal table reasoning systems, but with a distinctive emphasis on selective modality use. Related work on table-tool integration, such as TART, addresses different weaknesses of LLMs by adding specialized tools for table operations and explanation generation (Lu et al., 2024). TableDART instead focuses on routing across pretrained single-modality experts and an output-level fusion agent, with no full multimodal fine-tuning (Xing et al., 18 Sep 2025).

2. Architectural organization

TableDART is organized into three stages: Multimodal Encoding, Gating Network (Policy Training), and Dynamic Inference Pathways (Xing et al., 18 Sep 2025). The system integrates five components: a Table-as-Text expert $\mathcal{M}_t$, a Table-as-Image expert $\mathcal{M}_v$, a Query embedding model $\mathcal{E}_q$, a gating network $\mathcal{G}$, and a Fusion agent (Xing et al., 18 Sep 2025).

In the reported experiments, the Table-as-Text expert is TableGPT2-7B, the Table-as-Image expert is Ovis2-8B, the query embedder is Sentence-BERT (all-MiniLM-L6-v2), and the Fusion agent is implemented with Google Gemini 2.0 Flash (Xing et al., 18 Sep 2025). The text expert accepts serialized table text plus query; the image expert accepts table screenshots plus textual prompt. The query model produces a compact semantic representation of the question or statement (Xing et al., 18 Sep 2025).

The multimodal representation is formed by computing three embeddings:

$q$0

In the implementation described in the paper, $q$1 is obtained by attention-masked mean pooling of TableGPT2-7B input embeddings, $q$2 is obtained by spatial-temporal mean pooling of Ovis2-8B visual encoder outputs, and $q$3 is produced by MiniLM, giving a total input dimension of $q$4 (Xing et al., 18 Sep 2025).

A central design choice is that only the embedding layers of the experts are used to construct $q$5 and $q$6 during routing. The rest of the expert models are activated only after a path is chosen. This is one of the mechanisms by which the framework remains training-efficient (Xing et al., 18 Sep 2025).

3. Gating network and routing objective

The gating network is a 2-layer MLP that takes the concatenated representation $q$7 and outputs logits for three routing options: Text-only, Image-only, and Fusion (Xing et al., 18 Sep 2025). Its forward computation is:

$q$8

with $q$9 and $y$0 (Xing et al., 18 Sep 2025). The total parameter count is reported as 2,589,699, or approximately 2.59M (Xing et al., 18 Sep 2025).

At inference time, the routing distribution is computed as

$y$1

and the selected path is

$y$2

The route is chosen per table-query pair, not per dataset or per model configuration (Xing et al., 18 Sep 2025).

The training objective combines a task-alignment term and a resource term:

$y$3

The task loss is defined by a KL divergence between a target distribution over paths and the gate’s predicted distribution:

$y$4

where $y$5 is a vector of path scores, $y$6 is a temperature for the target distribution, and $y$7 is a temperature for the gate distribution (Xing et al., 18 Sep 2025). The resource loss penalizes expected computational cost:

$y$8

where the empirically measured cost vector is 0.73 for Text-only, 0.81 for Image-only, and 0.96 for Fusion (Xing et al., 18 Sep 2025).

The paper reports that the gate is trained on a 10,000-sample mixture consisting of 2,000 samples from each of WTQ, TABMWP, TAT-QA, TabFact, and InfoTabs, with a 1,500-sample validation set (Xing et al., 18 Sep 2025). HiTab and FeTaQA are not used in training and are evaluated zero-shot (Xing et al., 18 Sep 2025). Hyperparameters include a learning rate of $y$9, cosine schedule with 5% warmup, batch size 8, gradient accumulation 4, weight decay 0.01, gradient clipping 1.0, hidden dimension 256, dropout 0.1, temperatures $\mathcal{T}$0 and $\mathcal{T}$1, and resource weight $\mathcal{T}$2 (Xing et al., 18 Sep 2025).

4. Dynamic inference pathways and the fusion agent

After routing, TableDART follows one of three execution pathways. In the Text-only path, the system runs the table-text expert $\mathcal{T}$3 to produce an answer. In the Image-only path, it runs the image expert $\mathcal{T}$4 to produce an answer. In the Fusion path, it runs both experts and then passes their outputs, together with the original table and question, to the Fusion agent (Xing et al., 18 Sep 2025).

The Fusion agent receives the original question $\mathcal{T}$5, the full table $\mathcal{T}$6 in markdown, the text expert’s answer and explanation $\mathcal{T}$7, and the image expert’s answer and explanation $\mathcal{T}$8, along with dataset-specific formatting instructions (Xing et al., 18 Sep 2025). Its role is not limited to choosing one of the two answers. The paper explicitly describes two operating modes. As Arbitrator, it selects the better answer when the two experts disagree and one is clearly supported by the table. As Rescuer, it synthesizes a new answer when both experts are uncertain or partially wrong (Xing et al., 18 Sep 2025).

The paper provides qualitative examples of both modes. In a TABMWP case, the text model miscomputes a probability while the image model computes the correct value; the Fusion agent analyzes the table and the two rationales and selects the image model’s answer. In a HiTab case, neither expert is fully correct, but each returns a partially correct occupation; the Fusion agent combines the correct parts and produces the correct answer (Xing et al., 18 Sep 2025). This indicates that TableDART’s fusion operates at the level of output reasoning rather than simple feature concatenation.

A plausible implication is that the framework treats multimodality as a contingent resource rather than a fixed architectural obligation. The paper’s own analysis supports this by showing that many instances can be solved correctly by a single modality, while a smaller subset benefits from cross-modal rescue or arbitration (Xing et al., 18 Sep 2025).

5. Benchmarks, results, and efficiency

TableDART is evaluated on seven benchmarks: WTQ, TABMWP, TAT-QA, HiTab, FeTaQA, TabFact, and InfoTabs (Xing et al., 18 Sep 2025). The first five are TQA benchmarks, with FeTaQA evaluated using BLEU; TabFact and InfoTabs are TFV benchmarks evaluated using accuracy (Xing et al., 18 Sep 2025). The reported test set sizes are 4,344 for WTQ, 7,686 for TABMWP, 772 for TAT-QA, 1,586 for HiTab, 2,003 for FeTaQA, 6,845 for TabFact, and 5,400 for InfoTabs (Xing et al., 18 Sep 2025).

The main reported results are as follows.

Method	WTQ	TABMWP	TAT-QA
TableGPT2-7B	61.42	83.87	50.39
Ovis2-8B	58.76	87.00	47.67
HIPPO-8B	55.77	87.50	60.75
TableDART	70.58	84.54	62.05

Method	HiTab	FeTaQA (BLEU)	TabFact	InfoTabs	Average Acc
TableGPT2-7B	70.27	28.97	77.80	71.07	69.14
Ovis2-8B	68.59	34.70	80.80	74.11	69.49
HIPPO-8B	63.00	33.18	82.27	75.74	70.84
TableDART	74.37	36.11	81.37	76.22	74.86

Across these benchmarks, TableDART surpasses both constituent experts and outperforms HIPPO-8B on 5/7 benchmarks, with an average improvement of 4.02% accuracy and +2.93 BLEU points (Xing et al., 18 Sep 2025). The paper further reports that TableDART achieves 74.95% average accuracy on seen datasets and 74.37% on unseen datasets, whereas HIPPO-8B drops from 72.41% to 63.00% on unseen datasets; the corresponding improvement is reported as +18.05% accuracy and +2.93 BLEU on unseen evaluation (Xing et al., 18 Sep 2025).

Efficiency is analyzed by comparing Dynamic Routing with Non-Adaptive Fusion, where Fusion is applied to every instance. The reported average latency is 2.20 s for Dynamic Routing and 2.92 s for Non-Adaptive Fusion, while throughput is 17.77 tokens per second for Dynamic Routing and 3.85 for Non-Adaptive Fusion (Xing et al., 18 Sep 2025). The paper summarizes this as a latency reduction of about 24.5% and a throughput improvement of about 4.6× (Xing et al., 18 Sep 2025).

The ablation on the resource weight $\mathcal{T}$9 reports average accuracies of 66.73 for $q$0, 69.32 for $q$1, 67.64 for $q$2, 69.23 for $q$3, and 69.27 for $q$4 (Xing et al., 18 Sep 2025). This indicates that a moderate resource penalty yields the best average performance in the reported configuration.

6. Interpretation, complementarity, and limitations

The paper’s path-contribution analysis divides the evaluation space into three broad regimes. In 58.7% of cases, both unimodal experts are correct, so multimodal fusion is unnecessary. In about 24% of cases, only one unimodal expert is correct, with 17.2% solved only by the image path and 6.8% solved only by the text path. In about 17.3% of cases, both unimodal experts fail, and Fusion rescues about 2.4% of these hard cases (Xing et al., 18 Sep 2025). This analysis is central to TableDART’s rationale: text and image are complementary, but their utility is highly instance-dependent.

The framework should also be understood relative to adjacent strands of table research. TART targets table reasoning by integrating a table formatter, a tool maker, and an explanation generator, with a focus on precise table operations and explainability (Lu et al., 2024). By contrast, TableDART targets adaptive multimodal routing over pretrained experts, aiming to avoid static processing of both text and image for every example (Xing et al., 18 Sep 2025). In document table recognition, systems such as TDATR address a different problem—end-to-end recovery of HTML structure and cell-level alignment from table images—rather than question answering or fact verification over already given tables (Qin et al., 24 Mar 2026).

Several limitations are stated or implied in the reported evaluation. TableDART depends on the strength of its base experts; if both the text expert and the image expert are weak in a target domain, the framework cannot correct that without changing the experts themselves (Xing et al., 18 Sep 2025). The Image-only path and the Fusion path depend on table screenshots, so poor rendering or very large tables may degrade performance (Xing et al., 18 Sep 2025). The Fusion path also incurs higher latency because it requires running both experts and the Fusion agent (Xing et al., 18 Sep 2025). In addition, the Fusion agent relies on dataset-specific prompt instructions for output formatting (Xing et al., 18 Sep 2025).

Within the scope of the reported experiments, however, TableDART demonstrates that table multimodality need not be processed statically. The framework’s defining claim is that selective routing across pretrained single-modality experts, combined with an output-level fusion agent, can improve accuracy, generalization, and efficiency simultaneously (Xing et al., 18 Sep 2025).