ReTabVQA: Table VQA Benchmark

• ReTabVQA is a Table VQA benchmark focused on multi-step quantitative reasoning, requiring models to perform arithmetic, comparison, and synthesis across table entries.
• It comprises 120 manually curated QA pairs from diverse domains such as financial reports, sports statistics, survey results, and scientific tables, emphasizing reasoning beyond direct lookup.
• Empirical results reveal that a dual representation strategy combining OCR and natural-language narration significantly improves quantitative synthesis, highlighting key challenges in reasoning fidelity.

ReTabVQA is a Table VQA benchmark introduced in “TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription” (Yutong et al., 8 Oct 2025). It is described as a small but deliberately harder evaluation set designed to test whether a model can truly reason over table images rather than merely retrieve a visible value. In contrast to benchmarks that mostly emphasize direct lookup or straightforward extraction, ReTabVQA shifts evaluation toward compositional and multi-step quantitative reasoning, with questions that require arithmetic, comparison, and synthesis across multiple table entries.

1. Origin and motivation

ReTabVQA was motivated by a gap in prior table-image question answering benchmarks, especially TableVQA-Bench. The stated concern is that many existing questions can be answered by reading a single cell and returning it with little deeper inference. ReTabVQA was therefore constructed to probe a different capability profile: not only visual recognition and OCR, but also the ability to combine multiple values, preserve units, and execute quantitative reasoning chains over semi-structured tabular content (Yutong et al., 8 Oct 2025).

The benchmark’s design premise is that modern VLM and VLM+LLM systems often fail not because they cannot read the relevant table entries, but because they must combine those entries correctly. The paper characterizes the target behavior as “compositional and multi-step quantitative reasoning” and “multi-hop reasoning and quantitative synthesis across table entries.” A plausible implication is that ReTabVQA functions less as a scale benchmark than as a stress test for reasoning fidelity under visually grounded tabular evidence.

2. Dataset composition and construction

ReTabVQA is built by curating reasoning-oriented questions on top of TableVQA-Bench tables. The paper states that it contains 120 question–answer pairs derived from 60 table images, with 15 images in each of four domains: Financial Reports, Sports Statistics, Survey Results, and Scientific Tables. Each category contributes 30 QA pairs. The dataset is in English, since it is derived from the English TableVQA-Bench benchmark and uses English natural-language questions and answers (Yutong et al., 8 Oct 2025).

Domain	Tables	QA pairs
Financial Reports	15	30
Sports Statistics	15	30
Survey Results	15	30
Scientific Tables	15	30

The construction strategy is manual rather than procedural in the sense of a long annotation manual. For each source table image, the authors manually craft questions that require reasoning across multiple data points rather than asking for a single cell value. The questions are grounded in the original tables and are explicitly designed to force arithmetic and compositional inference.

The paper does not report a train/validation/test split for ReTabVQA. Instead, it is used as an evaluation benchmark. No separate split protocol is described, which makes the benchmark best understood as a curated diagnostic set for controlled evaluation rather than a conventional large-scale supervised dataset.

3. Reasoning profile and task definition

Within the general Table VQA formulation, the paper writes the task as

$A = f(T, Q),$

where $T$ is the table image, $Q$ is the question, and $A$ is the answer (Yutong et al., 8 Oct 2025). ReTabVQA is positioned as a harder instantiation of this task, one in which $f$ must support compositional reasoning over multiple table entries rather than direct extraction from a visible cell.

The question generation strategy explicitly targets operations such as multiplication, subtraction, percentage change, comparison, and synthesis across multiple rows and columns. The paper gives a concrete contrast. A simpler TableVQA-Bench-style question is “How many units were outstanding at December 31, 2007?”, answerable by direct lookup with 196. ReTabVQA uses the same source table to ask harder questions such as “What is the percentage change in the total fair value of outstanding units from 2006 to 2007?” and “For 2009, what was the net change in total fair value of units resulting from granted and vested/distributed activities?” These require computing totals from quantity times unit value, comparing values across years, and then performing percentage-change or subtraction calculations.

Operationally, the benchmark requires a model to identify relevant rows and columns, correctly associate headers and units, extract multiple values, perform arithmetic, and return the result in the right form. The case studies in the paper emphasize that errors can be amplified by contextual omissions such as missing units like “in millions” or “in thousands,” because a single extraction mistake can propagate through a multi-step computation.

4. Evaluation protocol and reported empirical results

Evaluation on ReTabVQA follows the same accuracy criterion used for TableVQA-Bench: accuracy is the metric, and a prediction is counted as correct if it contains the ground-truth answer (Yutong et al., 8 Oct 2025). This criterion is permissive in form, but the benchmark remains difficult because many answers are numeric and depend on derived quantities rather than copied spans.

The reported comparison on ReTabVQA is centered on a Generated OCR pipeline versus TALENT, using three Qwen2.5 model configurations.

Configuration	Generated OCR	TALENT
3B-3B	47.80%	52.50%
7B-7B	46.51%	55.06%
7B-3B	46.98%	55.00%

The reported gain is roughly 4.7–8.5 points, depending on configuration. The paper attributes this improvement to the dual-representation design: OCR spans provide precise symbolic content, while natural-language narration provides table-wide context that helps the LLM reason. The paper also notes that overall accuracy on ReTabVQA is lower than on the easier TableVQA-Bench setting, which it interprets as evidence that the reasoning burden is real. Its error analysis states that most failures are not due to retrieving the wrong cells, but due to incorrect numerical calculation after the relevant values have been identified.

The same paper connects this difficulty profile to a broader scaling analysis:

$A \approx 73.01 + 0.84 \log(S_V) + 2.66 \log(S_L),$

where $A$ is accuracy, $S_V$ is VLM parameter size, and $S_L$ is LLM parameter size. In the paper’s interpretation, once visual extraction is adequate, language-model reasoning power matters more than the vision side. ReTabVQA is presented as a setting where that asymmetry becomes especially visible because arithmetic and compositional reasoning dominate.

5. Relationship to TALENT

ReTabVQA is not an isolated dataset contribution; it is integral to the TALENT framework’s argument about representation. TALENT reframes Table VQA as an LLM-centric multimodal reasoning task in which a small VLM serves as a perception-narration module rather than as a monolithic solver. The paper defines the table representation as

$R(T) = \{\mathcal{O}(T), \mathcal{N}(T)\},$

where $T$0 is OCR output and $T$1 is natural-language narration (Yutong et al., 8 Oct 2025).

ReTabVQA is used to show why this dual representation is useful. A simple OCR string is presented as insufficient for multi-step arithmetic questions, while narration can help connect values, headers, and units in a reasoning-friendly form. This positioning is important because it shifts the bottleneck from perception alone to the interface between perception and reasoning. The benchmark thus serves as an empirical probe of whether a representation is aligned with downstream LLM reasoning requirements rather than merely with text extraction fidelity.

A common misconception is to treat ReTabVQA as just a more difficult OCR benchmark. The paper argues against that interpretation. Its analysis suggests that the dominant challenge is not reading isolated tokens, but performing quantitative synthesis over correctly retrieved evidence.

6. Distinctions from adjacent work and naming ambiguity

ReTabVQA should be distinguished from RE-Tab, introduced in “Enhancing TableQA through Verifiable Reasoning Trace Reward” (Kwok et al., 30 Jan 2026). RE-Tab is a plug-and-play framework that formalizes tool-based TableQA as a Partially Observable Markov Decision Process and adds verifiable reward feedback during state transition and simulative reasoning. Its central mechanism is TABROUGE, a reward over intermediate table states, and its empirical evaluations are conducted on WikiTQ, MMQA, and MMTU rather than on a table-image benchmark.

The proximity between the names can obscure a substantive methodological difference. ReTabVQA is a benchmark for table-image question answering with multi-step quantitative reasoning; RE-Tab is a reward-guided reasoning framework for stepwise table-state transformation. The former is primarily an evaluation resource; the latter is an inference-time control mechanism for TableQA agents. This suggests that the two contributions are complementary rather than interchangeable: one sharpens what is measured, while the other proposes how reasoning over tables can be guided.

In that sense, ReTabVQA occupies a specific place in the recent literature on structured visual reasoning. Its importance lies not in dataset scale, but in the difficulty profile it exposes: failures in computation, compositional inference, and context preservation that simpler benchmarks often miss.