Tough Tables: Analyzing Rich Tabular Complexity
- Tough Tables are complex tabular artifacts integrating heterogeneous data, annotations, and hierarchical formats to enable rich analytical reasoning.
- They are used widely across sectors where base data in spreadsheets and legacy systems ensures transparency, trust, and data provenance.
- Current benchmarks highlight challenges in structural robustness, long-context reading, and multi-step reasoning, spurring innovative table transformation methods.
“Tough Tables” denotes a line of research on tabular artifacts that remain analytically central precisely because they resist reduction to tidy, flat, uniformly machine-readable inputs. In the foundational qualitative study “Untidy Data: The Unreasonable Effectiveness of Tables,” tables are treated not as a disposable preprocessing stage but as a visualization and sensemaking idiom in their own right, especially for data workers who work with data as part of their job but do not identify as professional analysts or data scientists (Bartram et al., 2021). Later work extends this framing into benchmarks and methods for large, heterogeneous, hierarchical, semi-structured, and visually irregular tables, showing that current LLMs and MLLMs still struggle with long-context table reading, hierarchy perception, structural grounding, robustness to serialization, code-based transformation, and real-world table images (Xing et al., 5 Jun 2025, Wu et al., 16 Jun 2025, Huang et al., 1 May 2026, Ashury-Tahan et al., 26 Feb 2025, Abhyankar et al., 6 Nov 2025).
1. Conceptual foundations
The central claim of the Tough Tables literature is that tables are not merely “tidying” devices on the way to “real” analytics. Rather, they are living analytical artifacts that persist throughout the workflow and support organization, verification, experimentation, communication, and interpretation. In the study underlying this claim, 12 participants were recruited from nonprofit leadership, legal services, retail, GIS, financial services, public health, consulting, software, and operations; none had formal training in data analytics or data science. Excel and Google Sheets were ubiquitous even among participants who also used Salesforce, Tableau, Tableau Prep, Shopify, PeopleSoft, web scrapers, proprietary systems, or SQL workbenches. The researchers conducted remote Zoom sessions combining semi-structured interviews, sketching, and walkthroughs of a current or recent data task, and analyzed transcripts, videos, sketches, and notes with thematic coding (Bartram et al., 2021).
This foundation matters because it redefines what counts as analytic work. The paper argues that users want to see and “get their hands on” the underlying data throughout the analytics process, not only at the start of a linear cleanup pipeline. Manual manipulation, even when tedious, is described as part of how trust, confidence, and understanding are built. A common misconception addressed explicitly in this literature is that spreadsheet persistence signals lack of sophistication. The empirical claim is the opposite: spreadsheets remain prevalent because they afford directness, transparency, flexibility, and cognitively important human-readable structure that many BI and visual analytics systems abstract away (Bartram et al., 2021).
2. Rich tables, base data, and direct manipulation
A key theoretical construct in this literature is the “rich table.” Rich tables are tabular artifacts entangled with human sensemaking and therefore contain more than clean, machine-readable values. Their features may include wide, human-readable layout rather than tidy tall data; heterogeneity in cell meanings, formats, and values; missing or sparse cells; multiple datasets or worksheets; multiple data grains in the same workbook; and annotations, comments, formatting, and other spatial or visual marks. These are framed not as defects to be erased, but as productive resources for cognition and interaction (Bartram et al., 2021).
The most important table in many workflows was a “base data” or “master table” containing the core, lowest-level data needed for work. Participants often kept this table intact and distinct from derived views, treating it as a source of truth and a repository of provenance. This separation supported exploratory reasoning because users could manipulate a copy, check formulas, and cross-check outputs against the base table to understand where results came from. The master-table pattern also created a stable reference point for alternatives and variants, which the paper presents as cognitively important rather than merely procedural (Bartram et al., 2021).
Within and around these base tables, several recurrent practices were observed. Participants reorganized data spatially through sorting, filtering, hiding, grouping, reordering, multi-column headers, and adjacent summary zones. They marked up tables through “marginalia” such as notes, derived columns, provenance explanations, and bespoke categories, and through “annotations” embedded in the data body itself, such as highlighting, color coding, or symbols like dashes for missing values. They layered raw row-level data, summaries, rollups, and totals in the same tabular environment, and they spawned alternatives through copies, separate sheets, and “what-if” variants. The article’s broader implication is that tough tables are difficult not because they are malformed in a simple syntactic sense, but because they simultaneously encode data, transformations, provenance, uncertainty, and intent in one manipulable surface (Bartram et al., 2021).
3. Structural sources of difficulty
Subsequent work operationalizes tough tables as objects that violate simple assumptions such as “each row is a record, each column is an attribute.” “Auto-Tables” reports that over 30% of real spreadsheet-tables and web-tables in its survey do not conform to the relational standard, and its detailed summary states that roughly 30–50% of sampled real spreadsheet and web tables have such issues. The recurring patterns include horizontal homogeneity or header-as-data, repeating column groups, rows used as attributes or tables that need transposition, repeating row groups, composite cells, structural blanks, and subtitle rows that must be converted into a column (Li et al., 2023). This places table restructuring itself inside the Tough Tables problem, rather than outside it.
More recent benchmarks broaden the definition of difficulty. RealHiTBench targets hierarchical column headers, hierarchical row headers, nested sub-tables, explicit and implicit multi-table join, and miscellanies such as supplementary explanatory text and background colors (Wu et al., 16 Jun 2025). RUST-BENCH emphasizes semi-structured tables that mix structured fields with free text and require multi-hop reasoning across thousands of tokens; its benchmark tables average 45.1 rows overall and approximately 23K tokens per table, with more detailed statistics of 18,304.47 tokens per table in RB-Sports and 31,948.79 in RB-Science (Abhyankar et al., 6 Nov 2025). WildTableBench moves from structured text and clean rendered tables to naturally occurring table images with merged cells, uneven row and column spans, missing or partial borders, screenshots, scans, photos, blur, skew, occlusion, dense layouts, diverse domains, and color-dependent reasoning (Huang et al., 1 May 2026).
A plausible implication is that “toughness” in tables is multidimensional. Some tables are tough because they are rich human artifacts carrying layout, annotations, and multiple grains. Others are tough because they are structurally non-relational, hierarchically nested, visually degraded, or semi-structured. Still others are tough because semantically equivalent serializations induce unstable model behavior. The literature consistently treats these as real properties of deployed tables rather than benchmark noise (Bartram et al., 2021, Li et al., 2023, Wu et al., 16 Jun 2025, Huang et al., 1 May 2026).
4. Benchmarking tough tables
The benchmark literature makes the Tough Tables agenda explicit by evaluating expert-level table understanding, reasoning, transformation, and robustness under realistic conditions.
| Benchmark | Scale | Core focus |
|---|---|---|
| MMTU | 30,647 questions; 67,886 real tables | 25 task categories over real-world table tasks |
| RealHiTBench | 708 tables; 3,752 QA pairs | Hierarchical and realistic tables across LaTeX and PNG |
| WildTableBench | 402 table images; 928 questions | In-the-wild table image QA |
| RUST-BENCH | 7,966 questions; 2,031 tables | Semi-structured long tables with free text |
| ToRR | 10 datasets; 35 prompt configurations | Table reasoning and robustness across formats |
MMTU is the broadest task inventory in this group. It contains 30,647 questions over 67,886 real tables, organized into 25 task categories drawn from 52 datasets. Its tables come from web tables, spreadsheets, and relational tables, and the benchmark caps any single dataset at 1,000 questions. It includes 8,508 coding questions, consisting of 3,289 SQL questions, 1,593 Python/Pandas questions, and 3,626 spreadsheet-formula questions, and 22,139 non-coding questions. The average table size is 2,659 rows, 11 columns, and 33,251 cells. Even frontier reasoning models remain far from saturation: OpenAI o4-mini scores , DeepSeek-R1 scores , and the best chat models are lower, with GPT-4o at (Xing et al., 5 Jun 2025).
RealHiTBench was designed because earlier datasets were often dominated by flat tables, limited to a single modality, or too narrow in domain. It contains 708 tables and 3,752 QA pairs from 13 open platforms across 24 domains, with 127 tables so large that they are difficult to fit in a single conversation window. It introduces Structure Comprehending as a task specifically targeted at complex hierarchical tables, and reports that Chart Generation is the hardest task, with many models having PASS@1 below 30 and some near 0. Textual representations, especially LaTeX, generally outperform PNG-based visual inputs; GPT-4o with text input beats image input by about 15 points on average, and Gemini-1.5-Pro text beats image by about 10 points (Wu et al., 16 Jun 2025).
WildTableBench shifts attention from text-serialized tables to naturally occurring table images. It comprises 402 high-information-density table images and 928 manually annotated and verified questions across 17 subtypes in five categories. Only one evaluated model exceeds 50% accuracy, with Gemini-3-Pro at 67.9%; all remaining models range from 4.1% to 49.9%. Its diagnostics show a persistent top-left positional advantage and a bottom-right disadvantage in cell retrieval, and its error analysis concludes that the main bottleneck is reliable visual grounding, especially locating and recognition, rather than reasoning alone (Huang et al., 1 May 2026).
RUST-BENCH targets semi-structured tables that combine schema-like fields with unstructured text. It contains 7,966 questions from 2,031 real-world tables across RB-Science and RB-Sports, and its reasoning categories are non-exclusive, with Filtering/Selection at 75.89%, Temporal reasoning at 39.33%, Logical reasoning at 27.85%, Numerical at 26.83%, and Multi-hop reasoning at 26.18%. The paper reports that performance drops beyond about 35K–50K tokens and that even strong systems struggle with heterogeneous schemas, long-range evidence integration, and unanswerable detection; Qwen-QwQ reaches the top reported LLM-scores of 54.1 on RB-Science and 55.7 on RB-Sports (Abhyankar et al., 6 Nov 2025).
ToRR frames tough tables as a robustness problem as well as an accuracy problem. It spans 10 datasets, 6 tabular tasks, and 35 prompt configurations built from 7 table serializations and 4 structural perturbations. Its main result is brittleness: even strong models vary substantially across semantically equivalent representations. No serializer consistently outperforms others, and testing multiple prompt configurations matters because moving from 1 to 10 prompts increases Kendall’s by more than 0.35 on average. The benchmark’s overall leaders remain moderate rather than strong, with for claude-3-5-sonnet, gpt-4o, and deepseek-v3, and for claude-3-5-sonnet (Ashury-Tahan et al., 26 Feb 2025).
5. Methods for working with tough tables
One research direction treats tabular structure itself as a scaffold for reasoning. “Thinking with Tables” introduces an in-context prompting method in which the model is encouraged to organize relevant information into a table before answering a complex request with multiple conditions. Across eight LLMs and six request types, the method yields a 40.29 percent average relative performance gain, corresponding to +5.34 percentage points on average, with the largest absolute gain on Update at +11.96 percentage points. The paper also reports attention-value analysis on Llama-3.1-70B for Retrieval tasks, with text at and table at , supporting the claim that tabular structure helps the model focus more strongly and consistently on relevant features (Oh et al., 2024).
A second direction builds explicit reasoning structures over large tables. “Tree-of-Table” combines Table Condensation, Table Decomposition, hierarchical Table-Tree construction, and Table-Tree Execution. On BIRD, which the paper describes as spanning 37 domains and 33.4 GB of data with over 12,000 examples, more than 70% of questions involve tables longer than current LLM input limits, more than 90% require at least two tables, and about 20% require four or more tables. After condensation, tables for more than 60% of long BIRD questions fall below the model input limit. The method reports strong results across WikiTQ, TabFact, FeTaQA, and BIRD, including BIRD BLEU 15.70, ROUGE-1 0.53, ROUGE-2 0.26, and ROUGE-L 0.52 (Ji et al., 2024).
RealHiTBench’s TreeThinker focuses specifically on hierarchical tables. It converts headers into an explicit tree structure, aligns question keywords with header tuples, retrieves relevant sub-tables, and adds a ReAct-style multi-round refinement process. The paper reports, for GPT-4o, Numerical Reasoning F1 improving from 36.68 to 49.35 in one image+text setting, and Chart Generation PASS@1 improving from 14.29 to 33.55, a 134.7% improvement. Its ablations state that removing Tree Generation causes a large drop and that Tree Generation is often the most important component (Wu et al., 16 Jun 2025).
Other methods target transformation and interaction rather than QA alone. Auto-Tables formulates relationalization as synthesis over an eight-operator DSL consisting of stack, wide-to-long, transpose, pivot, explode, ffill, subtitles, and none. It compiles ATBench with 244 real test cases, including 218 single-step and 26 multi-step cases, and reports Hit@1 , Hit@2 , Hit@3 0, with 0.224s average latency (Li et al., 2023). TaFo treats conditional formatting as a predictive problem and uses a neuro-symbolic pipeline to infer both the trigger condition and the formatting properties. Trained on 1.8 million public workbooks and evaluated on 105K conditional-formatting tasks, it reports 36.7% / 46.8% / 64.3% execution match at top-1 / top-3 / top-5 for condition learning and 26.5%, 47.2%, and 55.4% for format learning (Singh et al., 14 Aug 2025).
Taken together, these methods indicate that tough-table research is not reducible to one task family. It includes prompt-time restructuring of evidence, hierarchical reasoning over headers and sub-tables, synthesis of multi-step transformations, and predictive assistance within spreadsheet-like interaction loops.
6. Misconceptions, failure modes, and design implications
A persistent misconception is that tables are merely preparatory and should disappear once visualization or modeling begins. The qualitative evidence argues instead that users want direct access to base data throughout the process, and that reorganizing, annotating, and copying tables are part of sensemaking itself (Bartram et al., 2021). A related misconception is that spreadsheet-centric practice reflects resistance to more advanced tools. The literature presents spreadsheet use as a rational response to transparency and trust requirements that many current visual analytics systems do not satisfy (Bartram et al., 2021).
Another misconception is that tougher table reasoning is primarily a matter of larger context windows. Several benchmarks show that this is incomplete. MMTU reports a “needle-in-a-haystack-in-table” effect in which accuracy falls sharply as the column index increases and drops below 0.5 when tables have more than 25 columns, even though ordinary NLP needle-in-a-haystack tests are nearly solved (Xing et al., 5 Jun 2025). WildTableBench shows that locating and recognition dominate error breakdowns in real table images (Huang et al., 1 May 2026). RealHiTBench argues that hierarchy perception is crucial, and RUST-BENCH shows that semi-structured mixtures of schema and free text defeat both rigid symbolic pipelines and purely text-centric prompting (Wu et al., 16 Jun 2025, Abhyankar et al., 6 Nov 2025).
Robustness is also a central controversy. ToRR finds that no table format is consistently best, that perturbations do not consistently help or hurt, and that single-format evaluation is unreliable. It further shows that 50 examples with 2 prompt configurations can yield similar reliability to 100 examples with 1 prompt configuration (Ashury-Tahan et al., 26 Feb 2025). This suggests that the evaluation of tough tables must measure stability across semantically equivalent representations, not only peak accuracy on one serialization.
The design implications are correspondingly concrete. The foundational table-interaction work argues that visual analytics systems should preserve rich human-legible structure, support wide and heterogeneous tables, retain annotations and spatial organization, and make it easy to keep a stable master table alongside mutable exploratory copies. It also calls for consequence-free experimentation, explicit data-flow visualization, clear representations of joins, pivots, and aggregations, no-code support for comparing alternative scenarios, and scaffolding for new users (Bartram et al., 2021). In the broader benchmark-and-methods literature, this agenda expands to hierarchy-aware reasoning, serialization-robust evaluation, explicit support for semi-structured and image-based tables, and methods that treat tables not as flattened text, but as structured environments in which understanding is built (Oh et al., 2024, Ji et al., 2024, Wu et al., 16 Jun 2025, Huang et al., 1 May 2026).
In this sense, the core message of Tough Tables is consistent across human-centered and model-centered work. Tables are tough not because they are defective remnants awaiting cleanup, but because they are one of the main places where analytical structure, provenance, uncertainty, and interpretation are made visible and manipulated.
