--- title: StructQA Benchmark for Table Reasoning url: https://www.emergentmind.com/topics/structqa-benchmark type: topic --- # StructQA Benchmark for Table Reasoning StructQA Benchmark is a diagnostic suite introduced to evaluate large language models (LLMs) on structural reasoning over tabular data, focusing on semantic understanding of tabular structure and permutation invariance. The benchmark emphasizes minimal assumptions about real-world table heterogeneity and aims to quantify model performance beyond plain row/column serialization by explicitly measuring whether models recognize underlying table structure. StructQA underpins the TAMO (Table As a Modality) framework and is specifically constructed to reveal systematic failings of text-only table serialization approaches, including state-of-the-art LLMs such as GPT-4 and open-source competitors [2512.00947]. ## 1. Dataset Definition and Construction The StructQA dataset is composed of 500 real-world flat Wikipedia tables sampled from the WikiTableQuestions corpus. Each table is paired with five distinct structural reasoning tasks, and for each task, three natural-language question templates are defined. This yields 5 × 3 × 500 = 7,500 question–answer pairs. Flatness here denotes the absence of nested hierarchies beyond column header groupings; tables include only leaf cells containing text, numerical values, or dates. No missing-cell or corrupted-cell cases are present. The schema broadly covers categorical, quantitive, and temporal data as found in Wikipedia’s open-domain tables. The benchmark is split into 60% train, 20% development, and 20% test partitions, preserving table diversity and type variety. Task categories are: 1. **Cell location**: “What is the value in column C of row r?” 2. **Column lookup**: “In row r, which column(s) contain value v?” 3. **Row lookup**: “In column C, which row(s) contain value v?” 4. **Column comprehension**: “List all distinct values in column C.” 5. **Row comprehension**: “List all values in row r across all columns.” These tasks are specifically designed to require non-trivial access to table structure, demanding that models recognize column–row relationships and set-based operations rather than relying on simple text pattern matching [2512.00947]. ## 2. Evaluation Protocol and Metrics StructQA uses three main metrics to quantify model performance: - **Exact-match accuracy**: $$ \mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl[\hat A_i = A_i\bigr] $$ where $N$ is the number of test questions, $\hat{A}_i$ is the predicted answer, and $A_i$ is the ground-truth answer for the $i$-th instance. - **Permutation accuracy**: $$ \mathrm{Acc}_{\pi} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl[\hat A_i^{\pi} = A_i\bigr] $$ where $\hat{A}_i^{\pi}$ is the model prediction on tables with rows and columns independently shuffled. - **Robustness**: $$ \mathrm{Robust} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl[\hat A_i = \hat A_i^{\pi}\bigr] $$ capturing how often model outputs remain identical before and after table permutation (permutation invariance). These metrics are deliberately designed to penalize models that overfit to serialization order and to assess genuine structural understanding [2512.00947]. ## 3. Baseline Model Performance A diverse set of LLMs and fine-tuning strategies were evaluated on StructQA, including zero-shot text-serialization, prompt-tuning, LoRA, full supervised fine-tuning (SFT), and the multimodal TAMO architecture. The table below summarizes core reported results as measured on the test set [2512.00947]: | Setting | Model | Acc (%) | Perm Acc (%) | Robust (%) | |----------------|---------------|:-------:|:------------:|:----------:| | Zero-shot | Pure text | 8.60 | 7.19 | 16.3 | | Prompt-tuning | Pure text | 37.80 | 29.93 | 31.1 | | | TAMO | 59.07 | 43.47 | 43.8 | | LoRA | Pure text | 45.67 | 35.87 | 39.7 | | | TAMO | 70.80 | 42.77 | 53.7 | | SFT | Pure text | 62.73 | 54.80 | 51.6 | | | TAMO | 71.60 | 63.89 | 64.1 | Commercial LLMs (GPT-3.5, GPT-4, GPT-4.1) and strong open models (DeepSeek-R1) exhibit moderate performance (e.g., GPT-4: 51.40%). Notably, specialist baselines such as TableLlama obtain much lower numbers, underscoring that vanilla table-focused models may not generalize to StructQA’s emphasis on structural diagnosis [2512.00947]. ## 4. TAMO Methodology Applied to StructQA The TAMO (Table As a Modality) architecture demonstrates the potential of multimodal table understanding by combining a hypergraph neural network table encoder with a standard LLM. Each table $\mathcal{T}$ is modeled as a hypergraph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, with nodes for leaf cells and hyperedges for rows, columns, and headers. Hypergraph Transformer layers propagate and aggregate structural information: - **Hyperedge update**: $$ h_e^{(l+1)} = \sigma \left( W_1 h_e^{(l)} + \sum_{v \in e} W_2 h_v^{(l)} \right) $$ - **Node update**: $$ h_v^{(l+1)} = \sigma \left( U_1 h_v^{(l)} + \sum_{e \ni v} U_2 h_e^{(l+1)} \right) $$ Structural embeddings are mean-pooled and mapped to a soft-prompt prefix for the LLM, and used jointly with serialized text and the question for answer generation: $$ p(\mathcal{A}|\mathcal{T}, \mathcal{Q}) = \prod_{i=1}^n p(a_i | \mathbf{X}_{st},\,\mathbf{X}_{tt},\,\mathbf{X}_{qt},\,a_{