TableLong: Structured Tables for Long-Context Reasoning

• TableLong is a data-centric pipeline that leverages structured tables to reinforce long-context reasoning in language models through RL post-training.
• It synthesizes high-quality, SQL-grounded table-question-answer triplets from diverse real-world environments to ensure verifiability and multi-hop reasoning.
• Empirical results show significant improvements, with gains of +8.24% on long-context benchmarks and robust out-of-domain performance.

TableLong is a data-centric pipeline for improving long-context reasoning in LLMs by using structured table data as reinforcement-learning post-training data rather than treating tables only as downstream inputs. It is motivated by the claim that structured table data with periodic structures shows strong potential for long-context reasoning, and it formalizes that claim through a mutual-information analysis of row-major table linearizations. On that basis, TableLong synthesizes high-quality, diverse, and verifiable table-question-answer data from executable table environments, then uses RL to improve retrieval, grounding, and multi-hop reasoning over long contexts. The reported effect is broad rather than narrowly tabular: the paper states gains of +8.24\% on average on long-context benchmarks and +8.06\% on average on out-of-domain benchmarks (Xie et al., 23 Mar 2026).

1. Problem formulation and conceptual scope

TableLong studies a specific question in long-context post-training: what kinds of training data are especially effective for improving long-context reasoning, and why. The target capability is not only long-context retrieval, but the more general ability to locate relevant information across long inputs, ground it correctly, and perform reasoning over it. The paper argues that existing work has focused more on training strategies and RL recipes than on the intrinsic properties of the training data itself, and it proposes that structured tables are unusually suitable because they provide long-range dependencies, scalability in input length, verifiability of answers, support for multi-hop reasoning, and strong grounding signals (Xie et al., 23 Mar 2026).

Within this framework, tables are not introduced primarily as a benchmark format. They are introduced as a source of post-training data. The practical rationale is that tables are easy to scale, easy to generate or collect, and easy to verify through SQL execution. The structural rationale is that, once linearized, cells from the same column recur at regular intervals, creating what the paper calls periodic non-vanishing dependencies. This contrasts with natural text, where long-distance dependencies are assumed to decay with distance. A central implication is that table-derived RL data may train models to maintain useful dependencies over long ranges more effectively than ordinary text does (Xie et al., 23 Mar 2026).

The task setting is organized around executable table environments. Using those environments, TableLong constructs question-answering instances whose answers are grounded in SQL execution. The pipeline therefore emphasizes three properties simultaneously: long context, nontrivial reasoning, and exact verifiability. This suggests that TableLong is best understood as a method for designing long-context training data, not as a new inference-time table reasoner or a new table parser.

2. Information-theoretic basis

The theoretical core of TableLong is an information-theoretic comparison between natural-language sequences and row-major linearized tables. For natural language, the paper adopts the empirical assumption that mutual information decays with distance $d$ according to

$$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$

so that

$\lim_{d \rightarrow \infty} I_{text}(d) = 0.$

A table $\mathcal{T}$ is modeled as an $n \times m$ array with headers $\mathcal{H}=\{H_1,\ldots,H_m\}$ and cells $T_{i,j}$, and it is linearized row-major as

$$\phi(\mathcal{T}) = [H_1, \ldots, H_m, T_{1,1}, \ldots, T_{1,m}, T_{2,1}, \ldots, T_{2,m}, \ldots, T_{n,m}]$$

with total sequence length

$L = m(n+1).$

For positions in the data region, the column index is

$\text{col}(t) = ((t - m - 1) \mod m) + 1 \in [m],$

and the average mutual information at lag $$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$0 is defined as

$$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$1

The analysis assumes Column Semantic Consistency,

$$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$2

and Column Distribution Distinctiveness,

$$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$3

A key quantity is the same-column mutual information

$$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$4

where $$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$5 is a uniformly random column. Under the stated assumptions, the paper proves that $$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$6 and that it is independent of row distance. The main theorem then states

$$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$7

This means that when the lag is a multiple of the column count, the average mutual information attains a periodic peak of constant height. The table sequence therefore contains regular, non-decaying long-range dependencies: the same-column cells are repeatedly aligned at distances $$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$8 (Xie et al., 23 Mar 2026).

The paper draws several asymptotic consequences. For tables with fixed $$I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5$$9 and growing $\lim_{d \rightarrow \infty} I_{text}(d) = 0.$0,

$\lim_{d \rightarrow \infty} I_{text}(d) = 0.$1

whereas natural text satisfies asymptotic decay. It also defines the effective dependency distance

$\lim_{d \rightarrow \infty} I_{text}(d) = 0.$2

and concludes that

$\lim_{d \rightarrow \infty} I_{text}(d) = 0.$3

The appendix extends this with a uniform attention requirement, according to which all prior same-column tokens remain informative, and a context length requirement, according to which the first and last cells of a column can still have mutual information $\lim_{d \rightarrow \infty} I_{text}(d) = 0.$4 despite being separated by $\lim_{d \rightarrow \infty} I_{text}(d) = 0.$5 positions. In effect, the theory frames table linearizations as training sequences that repeatedly force retrieval and reasoning over distributed but systematically related evidence.

3. Data synthesis pipeline

TableLong is implemented as a three-stage pipeline. The first stage initializes environments, the second constructs samples, and the third verifies and filters them. The resulting training instances are SQL-grounded and answer-verifiable (Xie et al., 23 Mar 2026).

Stage	Operation	Output
1	Environment initialization	SQLite-backed table environments
2	Sample construction	$\lim_{d \rightarrow \infty} I_{text}(d) = 0.$6 triplets
3	Verification and filtration	Retained samples with $\lim_{d \rightarrow \infty} I_{text}(d) = 0.$7

In the first stage, the paper aggregates over 10,000 real-world tables from BIRD, CoSQL, and Spider, plus additional scalable table data extracted from information-dense documents using LLMs. After cleaning and sampling, the corpus spans Finance, Sports, Healthcare, Science, etc., includes English and Chinese, covers context lengths from hundreds to 32k tokens, has an average cell density of 5.2 tokens, and ranges from single-table to 30-table instances. All tables are parsed into a unified SQLite environment (Xie et al., 23 Mar 2026).

In the second stage, LLMs generate SQL queries, natural-language questions, and answers obtained by executing the SQL. Each sample is a triplet

$\lim_{d \rightarrow \infty} I_{text}(d) = 0.$8

The SQL is generated along three task dimensions. Precise Retrieval restricts SQL to SELECT with specific WHERE clauses, forcing exact location of atomic information. Multi-hop Reasoning introduces aggregation such as SUM and AVG, forcing reasoning over many dispersed cells. Grounding introduces multi-table operations such as JOIN, forcing the model to associate the question with the correct tables among many candidates. The appendix gives a prompt that requires the generator to use real schema table names, avoid explicit SQL jargon, produce contextual and natural questions, and output only JSON of the form:

$n \times m$5

This stage therefore couples executable structure with natural-language supervision (Xie et al., 23 Mar 2026).

In the third stage, execution provides exact answers, and a consistency-based filtration mechanism retains only samples that are neither trivial nor unusable. For each instance $\lim_{d \rightarrow \infty} I_{text}(d) = 0.$9, the system generates $\mathcal{T}$0 candidate responses and computes a pass rate $\mathcal{T}$1 against the ground-truth answer. It discards $\mathcal{T}$2 instances as likely ambiguous, erroneous, or too hard, discards $\mathcal{T}$3 instances as too easy, and retains only $\mathcal{T}$4. The paper emphasizes that this makes the data simultaneously high-quality, diverse, scalable, and verifiable (Xie et al., 23 Mar 2026).

4. Reinforcement-learning formulation and training setup

The synthesized data is used for reinforcement-learning post-training with GRPO. The paper trains four backbones: Qwen3-32B, Qwen2.5-32B-Instruct, Deepseek-R1-Distill-Qwen-14B, and Deepseek-R1-Distill-Qwen-32B. Training is conducted with 64 NVIDIA H20 GPUs in the Verl framework (Xie et al., 23 Mar 2026).

The appendix gives the GRPO objective as

$\mathcal{T}$5

with token-level importance ratio

$\mathcal{T}$6

and normalized group advantage

$\mathcal{T}$7

The reward is described as outcome-based. Candidate responses are judged for semantic consistency with the executable ground truth, using gpt-oss-120B as the judge model. The paper does not provide a more granular symbolic reward decomposition beyond correctness and semantic consistency. It also uses dynamic rollout sampling to keep only informative groups with non-zero advantage variance, and it adopts a clip higher strategy from DAPO to accelerate convergence on positive signals while remaining safe on negative ones (Xie et al., 23 Mar 2026).

The reported hyperparameters are explicit. The training system uses TP = 8 with offload disabled. Maximum sequence capacity is 32,768, with maximum input length 28,672 and maximum output length 4,096. The global batch size is 128, the mini-batch size 32, the learning rate $\mathcal{T}$8, and the KL coefficient $\mathcal{T}$9. The rollout batch size is 256, the group size $n \times m$0, temperature 0.85, and top-$n \times m$1 1.0. The clipping ratios are $n \times m$2 and $n \times m$3 (Xie et al., 23 Mar 2026).

5. Empirical performance, scaling behavior, and ablations

The principal evaluation covers seven long-context benchmarks: LongBench-v2, Loong, BrowsCompLong, GSM-Infinite, Oolong-Synth, Ruler (QA2 subset), and MRCR. The main result is consistent improvement across all four backbones (Xie et al., 23 Mar 2026).

Backbone	Base avg	+ Ours avg
DS-R1-Distill-32B	40.12	48.36
DS-R1-Distill-14B	36.14	45.07
Qwen2.5-32B-Instruct	36.15	40.94
Qwen3-32B	40.40	45.34

For DS-R1-Distill-32B, the reported average gain is +8.24. The per-benchmark values are 42.35 → 45.73 on LongBench-v2, 38.17 → 45.30 on Loong, 64.22 → 74.31 on BrowsCompLong, 6.60 → 14.80 on GSM-Infinite, 39.43 → 51.41 on Oolong-Synth, 58.11 → 66.41 on Ruler, and 31.94 → 40.57 on MRCR. The paper also states that DS-R1-Distill-32B + Ours clearly outperforms Qwen-Long-L1 and approaches Deepseek-v3.1 (Xie et al., 23 Mar 2026).

The retrieval effect is especially clear in Needle-in-a-Haystack experiments. DS-R1-Distill-14B improves from 69.30% to 91.20%, a +31.60% gain. DS-R1-Distill-32B improves from 87.95% to 99.40%, a +13.02% gain. The paper interprets these results as evidence that TableLong improves retrieval robustness across depth positions in long documents (Xie et al., 23 Mar 2026).

The out-of-domain results are also positive. On GPQA-Diamond, AIME 2025, MultiChallenge, and LiveCodeBench, DS-R1-Distill-32B improves from 56.06 → 63.64, 60.00 → 70.00, 30.28 → 32.97, and 46.71 → 58.68, respectively. Averaged over these four tasks, the gain is +8.06, which matches the paper’s summary. The appendix also presents qualitative case studies in which the TableLong-trained model resolves schema-alignment errors in table reasoning and corrects flawed enumeration strategies in an AIME 2025 combinatorial problem, suggesting transfer beyond table-shaped tasks (Xie et al., 23 Mar 2026).

The scaling experiments analyze what properties of table data matter. Training on longer data helps: for DS-R1-Distill-32B, the average score rises from 46.97 at 4k to 47.45 at 8k and 48.36 at 16k. The paper further states that models trained within 16k generalize beyond 16k to 32k, 64k, and even 128k evaluation ranges. Increasing multi-hop complexity also helps: for the same backbone, average score rises from 46.30 with 0–30 cells involved, to 46.50 with 0–100, and 48.36 with 0–300+. Increasing grounding difficulty has a similar effect: 46.66 for 1 table, 47.11 for 1–5 tables, and 48.36 for 1–30 tables (Xie et al., 23 Mar 2026).

The structural ablations are particularly revealing. When semantics are removed but tabular structure is preserved, performance still improves over baseline by about +1.67% for 32B and +1.66% for 14B, indicating that structure alone helps. However, compared with full TableLong, the no-semantics condition drops by 6.57% on average and converges prematurely during RL. Removing visible delimiters causes only about a 0.77% drop, and replacing delimiters with irrelevant noise produces similar results. The paper uses these findings to argue that visible formatting markers are not the main source of the gains; the gains are attributed to deeper structural organization and the interaction between structure and semantics (Xie et al., 23 Mar 2026).

6. Position within table research, limitations, and open questions

TableLong occupies a distinct position within the broader literature on tables and LLMs. Tree-of-Table is a prompt-based, inference-time framework for using general LLMs on table understanding tasks when the underlying tables are too large, too relational, or too reasoning-heavy for flat prompting, and it addresses large-table reasoning through condensation, decomposition, and tree-structured execution (Ji et al., 2024). TAMO instead argues that tables should be treated as an independent modality integrated with text tokens via a hypergraph encoder and a decoder-only LLM (Li et al., 30 Nov 2025). TABVERSE studies how the same table content behaves across HTML, Markdown, LaTeX, and rendered images, and concludes that representation choice substantially affects table understanding, with HTML often the most robust text format and LaTeX remaining challenging (Ahsan et al., 8 Jun 2026). FastTab and StrucTab address table structure recognition and table parsing from images, emphasizing explicit grid reconstruction, intermediate structural supervision, and decomposed reward optimization rather than long-context RL data design (Hamdi et al., 21 May 2026, Li et al., 29 Jun 2026). TabRecSet is a dataset and annotation framework for end-to-end table recognition in the wild (Yang et al., 2023). Older works such as AutoTableComplete, which studies KB-driven table completion from tabular queries, and the fast large-table grid UI method based on interpolation over indexed keys, solve still different problems (Bandyopadhyay et al., 2019, Ponomarev, 2016). This suggests that TableLong is best understood as a method for long-context post-training data construction rather than a method for table parsing, table representation benchmarking, or inference-time table reasoning.

The limitations stated for TableLong are correspondingly specific. The paper focuses only on structured tabular data. It does not explore other modalities such as documents or graphs, does not study coupling effects between data types, and does not develop specialized RL algorithms specifically for table-induced reasoning (Xie et al., 23 Mar 2026). The paper also does not present a direct SFT-versus-RL ablation in the provided text. Its open questions are therefore largely about generalization of the data-design principle: how other data types compare, how multiple RL data types should be mixed, and whether specialized RL algorithms could amplify the observed gains.

Taken as a whole, TableLong advances a strong claim about long-context learning: effective post-training is not only a matter of larger context windows or more aggressive RL recipes, but also of choosing data with the right dependency structure. Its most distinctive contribution is to connect that claim to a concrete theorem,

$n \times m$4

and then to show that a simple, scalable, and verifiable synthesis pipeline built around that structure produces measurable gains in retrieval, grounding, multi-hop reasoning, and broader long-context generalization (Xie et al., 23 Mar 2026).