--- title: 'TableLong: Structured Tables for Long-Context Reasoning' url: https://www.emergentmind.com/topics/tablelong type: topic --- # TableLong: Structured Tables for Long-Context Reasoning TableLong is a data-centric pipeline for improving long-context reasoning in large language models by using structured table data as reinforcement-learning post-training data rather than treating tables only as downstream inputs. It is motivated by the claim that structured table data with periodic structures shows strong potential for long-context reasoning, and it formalizes that claim through a mutual-information analysis of row-major table linearizations. On that basis, TableLong synthesizes high-quality, diverse, and verifiable table-question-answer data from executable table environments, then uses RL to improve retrieval, grounding, and multi-hop reasoning over long contexts. The reported effect is broad rather than narrowly tabular: the paper states gains of **+8.24\% on average** on long-context benchmarks and **+8.06\% on average** on out-of-domain benchmarks [2603.21719]. ## 1. Problem formulation and conceptual scope TableLong studies a specific question in long-context post-training: what kinds of training data are especially effective for improving long-context reasoning, and why. The target capability is not only long-context retrieval, but the more general ability to locate relevant information across long inputs, ground it correctly, and perform reasoning over it. The paper argues that existing work has focused more on training strategies and RL recipes than on the intrinsic properties of the training data itself, and it proposes that structured tables are unusually suitable because they provide long-range dependencies, scalability in input length, verifiability of answers, support for multi-hop reasoning, and strong grounding signals [2603.21719]. Within this framework, tables are not introduced primarily as a benchmark format. They are introduced as a source of post-training data. The practical rationale is that tables are easy to scale, easy to generate or collect, and easy to verify through SQL execution. The structural rationale is that, once linearized, cells from the same column recur at regular intervals, creating what the paper calls **periodic non-vanishing dependencies**. This contrasts with natural text, where long-distance dependencies are assumed to decay with distance. A central implication is that table-derived RL data may train models to maintain useful dependencies over long ranges more effectively than ordinary text does [2603.21719]. The task setting is organized around executable table environments. Using those environments, TableLong constructs question-answering instances whose answers are grounded in SQL execution. The pipeline therefore emphasizes three properties simultaneously: long context, nontrivial reasoning, and exact verifiability. This suggests that TableLong is best understood as a method for designing long-context training data, not as a new inference-time table reasoner or a new table parser. ## 2. Information-theoretic basis The theoretical core of TableLong is an information-theoretic comparison between natural-language sequences and row-major linearized tables. For natural language, the paper adopts the empirical assumption that mutual information decays with distance \(d\) according to \[ I_{text}(d) \sim C \cdot d^{-\alpha}, \quad \text{with } \alpha \approx 0.5 \] so that \[ \lim_{d \rightarrow \infty} I_{text}(d) = 0. \] A table \(\mathcal{T}\) is modeled as an \(n \times m\) array with headers \(\mathcal{H}=\{H_1,\ldots,H_m\}\) and cells \(T_{i,j}\), and it is linearized row-major as \[ \phi(\mathcal{T}) = [H_1, \ldots, H_m, T_{1,1}, \ldots, T_{1,m}, T_{2,1}, \ldots, T_{2,m}, \ldots, T_{n,m}] \] with total sequence length \[ L = m(n+1). \] For positions in the data region, the column index is \[ \text{col}(t) = ((t - m - 1) \mod m) + 1 \in [m], \] and the average mutual information at lag \(d\) is defined as \[ \bar{I}_{table}(d) = \frac{1}{|\mathcal{S}_d|} \sum_{t \in \mathcal{S}_d} I(W_t; W_{t+d}), \quad \mathcal{S}_d = \{t : m < t \leq L-d\}. \] The analysis assumes **Column Semantic Consistency**, \[ T_{i,j} \sim P_j, \quad \forall i \in [n],\, \forall j \in [m], \] and **Column Distribution Distinctiveness**, \[ D_{KL}(P_j \| P_k) > 0, \quad \forall j \neq k. \] A key quantity is the same-column mutual information \[ I^{same} := I(T_{i,J}; T_{k,J}), \] where \(J\) is a uniformly random column. Under the stated assumptions, the paper proves that \(I^{same} > 0\) and that it is independent of row distance. The main theorem then states \[ \bar{I}_{table}(km) = I^{same} > 0, \quad \forall k \in \{1,2,\ldots,n-1\}. \] This means that when the lag is a multiple of the column count, the average mutual information attains a periodic peak of constant height. The table sequence therefore contains regular, non-decaying long-range dependencies: the same-column cells are repeatedly aligned at distances \(m, 2m, 3m, \dots\) [2603.21719]. The paper draws several asymptotic consequences. For tables with fixed \(m\) and growing \(n\), \[ \liminf_{d \rightarrow \infty} \bar{I}_{table}(d) \geq I^{same} > 0, \] whereas natural text satisfies asymptotic decay. It also defines the effective dependency distance \[ D_{eff}(\tau) = \sup \{d : I(d) \ge \tau\} \] and concludes that \[ D_{eff}^{table}(\tau) = +\infty, \qquad D_{eff}^{text}(\tau) < +\infty \quad \text{for any } \tau \in (0, I^{same}). \] The appendix extends this with a **uniform attention requirement**, according to which all prior same-column tokens remain informative, and a **context length requirement**, according to which the first and last cells of a column can still have mutual information \(I^{same} > 0\) despite being separated by \((n-1)m\) positions. In effect, the theory frames table linearizations as training sequences that repeatedly force retrieval and reasoning over distributed but systematically related evidence. ## 3. Data synthesis pipeline TableLong is implemented as a three-stage pipeline. The first stage initializes environments, the second constructs samples, and the third verifies and filters them. The resulting training instances are SQL-grounded and answer-verifiable [2603.21719]. | Stage | Operation | Output | |---|---|---| | 1 | Environment initialization | SQLite-backed table environments | | 2 | Sample construction | \((\text{Raw Table}, \text{Question}, \text{Answer})\) triplets | | 3 | Verification and filtration | Retained samples with \(0 < P < 1\) | In the first stage, the paper aggregates **over 10,000 real-world tables** from **BIRD**, **CoSQL**, and **Spider**, plus additional scalable table data extracted from information-dense documents using LLMs. After cleaning and sampling, the corpus spans **Finance, Sports, Healthcare, Science, etc.**, includes **English and Chinese**, covers context lengths from hundreds to **32k tokens**, has an average cell density of **5.2 tokens**, and ranges from **single-table** to **30-table** instances. All tables are parsed into a unified **SQLite** environment [2603.21719]. In the second stage, LLMs generate SQL queries, natural-language questions, and answers obtained by executing the SQL. Each sample is a triplet \[ (\text{Raw Table}, \text{Question}, \text{Answer}). \] The SQL is generated along three task dimensions. **Precise Retrieval** restricts SQL to **SELECT** with specific **WHERE** clauses, forcing exact location of atomic information. **Multi-hop Reasoning** introduces aggregation such as `SUM` and `AVG`, forcing reasoning over many dispersed cells. **Grounding** introduces multi-table operations such as `JOIN`, forcing the model to associate the question with the correct tables among many candidates. The appendix gives a prompt that requires the generator to use real schema table names, avoid explicit SQL jargon, produce contextual and natural questions, and output only JSON of the form: ```json {"sql": "...", "natural_language_question": "..."} ``` This stage therefore couples executable structure with natural-language supervision [2603.21719]. In the third stage, execution provides exact answers, and a consistency-based filtration mechanism retains only samples that are neither trivial nor unusable. For each instance \((x, y_{gt})\), the system generates \(N\) candidate responses and computes a pass rate \(P\) against the ground-truth answer. It discards \(P=0\) instances as likely ambiguous, erroneous, or too hard, discards \(P=1\) instances as too easy, and retains only \(0 < P < 1\). The paper emphasizes that this makes the data simultaneously high-quality, diverse, scalable, and verifiable [2603.21719]. ## 4. Reinforcement-learning formulation and training setup The synthesized data is used for reinforcement-learning post-training with **GRPO**. The paper trains four backbones: **Qwen3-32B**, **Qwen2.5-32B-Instruct**, **Deepseek-R1-Distill-Qwen-14B**, and **Deepseek-R1-Distill-Qwen-32B**. Training is conducted with **64 NVIDIA H20 GPUs** in the **Verl** framework [2603.21719]. The appendix gives the GRPO objective as \[ \begin{aligned} \mathcal{J}_{\mathrm{GRPO}(\theta) &= \mathbb{E}_{(q,a)\sim\mathcal{D},\, \{o_i\}_{i=1}^{G}\sim \pi_{\theta_{\mathrm{old}(\cdot \mid q)} \Bigg[ \frac{1}{G}\sum_{i=1}^{G} \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \Big( \min\big( r_{i,t}(\theta)\hat{A}_{i,t}, \operatorname{clip}(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_{i,t} \big) - \beta D_{\mathrm{KL}(\pi_{\theta} \Vert \pi_{\mathrm{ref}) \Big) \Bigg] \end{aligned} \] with token-level importance ratio \[ r_{i,t}(\theta) = \frac{\pi_{\theta}\big(o_{i,t} \mid q, o_{i,0, \] and then to show that a simple, scalable, and verifiable synthesis pipeline built around that structure produces measurable gains in retrieval, grounding, multi-hop reasoning, and broader long-context generalization [2603.21719]. Source: https://www.emergentmind.com/topics/tablelong