ReAcTable: Robust Table QA Framework

• ReAcTable is a framework for Table QA that operationalizes LLMs using iterative intermediate table formation and voting-based answer selection.
• It integrates LLM-generated reasoning with Python and SQL execution for multi-step data transformation and robust error recovery.
• Experimental results on WikiTQ, TabFact, and FeTaQA benchmarks demonstrate improved accuracy and reliability over traditional CoT or ReAct methods.

ReAcTable is a framework designed to enhance Table Question Answering (TQA) by operationalizing LLMs through tightly integrated external tool use and stepwise decomposition. Its development was motivated by the specialized requirements of TQA, including logical inference over schema-rich and noisy tabular data, the need for robust data transformation pipelines, and the generalization limitations of both purely fine-tuned systems and execution-agnostic LLM prompting techniques. ReAcTable builds directly on the ReAct (“Reason + Act”) paradigm, adapting it for tabular data via structured intermediate data representations and voting-based robustness mechanisms (Zhang et al., 2023).

1. Table Question Answering and the Motivation for ReAcTable

Table Question Answering tasks require producing answers to natural language queries over semi-structured tables. The complexity results from challenges such as interpreting table schemas, handling heterogeneous data (e.g., embedded codes, non-canonical field naming), managing analytic transformations (aggregation, joins, string manipulation), and coping with data quality issues. Prior models for TQA—such as Tapas and Tapex—primarily focused on fine-tuning Transformer backbones or LLMs to generate answers directly or via code synthesis. Recent advances incorporated Chain-of-Thought (CoT) prompting and general ReAct reasoning, but lacked mechanisms for iteratively structuring intermediate results and robustly managing LLM errors in executable code.

ReAcTable was designed to bridge the gap by:

• Integrating LLM-driven reasoning with direct execution of Python and SQL code.
• Structuring multi-step problem solving through intermediate table formation after each step.
• Reducing answer uncertainty and non-determinism through explicit voting protocols across reasoning chains.
• Automating error handling, normalization, and re-prompting in code execution phases.

2. System Architecture and Iterative Workflow

ReAcTable implements an iterative agentic process involving the following core components:

• LLM Core: Receives the natural language question and (possibly iteratively generated) tables, and emits reasoning statements or code (Python/SQL).
• External Executors: SQL (via SQLite) and Python (via Pandas) backends to execute LLM-generated code and return result tables.
• Intermediate Data Representation: Each step’s successful code execution produces a new table, which is recursively passed as input to the next LLM step. This transforms the raw tabular data into increasingly “answerable” forms.
• Majority Voting Engine: Repeating the entire reasoning–acting–executing chain multiple times, then aggregating final outputs for reliability.

Full workflow:

1. Inputs: Table $T_0$, question $N$.
2. For each voting run:
   • Initialize tables list: $T = [T_0]$.
   • While loop:
     • Generate prompt from current $T$ and $N$.
     • LLM outputs: code (to be executed, yielding $T'$) or answer (answer is appended).
     • If code, add $T'$ to $T$ and repeat.
     • Terminate upon receiving a direct answer.
   • Repeat for $n$ runs, performing aggregation to resolve the final answer.

Algorithmic Representation (simplified):

for n times: tabs = [T0] while True: prompt = preparePrompt(tabs, N) pred = LLM(prompt, t) if pred is code: T_prime = Execute(pred, tabs) tabs.append(T_prime) else: answers.append(pred) break return getMajority(answers)

Voting options include:

• Simple majority (default and SOTA),
• Tree-exploration (sampling multiple outputs per step and exploring solution paths),
• Execution-based voting (selecting branches by log-probabilities for identical answers).

3. Technical Enhancements over General Reason–Act Paradigms

ReAcTable innovates on generic ReAct and CoT architectures by:

• Embedding both Python and SQL toolchains, exploiting SQL for relational operations and Python for flexible, non-relational data cleaning or string extraction.
• Introducing an execution error recovery mechanism: code errors trigger automatic retries on previous tables, normalization of column names, and auto-import of missing Python modules.
• Explicitly structuring partial results: every code-producing reasoning step outputs an intermediate table, making forthcoming steps over the newly relevant data distribution rather than the original.
• Decoupling reasoning-chain depth from input length limitations, via intermediate table checkpoints.

Comparison overview:

Method	Tool Use	Data Structure Awareness	Voting/Aggregation	Training Required
CoT	No	No	No	Yes/No
ReAct	Yes	Limited	No	No
ReAcTable	Yes	Yes (interm. tables)	Yes	No

4. Addressed TQA Challenges: Empirical Examples

Key challenges handled by ReAcTable include:

• Complex Data Semantics: Multi-step questions requiring string parsing (e.g., extracting country codes from within data fields).
• Noisy and Inconsistent Data Handling: Robust execution paths recovering from failed SQL/Python runs, typographical inconsistencies, or uncleaned data.
• Intricate Transformations: Decomposition of complex aggregation or filtering into simpler slices and mutations over intermediate tables.
• Analytic/Compositional Questions: Chaining of SQL and Python for preparatory computation aligns with analytical problem decomposition common in expert TQA.

Example:

Question: "Which country had the most cyclists finish in the top-10?"

• SQL: Filter rows for top-10 ranks → Cyclist column.
• Python: Regex-extract country codes from Cyclist strings.

  def get_country(s): return re.search("%%%%10%%%%", s).group(1) T1['Country'] = T1.apply(lambda x: get_country(x['Cyclist']), axis=1)

• SQL: Aggregate by country, count, sort, select top-1.

  SELECT Country, COUNT(*) FROM T2 GROUP BY Country ORDER BY COUNT(*) DESC LIMIT 1;

• LLM Output: Return "Italy".

5. Experimental Validation and Performance Analysis

ReAcTable’s experimental results demonstrate state-of-the-art or competitive performance:

• WikiTQ Benchmark:
  • Simple voting: 68.0% accuracy (previous SOTA Dater: 65.9%).
  • No majority voting: 65.8%.
• TabFact:
  • Simple voting: 86.1% (on par with best non-fine-tuned baseline).
• FeTaQA:
  • ROUGE-1: 0.71, ROUGE-2: 0.46, ROUGE-L: 0.61 (best in its set).

Ablation studies quantify the importance of each component:

• Removing intermediate tables drops accuracy to 49.4% (Codex-CoT, no voting).
• Absence of Python executor results in losses: WikiTQ (68.0% → 64.5%), TabFact (86.1% → 76.2%).
• Majority voting increases robustness, with simple majority being most reliable for code-capable models.

Most problem instances are solved within two reasoning–execution iterations; longer chains correlate with elevated question complexity and error frequency.

6. Limitations and Future Research

Identified limitations include:

• The need for manual prompt and few-shot example curation (potential future direction: learning optimal prompt sets).
• Restriction to single-table contexts—multi-table join reasoning is unaddressed.
• Optimal majority voting mechanism selection remains an open question, and computational cost scales with $n$ in voting.
• The system's performance is sensitive to LLM format preferences and cannot fully eliminate failures due to LLM hallucination or ambiguous reasoning chains.

7. Conclusion and Impact

ReAcTable presents a robust, extensible, and minimally engineered pathway to high-performance TQA, relying on LLMs combined with code execution and explicit chaining through intermediate representations (Zhang et al., 2023). Its evidence-based design—particularly the iterative table formation, dual-use code executor architecture, and voting-based answer selection—enables it to supersede general ReAct or CoT approaches, providing a practical bridge between zero/few-shot LLM question answering and the analytic rigor required for competitive tabular data understanding. No model training or backpropagation is required, and the framework generalizes across tasks, toolchains, and LLM variants. This suggests broad applicability across real-world, tool-augmented TQA deployments in research and industry.