Contexts are Never Long Enough: Structured Reasoning for Scalable Question Answering over Long Document Sets

Abstract: Real-world document question answering is challenging. Analysts must synthesize evidence across multiple documents and different parts of each document. However, any fixed LLM context window can be exceeded as document collections grow. A common workaround is to decompose documents into chunks and assemble answers from chunk-level outputs, but this introduces an aggregation bottleneck: as the number of chunks grows, systems must still combine and reason over an increasingly large body of extracted evidence. We present SLIDERS, a framework for question answering over long document collections through structured reasoning. SLIDERS extracts salient information into a relational database, enabling scalable reasoning over persistent structured state via SQL rather than concatenated text. To make this locally extracted representation globally coherent, SLIDERS introduces a data reconciliation stage that leverages provenance, extraction rationales, and metadata to detect and repair duplicated, inconsistent, and incomplete records. SLIDERS outperforms all baselines on three existing long-context benchmarks, despite all of them fitting within the context window of strong base LLMs, exceeding GPT-4.1 by 6.6 points on average. It also improves over the next best baseline by ~19 and ~32 points on two new benchmarks at 3.9M and 36M tokens, respectively.

• The paper presents SLIDERS, which decouples extraction from reasoning to overcome the evidence aggregation bottleneck in long-context question answering.
• It introduces a five-stage pipeline—including contextualized chunking, schema induction, and SQL-based query synthesis—that delivers up to 14% accuracy gains on global evidence synthesis tasks.
• SLIDERS offers enhanced scalability, auditability, and cost efficiency by constructing a deduplicated, relational database for robust performance on ultra-long document benchmarks.

Structured Reasoning for Long-Context Question Answering: An Essay on SLIDERS

Motivation and Problem Setting

Long-context question answering (QA) over large document sets remains a central challenge in natural language processing, particularly in knowledge-intensive domains such as finance, healthcare, and legal analysis. Modern LLMs, even those supporting million-token context windows, are unable to reliably process and synthesize evidence scattered across extended multi-document corpora. Standard chunking-based solutions address local context overflow but suffer from a fundamental aggregation bottleneck: evidence extracted from numerous chunks ultimately requires global reasoning, which induces a secondary long-context problem (Figure 1).

Figure 1: Chunking based methods regenerate the long-context problem. SLIDERS mitigates it by using structured reasoning.

The SLIDERS Framework

SLIDERS (Scalable Long-document Integration through Decomposed Extraction and Reconciliation System) is a framework for QA over long document sets that eschews purely unstructured, concatenation-based prompting in favor of structured reasoning. Instead of aggregating free-text outputs from multiple chunks, SLIDERS first extracts salient information from each chunk into a persistent relational database, enabling downstream reasoning via SQL queries rather than natural language synthesis.

The pipeline encompasses five sequential operations: contextualized chunking, schema induction, structured extraction, data reconciliation, and question answering (Figure 2).

Figure 2: SLIDERS overview showing the process of contextualization of chunks, schema induction, structured extraction, data reconciliation, and SQL-based question answering over the reconciled database.

Chunking and Contextualization

SLIDERS partitions each document into context-aware, structurally coherent chunks using both global (title, description) and local (section headers, tables) metadata. This ensures that extracted information is locally interpretable and retains sufficient context for accurate entity and relation extraction.

Schema Induction and Structured Extraction

Using both the input question and document metadata, an LLM induces a task-specific extraction schema comprising tables, fields, types, and normalization rules. Extraction is then cast as a structured IR task: for each chunk and schema, the system generates normalized records including provenance and extraction rationale. A relevance-gating stage precedes extraction to minimize hallucinated entries when information is absent.

Reconciliation and Global Reasoning

After extraction, SLIDERS reconciles partial, redundant, and possibly conflicting records across all chunks to construct a globally coherent, deduplicated database. The data reconciliation agent leverages provenance, rationale, and metadata to perform deduplication, conflict resolution, and consolidation within entity-keyed partitions. This approach sharply reduces the quadratic complexity of naive pairwise record comparison and is critical to producing reliable downstream answers as document sets grow (see Reconciliation Analysis section).

SQL-Based Question Answering

With a reconciled, auditable database, the system answers questions by generating and executing SQL queries. The QA agent iteratively composes and refines SQL over the structured representation, bridging LLM-driven semantic mapping and deterministic symbolic computation.

Empirical Results

SLIDERS was evaluated on both existing long-context QA benchmarks (which fit within strong LLM context windows) and newly constructed ultra-long-document benchmarks extending to 3.9M and 36M tokens. The empirical results are summarized as follows:

• On long-context benchmarks (≤360k tokens): SLIDERS outperforms the strongest base LLMs (incl. GPT-4.1), RAG, and agent-based baselines by an average of 6.6 points, with statistically significant improvements across the board. Notably, in aggregation-heavy tasks requiring global evidence synthesis, such as Oolong, the gains are as large as 14% absolute accuracy.
• On ultra-long-document benchmarks (>3.9M tokens): SLIDERS is the only evaluated system to scale beyond prevailing LLM context limits and achieves 78.9% accuracy on Wikipedia (WikiCeleb100, 3.9M tokens) and 55.2% on large-scale financial filings (FinQ100, 36M tokens), outperforming agentic baselines by margins up to 32 points and executing at 13× lower cost.
• Model-agnostic improvements: When paired with open-source models (Qwen3.5 122B-A10B), SLIDERS continues to confer substantial accuracy gains, confirming that the structured-reasoning pipeline, rather than proprietary LLM capability, is the principal source of improvement.

These results are visually summarized below:

(Figure 3)

Figure 3: Accuracy across long-context and ultra-long benchmarks. SLIDERS consistently outperforms all baselines and is the only system to scale beyond 3.9M tokens.

Ablations and Analysis

Ablation studies demonstrate the necessity of each SLIDERS component: when chunking, data reconciliation, or SQL-based QA are removed, accuracy diminishes significantly, particularly for tasks requiring global aggregation. Reconciliation is essential for large, noisy financial document sets, yielding up to 70–90% reduction in redundant entries per resolved key (see Reconciliation Analysis).

The data reconciliation stage is efficient and adaptive, resolving >90% of primary-key groups in a single iteration. Its operation breakdown reveals that conflict resolution dominates in financial corpora, whereas deduplication and consolidation are more important in multi-document biographical settings.

(Figure 4)

Figure 4: Analysis of the reconciliation agent: (a) density of row reduction per primary key group, (b) fraction of groups resolved within k iterations, (c) operation-type breakdown, (d) average rows per key before and after reconciliation.

Cost and latency analyses indicate that SLIDERS remains cost-competitive with—often cheaper than—agentic baselines, and its two-phase pipeline enables efficient amortization for multi-query analysis over a fixed document collection.

Auditability, Interpretability, and Error Analysis

A key ancillary benefit of SLIDERS' structured approach is enhanced auditability. All extracted and reconciled facts are annotated with provenance, facilitating targeted human verification and error analysis. Human-in-the-loop studies corroborate evaluation reliability, with Cohen's $\kappa = 0.758$ vs. LLM-as-a-judge supervision, and error tracing is greatly expedited via persistent record-level evidence.

Implications, Limitations, and Future Directions

The implications of SLIDERS are multifold:

• Scalability and generalization: SLIDERS demonstrates that database-centric intermediate representations enable long-context QA to scale beyond current LLM context windows, both in input size and task complexity.
• Decoupling of extraction and reasoning: By separating conversion (text $\rightarrow$ schema/records) from symbolic reasoning (SQL), the system can exploit LLM strengths (semantic mapping, schema induction) while offloading hard, verifiable computation (aggregation, filtering, evidence synthesis) to robust relational infrastructure.
• Auditability and compliance: Persistent provenance and structured rationales support fine-grained audit and facilitate compliance in high-stakes settings (e.g., financial, legal, healthcare).
• Framework extensibility: While this work targets QA, SLIDERS' persistent database representations and modular pipeline also generalize to open-ended tasks such as multi-document summarization and synthesis.

The primary limitation is that SLIDERS' schema-first approach is not suited to tasks that require fundamentally unstructured or highly subjective reasoning not reducible to relational modeling. Latency overhead (2–3 min) precludes some real-time applications, but is acceptable for analyst-grade settings. Additionally, ultimate accuracy on the most challenging benchmarks (55% on FinQ100) remains insufficient for end-to-end automation in safety-critical workflows, recommending continued human review.

Future research is anticipated in several directions:

• Improved schema induction and adaptation to non-relational tasks.
• Integration of advanced entity resolution, e.g., joint resolution across semi-structured free-text.
• Optimization of multi-modal document processing pipelines.
• Interactive and conversational interfaces for explainable, mixed-initiative workflows bridging SLIDERS with domain analysts.

Conclusion

SLIDERS introduces a paradigm shift for scalable, auditable QA over long document collections by decoupling information extraction and symbolic reasoning via structured IR, data reconciliation, and SQL-driven answer synthesis. It achieves state-of-the-art accuracy across both long-context and ultra-long-context benchmarks, with strong gains orthogonal to base LLM quality, and advances the reliability and auditability of LLM-based systems for knowledge-intensive, large-scale document analysis.