LiteCoST: Structure-First Long-Doc QA

• LiteCoST is a structure-first question answering framework that transforms long, noisy documents into compact, query-specific representations such as tables, graphs, or lists.
• The framework leverages a two-pillar architecture, combining schema-aware Chain-of-Structured-Thought (CoST) with small language model fine-tuning and Group Relative Policy Optimization (GRPO) to ensure high-quality structured outputs.
• Empirical results show LiteCoST achieves LLM-comparable accuracy with 2–4x lower latency, demonstrating its efficiency in long-document QA across multiple domains.

LiteCoST is a two-pillar framework for long-document question answering that treats the task as QA-by-structuring rather than direct answer generation from raw context. It combines Chain-of-Structured-Thought (CoST), a schema-aware prompting method for a strong teacher LLM, with small LLM (SLM) fine-tuning based on teacher-generated CoST supervision and Group Relative Policy Optimization (GRPO). The framework targets queries whose answers can be grounded in a compact, query-specific structured representation such as a table, graph, or serialized chunk set, and reports LLM-comparable quality on multi-domain long-document QA using 3B/7B SLMs, while delivering 2–4x lower latency than GPT-4o and DeepSeek-R1 (671B) (Liang et al., 31 Mar 2026).

1. Problem formulation and structure-first premise

LiteCoST studies a specific regime of long-document QA in which a natural-language question $Q$ and a collection of long, noisy documents $D$ are transformed into a serialized structured output (SSO) $S$, from which the final answer $A$ is derived. The intended pipeline is to induce a minimal query-specific schema, populate it with evidence, normalize entities, units, and time expressions, align records across dispersed evidence, serialize the result, and compute the answer from the resulting structure. This differs from conventional long-context QA, where a model is prompted directly with $(Q,D)$ and asked to reason end-to-end over raw text (Liang et al., 31 Mar 2026).

The motivation is that direct long-context reasoning is brittle when evidence is dispersed, heterogeneous units must be reconciled, and output consistency has to be maintained over long sequences. LiteCoST therefore adopts a structure-first view: structured intermediates reduce noise, expose evidence explicitly, and make extraction and reasoning more auditable. The framework is not presented as a universal solution for all QA, but for tasks where a minimal structured representation can be built and the answer is derivable from that representation.

The paper states the central desideratum as

$$\text{LLM}(Q, S_{\text{SLM}}) \;\approx\; \text{LLM}(Q, S_{\text{LLM}}) \quad \text{and} \quad \text{Latency}(S_{\text{SLM}}) \;\ll\; \text{Latency}(S_{\text{LLM}}).$$

The intended meaning is that the SLM-generated structure should be almost as useful for downstream answering as the teacher LLM’s structure, but much faster to produce.

2. Two-pillar architecture

LiteCoST is organized around two components. Pillar 1 is CoST, a schema-aware instruction template that guides a strong LLM to produce both a step-wise reasoning trace and a final structured output. Pillar 2 is SLM fine-tuning, in which compact models are trained on teacher-generated CoST data in two stages: supervised fine-tuning for structural alignment, followed by GRPO with rewards for answer quality, format quality, and process consistency (Liang et al., 31 Mar 2026).

The CoST template is described as schema-aware and structure-first. The output is divided into two tagged regions: <reasoning>...</reasoning> for the trace and <answer>...</answer> for the serialized structure. The resulting training example has the form

$z = (i, d, c^*, y^*),$

where $i$ is the question, $d$ is the document input, $c^*$ is the CoST trace, and $D$0 is the structured output.

This design uses a strong LLM offline to generate auditable supervision, then transfers that behavior into smaller models for deployment. A plausible implication is that LiteCoST treats large-model reasoning primarily as a data-generation and supervision resource, not as the runtime execution engine.

3. Chain-of-Structured-Thought

CoST differs from ordinary chain-of-thought by making the reasoning process explicitly target a structured artifact rather than a free-form textual rationale. The paper decomposes CoST into four stages. (A1) Structure Analysis selects the output type—such as table or graph—and induces a query-specific schema. (A2) CoST Trace Generation performs extraction, alignment, normalization, and deterministic serialization. (A3) Quality Verification uses LLM-as-a-Judge, specifically GPT-4o, to retain only outputs that exactly match the reference answer. (A4) Iterative Refinement applies an Iterative Structuralizer to regenerate or supplement low-quality structures (Liang et al., 31 Mar 2026).

The iterative update is written as

$D$1

with the intended meaning that the current structure is preserved if it is sufficient for the question, and otherwise refined recursively.

The framework explicitly allows tables, graphs, and chunks / lists / record sets as structured outputs. On Loong, after selection, the reported structure distribution was 823 tables, 400 graphs, and 377 text chunks. These outputs are serialized into a machine-checkable representation and paired with the CoST trace, which records structure selection, schema construction, normalization, alignment, and verification decisions. This makes the supervision auditable in a way ordinary final-answer supervision is not.

4. SLM fine-tuning and optimization

The SLM training pipeline has two stages. In Supervised Fine-Tuning (SFT), the model learns the mapping $D$2, thereby acquiring structure handling, format discipline, and step-wise extraction behavior. In GRPO, the model samples groups of candidate outputs and receives normalized group-relative advantages

$D$3

The manuscript prints a corrupted version of the GRPO objective, but the intended formulation is a PPO-style clipped policy objective with KL regularization (Liang et al., 31 Mar 2026).

The reward design has three components. Format compliance reward gives 0.5 if the output contains a single reasoning sequence inside <reasoning>...</reasoning> and a final answer inside <answer>...</answer> with no extraneous content, 1.0 if the reasoning is also explicitly step-labeled, and 0 otherwise. Answer correctness reward combines rule-based structural checks and semantic similarity:

$D$4

with $D$5, so the answer reward is weighted 30% structural and 70% semantic. Process reward evaluates step-wise consistency against teacher steps:

$D$6

The paper further defines a scaled process reward

$D$7

where $D$8 modulates the trajectory-level effect of the process reward.

The main backbones are LLaMA-3.2-3B-Instruct and Qwen2-7B-Instruct. Reported SFT settings are 3 epochs, learning rate $D$9, batch size 16, and LoRA with rank 16 and alpha 32. GRPO uses learning rate $S$0, batch size 16, and 5 sampled generations per query. The frameworks are trl and verl, and the maximum generation length is 2048 tokens.

5. Empirical results

LiteCoST is evaluated primarily on Loong, a benchmark with 1,600 test samples, 3 domains (Finance, Legal, Paper), 4 task categories (Spotlight Locating, Comparison, Clustering, Chain of Reasoning), and 4 document-length settings. It is also evaluated on LongBench using NarrativeQA, Qasper, HotpotQA, and 2WikiMultihopQA. On Loong, the main reported metrics are AS (Average Score) and PR (Perfect Rate), with GPT-4o used as judge over Accuracy, Hallucination, and Completeness. Latency is reported as average time per sample (Liang et al., 31 Mar 2026).

Model	Loong Finance (AS / PR)	Avg. time per sample
LLaMA-3.2-3B-Instruct (Base)	49.37 / 0.11	—
LLaMA-3.2-3B-Instruct (Ours)	76.95 / 0.40	8.04s
Qwen2-7B-Instruct (Base)	62.10 / 0.26	11.89s
Qwen-LiteCoST 7B	79.93 / 0.48	12.09s
GPT-4o	79.32 / 0.54	21.15s
DeepSeek-R1	78.18 / 0.46	44.44s

On Finance, LiteCoST yields large backbone-specific gains: LLaMA-3.2-3B-Instruct (Ours) improves by +27.58 AS and +0.29 PR over its base model, while Qwen2-7B-Instruct (Ours) improves by +17.83 AS and +0.22 PR. The Qwen-LiteCoST 7B result of 79.93 AS / 0.48 PR slightly exceeds GPT-4o in AS on this benchmark, while GPT-4o remains higher in PR. The latency results show that Qwen-LiteCoST is nearly 2× faster than GPT-4o and nearly 4× faster than DeepSeek-R1.

The paper also reports that CoST-generated structured data improves downstream reasoning by strong LLMs on Loong. Replacing raw documents with structured data produced overall score gains of +12.41 for Qwen2-72B-Instruct, +8.77 for GPT-4o-mini, +9.04 for GPT-4o, and +8.47 for Claude-3.5-Sonnet, with corresponding PR gains of +0.10, +0.05, +0.04, and +0.08. On prompting alone, CoST also outperforms zero-shot, CoT, and DSPy/MiPRO-style prompt optimization; for example, on Finance, GPT-4o improves from 79.32 with zero-shot and 80.08 with CoT to 82.39 with CoST.

Generalization beyond Finance is also reported. On Loong Legal, LLaMA-LiteCoST improves from 30.67 to 45.45, and Qwen-LiteCoST improves from 38.05 to 44.94. On LongBench F1, Qwen-LiteCoST reports 30.40 / 44.64 / 68.39 / 65.73, compared with GPT-4o at 28.68 / 43.39 / 67.68 / 68.29, outperforming GPT-4o on NarrativeQA, Qasper, and HotpotQA.

6. Ablations, failure modes, and scope

The ablation results isolate both the structural prompting contribution and the training contribution. On Finance, the ordering is consistently zero-shot < CoT < CoST; for Qwen2.5-14B-Instruct, AS rises from 75.60 to 77.51 to 79.01, and for GPT-4o, from 79.32 to 80.08 to 82.39. The appendix also shows that GRPO improves over SFT alone: for LLaMA-3B, SFT = 72.27 and SFT+GRPO = 76.95; for Qwen-7B, SFT = 76.83 and SFT+GRPO = 79.93 (Liang et al., 31 Mar 2026).

Reward ablations indicate that both outcome and process rewards matter. On Finance, for the LLaMA backbone, the full model scores 76.95, compared with 75.52 without process reward and 72.55 without outcome reward. For Qwen, the full model scores 79.93, compared with 77.39 without process reward and 75.43 without outcome reward. The paper interprets outcome reward as the dominant driver of final accuracy, with process reward still providing a meaningful gain.

The paper also identifies a concrete failure mode: if the structured representation omits or corrupts a critical value, the structure itself becomes the bottleneck. In the reported Spotlight Locating example, the question asks for the company with $S$1 in Accounts Receivable; the ground truth is CIRTRAN CORP, but the extracted table assigns None to that field, leading to the answer “Not Provided.” This supports the paper’s more nuanced claim that structured representations can slightly hurt tasks where compression loses subtle contextual clues, especially Spotlight Locating and Chain-of-Reasoning.

The stated limitations are that broader generalization to more distinct domains remains to be fully verified, domain-specific document QA datasets are scarce, and the framework depends heavily on a strong teacher LLM—especially GPT-4o and GPT-4o-mini—for CoST generation, verification, iterative refinement, and some reward judging. LiteCoST is therefore best understood as a teacher-distilled, structure-first framework whose strongest operating regime is long-document QA where the answer is naturally grounded in a compact structured representation.