LIT-RAGBench: RAG Generator Benchmark

• LIT-RAGBench is a benchmark that evaluates Retrieval-Augmented Generation (RAG) generators using controlled multi-document contexts and five key evaluation categories.
• It assesses integration, reasoning, logic, table understanding, and abstention capabilities through fictional scenarios and bilingual datasets to simulate realistic retrieval noise.
• Empirical results reveal that even state-of-the-art models like GPT-5 face challenges in accuracy, highlighting the need for improved multi-document evidence handling and safe abstention.

Searching arXiv for LIT-RAGBench and closely related RAG benchmarking work. arXiv search query: "LIT-RAGBench RAG benchmark generator capabilities" LIT-RAGBench is a benchmark for evaluating the Generator in Retrieval-Augmented Generation (RAG) systems under controlled retrieval conditions. Its name abbreviates Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark, and its design targets capabilities that arise when a generator must operate over long, noisy, multi-document contexts rather than answer from parametric memory alone. The benchmark defines five evaluation categories—Integration, Reasoning, Logic, Table, and Abstention—uses fictional entities and scenarios so that answers must be grounded in the provided documents, and provides a bilingual Japanese/English dataset with category-wise and overall scoring via LLM-as-a-Judge (Itai et al., 6 Mar 2026).

1. Scope and formal problem definition

LIT-RAGBench isolates the generator side of a RAG pipeline. In the formulation used by the benchmark, a retriever $\mathcal{R}$ receives a retrieval query $x_r = f(q)$ derived from a user question $q$, and returns a set of chunks

$C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$

from an external corpus $\mathcal{E}$. A generator $\mathcal{G}$ then produces an answer

$y = \mathcal{G}(x_g), \quad x_g = (\tau, q, C),$

where $\tau$ is the task instruction and $C$ is the concatenated retrieved context (Itai et al., 6 Mar 2026).

The benchmark makes explicit a distinction between $C^{+}$, the relevant chunks needed to answer $x_r = f(q)$0, and $x_r = f(q)$1, irrelevant or distracting chunks. This distinction matters because the retriever does not guarantee that all relevant chunks are present, nor that ranking is ideal. The generator must therefore identify and use $x_r = f(q)$2, ignore $x_r = f(q)$3, and in some cases abstain when the evidence is missing, contradictory, or incomplete (Itai et al., 6 Mar 2026).

The motivating claim is that practical generators must handle long, noisy context; integrate evidence from $x_r = f(q)$4–$x_r = f(q)$5 sources; perform multi-step reasoning and numerical calculation; interpret tables in HTML, Markdown, and CSV; resolve logical or semantic mismatches; and explicitly refuse when the provided evidence is inadequate. Existing benchmarks are described as offering limited simultaneous coverage of these capabilities under unified conditions, often entangling retriever and generator performance or under-representing table parsing, fine-grained logical inclusion, and realistic abstention failure modes (Itai et al., 6 Mar 2026). In contrast, RAGBench formalizes a broader explainable evaluation framework for full RAG pipelines through TRACe—uTilization, Relevance, Adherence, and Completeness—over a large multi-domain benchmark (Friel et al., 2024). LIT-RAGBench narrows the focus to the generator and fixes retrieval so that generator behavior can be examined independently (Itai et al., 6 Mar 2026).

2. Capability taxonomy and compositional design

LIT-RAGBench defines the category set

$x_r = f(q)$6

for Integration, Reasoning, Logic, Table, and Abstention. The “Main” categories are

$x_r = f(q)$7

and Abstention is treated separately; it does not co-occur with the Main categories (Itai et al., 6 Mar 2026).

Each category is subdivided into evaluation aspects. For a question $x_r = f(q)$8, the involved aspect set $x_r = f(q)$9 satisfies $q$0, and if two aspects co-occur, they must come from different categories. This gives a controlled compositional regime in which a question may test, for example, Reasoning × Table or Logic × Integration, without collapsing all capabilities into a single undifferentiated score (Itai et al., 6 Mar 2026).

Category	Core focus	Evaluation aspects
Integration	Multi-document evidence integration	Multi-source integration with $q$1
Reasoning	Inference and calculation over retrieved evidence	Multi-hop reasoning; numerical calculation
Logic	Logical and semantic alignment under expression mismatch	Synonym interpretation; numerical inclusion; conceptual inclusion
Table	Tabular understanding in retrieved chunks	HTML; HTML with merged cells; Markdown; CSV
Abstention	Refusal under unavailable or incoherent evidence	Insufficient evidence; contradictory evidence; incomplete chunk

Integration targets cases where multiple relevant chunks contain complementary or differently formatted evidence. Reasoning covers both multi-hop inference across documents and numerical calculation such as totals, averages, profit margins, or growth rates. Logic targets semantic equivalence and inclusion, including unit conversion, numeric range interpretation, and hierarchical concept relations. Table evaluates whether the model can correctly parse tables under structural variation and chunk disruption. Abstention evaluates whether the model can refuse when $q$2, when retrieved evidence is contradictory, or when chunk boundaries make the question unanswerable (Itai et al., 6 Mar 2026).

A central design feature is category co-occurrence. The benchmark explicitly permits compound tasks such as multi-hop reasoning over an HTML table with merged cells. In the paper’s example, identifying a research theme adopted by a fictional organization requires linking a partnership statement to a grant table and correctly interpreting a rowspanned or colspan-structured table entry (Itai et al., 6 Mar 2026).

3. Dataset construction, fictionalization, and bilingual release

The dataset is defined as

$q$3

where each instance contains a question $q$4, reference answer $q$5, relevant chunks $q$6, distractor chunks $q$7, and aspect labels $q$8. A key constraint is

$q$9

which is intended to simulate realistic long-context RAG conditions (Itai et al., 6 Mar 2026).

Construction was human-centered. Three native Japanese speakers designed scenarios and questions, using GPT-5 only as an auxiliary drafting tool for scenario text and candidate documents. All items were manually reviewed, filtered by two independent annotators, and retained only when both approved. Quality checks required that the question match the intended aspect pattern, that $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$0 be derivable from $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$1, that the item not be answerable from pre-trained knowledge alone, and that fictional information not contradict real-world facts (Itai et al., 6 Mar 2026).

The benchmark follows the methodology of Kirchenbauer et al. in using fictional companies, products, people, and institutions. This design suppresses leakage from the model’s internal knowledge and reduces spurious success due to memorized facts. Correct answering is therefore intended to require grounding in the provided documents rather than recall from pretraining (Itai et al., 6 Mar 2026).

Chunks are normalized to approximately 512 tokens using tiktoken, and distractor chunks are topically related but do not contain the answer. For tables, large tables may be chunked, with header information repeated in each chunk, and chunk order is shuffled to simulate retriever output order and reduce position bias (Itai et al., 6 Mar 2026).

The Japanese dataset contains 114 questions. The Main categories contribute 54 questions, of which 12 have a single aspect and 42 have two aspects. Abstention contributes 54 Insufficient Evidence questions by dropping $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$2, plus 3 Contradictory Evidence questions and 3 Incomplete Chunk questions, yielding the total of 114. An English dataset of equal size was created by GPT-5 translation of every Japanese instance with human curation (Itai et al., 6 Mar 2026).

4. Evaluation protocol, scoring, and model suite

At evaluation time, the generator receives a task instruction $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$3, a question $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$4, and a concatenated context $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$5, with chunk order randomized per run. For Main categories, the generator input is

$C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$6

For Insufficient Evidence abstention cases, the input is

$C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$7

with $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$8 (Itai et al., 6 Mar 2026).

The output is a single natural-language answer $C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)$9. For abstention items, correctness requires an explicit statement that the question cannot be answered from the given context, or, in contradictory-evidence cases, an explicit acknowledgement of contradiction as encoded in the reference answer (Itai et al., 6 Mar 2026).

Automatic scoring is performed by an evaluator LLM $\mathcal{E}$0, instantiated as GPT-4.1 (2025-04-14). The judge computes a binary decision

$\mathcal{E}$1

No partial credit is assigned (Itai et al., 6 Mar 2026).

Category-wise accuracy is defined over the subset $\mathcal{E}$2 of questions labeled with category $\mathcal{E}$3: $\mathcal{E}$4 Overall accuracy is the average of category-wise accuracies across all five categories: $\mathcal{E}$5 If a question involves multiple aspects from different Main categories and is answered correctly, it is counted as correct for all relevant categories (Itai et al., 6 Mar 2026).

The benchmark evaluates two model groups. The API-based group includes GPT-5, GPT-5-mini, GPT-5-nano, o3, o4-mini, GPT-4.1, GPT-4.1-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, and Claude-Sonnet-4. The open-weight group includes Gemma-3-27B-Instruct, Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Qwen3-235B-A22B-Instruct, and Qwen3-235B-A22B-Thinking. Reasoning models are evaluated with their maximum reasoning token length, and the reported decoding settings are temperature $\mathcal{E}$6 and top_p $\mathcal{E}$7 (Itai et al., 6 Mar 2026).

5. Empirical results and observed failure modes

The headline empirical result is that no model achieves $\mathcal{E}$8 in either Japanese or English (Itai et al., 6 Mar 2026). The strongest overall model is GPT-5, with overall accuracy reported as approximately 0.872 in both languages. Among open-weight models, Qwen3-235B-A22B-Instruct reaches 0.859, while Qwen3-235B-A22B-Thinking reaches 0.821 (Itai et al., 6 Mar 2026).

Category-wise results are heterogeneous. For Integration, top models reach roughly 0.83–0.92 in each language. For Reasoning, o3 achieves 0.957 in Japanese and 0.870 in English, and is described as best on numerical reasoning. For Logic, several models cluster around 0.83–0.90. For Table, Gemini-2.5-Flash achieves 0.871 in Japanese and 0.903 in English, the strongest table performance reported. For Abstention, Claude-Sonnet-4 achieves 0.950 in Japanese and 0.967 in English, the best abstention score (Itai et al., 6 Mar 2026).

The benchmark’s analysis emphasizes that strong abstention can coincide with over-abstention on answerable cases. The reported average over-abstention rates on Main questions are 0.259 for Claude-Sonnet-4, 0.213 for Llama-3.1-8B-Instruct, 0.157 for Llama-3.3-70B-Instruct, 0.120 for Qwen3-235B-A22B-Thinking, 0.065 for GPT-5, and 0.028 for GPT-4.1-mini. This is presented as a safety–usefulness trade-off: strong abstention behavior can reduce hallucination but also reduce utility (Itai et al., 6 Mar 2026).

Qualitative analysis identifies distinct failure patterns by category. In Integration, models often fail when relevant chunks have no direct lexical match with the question or when only one document contains a special note that changes the final answer. In Reasoning, even strong models can reason qualitatively correctly yet make small arithmetic errors, while smaller models may fail to identify intermediate facts or refuse to answer when cues are implicit. In Logic, typical mistakes include unit mismatches such as returning “500 MB” when the question requires GB, and incorrect rendering of large Japanese number units such as “760 million yen” instead of “7.6 billion yen.” In Table, most models parse simple tables reasonably well but struggle with HTML tables containing merged cells, and with tables split across shuffled chunks, even when headers are duplicated (Itai et al., 6 Mar 2026).

Abstention errors are especially revealing. On Insufficient Evidence items, many models hallucinate plausible but unsupported answers. On Contradictory Evidence items, Claude-Sonnet-4 is described as consistently identifying inconsistencies and abstaining, whereas other models often rely on a single chunk and ignore contradiction. On Incomplete Chunk items, most models abstain, but some hallucinate by relying on general knowledge despite ambiguity in the supplied context (Itai et al., 6 Mar 2026).

6. Position in the benchmark landscape, limitations, and future directions

LIT-RAGBench is presented as a unified generator benchmark that complements rather than replaces other RAG evaluations. Relative to prior benchmarks such as RGB, FRAMES, RAGBench, RAGTruth, CRAG, and MMQA, it is characterized as a single framework covering multi-document integration, multi-hop and numerical reasoning, logical or semantic inclusion, structurally diverse tables, and abstention under realistic failure modes, while holding retrieval fixed so that generator behavior can be isolated (Itai et al., 6 Mar 2026). RAGBench, by contrast, emphasizes explainable span-level evaluation for full RAG systems through TRACe across a large multi-domain benchmark (Friel et al., 2024).

A plausible implication is that LIT-RAGBench and RAGCap-Bench occupy complementary layers of the evaluation stack: LIT-RAGBench isolates generator behavior under fixed retrieved context, whereas RAGCap-Bench evaluates process-level capabilities such as planning, evidence extraction, grounded reasoning, and noise robustness in agentic RAG workflows (Lin et al., 15 Oct 2025). A broader benchmark ecology is also visible. SMARTFinRAG studies document-centric financial RAG with joint retrieval and generation evaluation (Zha, 25 Apr 2025). Legal RAG Bench evaluates end-to-end legal RAG and decomposes failures into retrieval errors, reasoning errors, and hallucinations (Butler et al., 2 Mar 2026). RAGRouter-Bench addresses adaptive paradigm selection across 7,727 queries and 21,460 documents and argues that no single RAG paradigm is universally optimal (Wang et al., 30 Jan 2026). This suggests that LIT-RAGBench is most naturally interpreted as a generator-specialized component within a larger modular evaluation program for RAG systems.

The benchmark’s stated limitations are substantial. The dataset size—114 questions per language—is intentionally compact and high quality but small compared with large-scale benchmarks. Aspect imbalance is explicit: within Abstention, Insufficient Evidence dominates, while Contradictory Evidence and Incomplete Chunk are scarce. The fictional-world design is useful for avoiding contamination but may not capture all properties of real web or enterprise data, such as OCR noise or messy formatting. Evaluation depends on GPT-4.1 as judge and therefore inherits LLM-as-a-Judge bias. The benchmark also aims to capture “minimum occurrence patterns” for each aspect rather than exhaustive coverage (Itai et al., 6 Mar 2026).

Future work is directed toward scaling and diversifying the dataset while retaining human-verifiable quality, extending toward Agentic RAG, expanding to additional languages, and studying more nuanced logical and abstention scenarios. The benchmark is released as open source, with dataset, prompt templates, evaluation code, category-wise and overall scoring scripts, and over-abstention analysis available in the associated repository (Itai et al., 6 Mar 2026).