---
title: 'LIT-RAGBench: RAG Generator Benchmark'
url: https://www.emergentmind.com/topics/lit-ragbench
type: topic
---

# LIT-RAGBench: RAG Generator Benchmark

Searching arXiv for LIT-RAGBench and closely related RAG benchmarking work.
arXiv search query: "LIT-RAGBench RAG benchmark generator capabilities"
LIT-RAGBench is a benchmark for evaluating the **Generator** in Retrieval-Augmented Generation (RAG) systems under controlled retrieval conditions. Its name abbreviates **Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark**, and its design targets capabilities that arise when a generator must operate over long, noisy, multi-document contexts rather than answer from parametric memory alone. The benchmark defines five evaluation categories—Integration, Reasoning, Logic, Table, and Abstention—uses fictional entities and scenarios so that answers must be grounded in the provided documents, and provides a bilingual Japanese/English dataset with category-wise and overall scoring via LLM-as-a-Judge [2603.06198].

## 1. Scope and formal problem definition

LIT-RAGBench isolates the generator side of a RAG pipeline. In the formulation used by the benchmark, a retriever \(\mathcal{R}\) receives a retrieval query \(x_r = f(q)\) derived from a user question \(q\), and returns a set of chunks
\[
C = \{c_1, \dots, c_n\} = \mathcal{R}(x_r)
\]
from an external corpus \(\mathcal{E}\). A generator \(\mathcal{G}\) then produces an answer
\[
y = \mathcal{G}(x_g), \quad x_g = (\tau, q, C),
\]
where \(\tau\) is the task instruction and \(C\) is the concatenated retrieved context [2603.06198].

The benchmark makes explicit a distinction between \(C^{+}\), the relevant chunks needed to answer \(q\), and \(C^{-}\), irrelevant or distracting chunks. This distinction matters because the retriever does not guarantee that all relevant chunks are present, nor that ranking is ideal. The generator must therefore identify and use \(C^{+}\), ignore \(C^{-}\), and in some cases abstain when the evidence is missing, contradictory, or incomplete [2603.06198].

The motivating claim is that practical generators must handle long, noisy context; integrate evidence from \(2\)–\(3\) sources; perform multi-step reasoning and numerical calculation; interpret tables in HTML, Markdown, and CSV; resolve logical or semantic mismatches; and explicitly refuse when the provided evidence is inadequate. Existing benchmarks are described as offering limited simultaneous coverage of these capabilities under unified conditions, often entangling retriever and generator performance or under-representing table parsing, fine-grained logical inclusion, and realistic abstention failure modes [2603.06198]. In contrast, RAGBench formalizes a broader explainable evaluation framework for full RAG pipelines through TRACe—uTilization, Relevance, Adherence, and Completeness—over a large multi-domain benchmark [2407.11005]. LIT-RAGBench narrows the focus to the generator and fixes retrieval so that generator behavior can be examined independently [2603.06198].

## 2. Capability taxonomy and compositional design

LIT-RAGBench defines the category set
\[
\Theta = \{\theta_I, \theta_R, \theta_L, \theta_T, \theta_A\},
\]
for Integration, Reasoning, Logic, Table, and Abstention. The “Main” categories are
\[
\Theta_{\text{Main}} = \Theta \setminus \{\theta_A\},
\]
and Abstention is treated separately; it does not co-occur with the Main categories [2603.06198].

Each category is subdivided into evaluation aspects. For a question \(q\), the involved aspect set \(\Psi(q)\) satisfies \(1 \le |\Psi(q)| \le 2\), and if two aspects co-occur, they must come from different categories. This gives a controlled compositional regime in which a question may test, for example, Reasoning × Table or Logic × Integration, without collapsing all capabilities into a single undifferentiated score [2603.06198].

| Category | Core focus | Evaluation aspects |
|---|---|---|
| Integration | Multi-document evidence integration | Multi-source integration with \(|C^{+}| \in \{2,3\}\) |
| Reasoning | Inference and calculation over retrieved evidence | Multi-hop reasoning; numerical calculation |
| Logic | Logical and semantic alignment under expression mismatch | Synonym interpretation; numerical inclusion; conceptual inclusion |
| Table | Tabular understanding in retrieved chunks | HTML; HTML with merged cells; Markdown; CSV |
| Abstention | Refusal under unavailable or incoherent evidence | Insufficient evidence; contradictory evidence; incomplete chunk |

Integration targets cases where multiple relevant chunks contain complementary or differently formatted evidence. Reasoning covers both multi-hop inference across documents and numerical calculation such as totals, averages, profit margins, or growth rates. Logic targets semantic equivalence and inclusion, including unit conversion, numeric range interpretation, and hierarchical concept relations. Table evaluates whether the model can correctly parse tables under structural variation and chunk disruption. Abstention evaluates whether the model can refuse when \(C^{+} = \emptyset\), when retrieved evidence is contradictory, or when chunk boundaries make the question unanswerable [2603.06198].

A central design feature is category co-occurrence. The benchmark explicitly permits compound tasks such as multi-hop reasoning over an HTML table with merged cells. In the paper’s example, identifying a research theme adopted by a fictional organization requires linking a partnership statement to a grant table and correctly interpreting a rowspanned or colspan-structured table entry [2603.06198].

## 3. Dataset construction, fictionalization, and bilingual release

The dataset is defined as
\[
D = \{(q_i, a_i, C_i^{+}, C_i^{-}, \Psi(q_i)) \mid 1 \le i \le m\},
\]
where each instance contains a question \(q_i\), reference answer \(a_i\), relevant chunks \(C_i^{+}\), distractor chunks \(C_i^{-}\), and aspect labels \(\Psi(q_i)\). A key constraint is
\[
|C_i^{+}| + |C_i^{-}| \ge 8,
\]
which is intended to simulate realistic long-context RAG conditions [2603.06198].

Construction was human-centered. Three native Japanese speakers designed scenarios and questions, using GPT-5 only as an auxiliary drafting tool for scenario text and candidate documents. All items were manually reviewed, filtered by two independent annotators, and retained only when both approved. Quality checks required that the question match the intended aspect pattern, that \(a_i\) be derivable from \(C_i^{+}\), that the item not be answerable from pre-trained knowledge alone, and that fictional information not contradict real-world facts [2603.06198].

The benchmark follows the methodology of Kirchenbauer et al. in using fictional companies, products, people, and institutions. This design suppresses leakage from the model’s internal knowledge and reduces spurious success due to memorized facts. Correct answering is therefore intended to require grounding in the provided documents rather than recall from pretraining [2603.06198].

Chunks are normalized to approximately **512 tokens** using tiktoken, and distractor chunks are topically related but do not contain the answer. For tables, large tables may be chunked, with header information repeated in each chunk, and chunk order is shuffled to simulate retriever output order and reduce position bias [2603.06198].

The Japanese dataset contains **114 questions**. The Main categories contribute **54 questions**, of which **12** have a single aspect and **42** have two aspects. Abstention contributes **54 Insufficient Evidence** questions by dropping \(C^{+}\), plus **3 Contradictory Evidence** questions and **3 Incomplete Chunk** questions, yielding the total of 114. An English dataset of equal size was created by GPT-5 translation of every Japanese instance with human curation [2603.06198].

## 4. Evaluation protocol, scoring, and model suite

At evaluation time, the generator receives a task instruction \(\tau\), a question \(q_i\), and a concatenated context \(C_i = C_i^{+} \cup C_i^{-}\), with chunk order randomized per run. For Main categories, the generator input is
\[
x_g = (\tau, q_i, C_i).
\]
For Insufficient Evidence abstention cases, the input is
\[
x_g = (\tau, q_i, C_i^{-}),
\]
with \(C_i^{+} = \emptyset\) [2603.06198].

The output is a single natural-language answer \(y_i\). For abstention items, correctness requires an explicit statement that the question cannot be answered from the given context, or, in contradictory-evidence cases, an explicit acknowledgement of contradiction as encoded in the reference answer [2603.06198].

Automatic scoring is performed by an evaluator LLM \(\mathcal{J}\), instantiated as **GPT-4.1 (2025-04-14)**. The judge computes a binary decision
\[
\mathcal{J}(q_i, a_i, y_i) =
\begin{cases}
1 & \text{if } y_i \text{ is semantically consistent with } a_i,\\
0 & \text{otherwise.}
\end{cases}
\]
No partial credit is assigned [2603.06198].

Category-wise accuracy is defined over the subset \(Q_{\theta}\) of questions labeled with category \(\theta\):
\[
\text{Accuracy}(\theta) = \frac{1}{|Q_{\theta}|} \sum_{q_i \in Q_{\theta}} \mathcal{J}(q_i, a_i, y_i).
\]
Overall accuracy is the average of category-wise accuracies across all five categories:
\[
\overline{\text{Accuracy}} = \frac{1}{|\Theta|} \sum_{\theta \in \Theta} \text{Accuracy}(\theta).
\]
If a question involves multiple aspects from different Main categories and is answered correctly, it is counted as correct for all relevant categories [2603.06198].

The benchmark evaluates two model groups. The API-based group includes GPT-5, GPT-5-mini, GPT-5-nano, o3, o4-mini, GPT-4.1, GPT-4.1-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, and Claude-Sonnet-4. The open-weight group includes Gemma-3-27B-Instruct, Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Qwen3-235B-A22B-Instruct, and Qwen3-235B-A22B-Thinking. Reasoning models are evaluated with their maximum reasoning token length, and the reported decoding settings are temperature \(0.0\) and top\_p \(1.0\) [2603.06198].

## 5. Empirical results and observed failure modes

The headline empirical result is that **no model achieves \(\overline{\text{Accuracy}} \ge 0.9\)** in either Japanese or English [2603.06198]. The strongest overall model is **GPT-5**, with overall accuracy reported as approximately **0.872** in both languages. Among open-weight models, **Qwen3-235B-A22B-Instruct** reaches **0.859**, while **Qwen3-235B-A22B-Thinking** reaches **0.821** [2603.06198].

Category-wise results are heterogeneous. For Integration, top models reach roughly **0.83–0.92** in each language. For Reasoning, **o3** achieves **0.957** in Japanese and **0.870** in English, and is described as best on numerical reasoning. For Logic, several models cluster around **0.83–0.90**. For Table, **Gemini-2.5-Flash** achieves **0.871** in Japanese and **0.903** in English, the strongest table performance reported. For Abstention, **Claude-Sonnet-4** achieves **0.950** in Japanese and **0.967** in English, the best abstention score [2603.06198].

The benchmark’s analysis emphasizes that strong abstention can coincide with **over-abstention** on answerable cases. The reported average over-abstention rates on Main questions are **0.259** for Claude-Sonnet-4, **0.213** for Llama-3.1-8B-Instruct, **0.157** for Llama-3.3-70B-Instruct, **0.120** for Qwen3-235B-A22B-Thinking, **0.065** for GPT-5, and **0.028** for GPT-4.1-mini. This is presented as a safety–usefulness trade-off: strong abstention behavior can reduce hallucination but also reduce utility [2603.06198].

Qualitative analysis identifies distinct failure patterns by category. In Integration, models often fail when relevant chunks have no direct lexical match with the question or when only one document contains a special note that changes the final answer. In Reasoning, even strong models can reason qualitatively correctly yet make small arithmetic errors, while smaller models may fail to identify intermediate facts or refuse to answer when cues are implicit. In Logic, typical mistakes include unit mismatches such as returning “500 MB” when the question requires GB, and incorrect rendering of large Japanese number units such as “760 million yen” instead of “7.6 billion yen.” In Table, most models parse simple tables reasonably well but struggle with HTML tables containing merged cells, and with tables split across shuffled chunks, even when headers are duplicated [2603.06198].

Abstention errors are especially revealing. On Insufficient Evidence items, many models hallucinate plausible but unsupported answers. On Contradictory Evidence items, Claude-Sonnet-4 is described as consistently identifying inconsistencies and abstaining, whereas other models often rely on a single chunk and ignore contradiction. On Incomplete Chunk items, most models abstain, but some hallucinate by relying on general knowledge despite ambiguity in the supplied context [2603.06198].

## 6. Position in the benchmark landscape, limitations, and future directions

LIT-RAGBench is presented as a unified generator benchmark that complements rather than replaces other RAG evaluations. Relative to prior benchmarks such as RGB, FRAMES, RAGBench, RAGTruth, CRAG, and MMQA, it is characterized as a single framework covering multi-document integration, multi-hop and numerical reasoning, logical or semantic inclusion, structurally diverse tables, and abstention under realistic failure modes, while holding retrieval fixed so that generator behavior can be isolated [2603.06198]. RAGBench, by contrast, emphasizes explainable span-level evaluation for full RAG systems through TRACe across a large multi-domain benchmark [2407.11005].

A plausible implication is that LIT-RAGBench and **RAGCap-Bench** occupy complementary layers of the evaluation stack: LIT-RAGBench isolates generator behavior under fixed retrieved context, whereas RAGCap-Bench evaluates process-level capabilities such as planning, evidence extraction, grounded reasoning, and noise robustness in agentic RAG workflows [2510.13910]. A broader benchmark ecology is also visible. **SMARTFinRAG** studies document-centric financial RAG with joint retrieval and generation evaluation [2504.18024]. **Legal RAG Bench** evaluates end-to-end legal RAG and decomposes failures into retrieval errors, reasoning errors, and hallucinations [2603.01710]. **RAGRouter-Bench** addresses adaptive paradigm selection across 7,727 queries and 21,460 documents and argues that no single RAG paradigm is universally optimal [2602.00296]. This suggests that LIT-RAGBench is most naturally interpreted as a generator-specialized component within a larger modular evaluation program for RAG systems.

The benchmark’s stated limitations are substantial. The dataset size—**114 questions per language**—is intentionally compact and high quality but small compared with large-scale benchmarks. Aspect imbalance is explicit: within Abstention, Insufficient Evidence dominates, while Contradictory Evidence and Incomplete Chunk are scarce. The fictional-world design is useful for avoiding contamination but may not capture all properties of real web or enterprise data, such as OCR noise or messy formatting. Evaluation depends on GPT-4.1 as judge and therefore inherits LLM-as-a-Judge bias. The benchmark also aims to capture “minimum occurrence patterns” for each aspect rather than exhaustive coverage [2603.06198].

Future work is directed toward scaling and diversifying the dataset while retaining human-verifiable quality, extending toward **Agentic RAG**, expanding to additional languages, and studying more nuanced logical and abstention scenarios. The benchmark is released as open source, with dataset, prompt templates, evaluation code, category-wise and overall scoring scripts, and over-abstention analysis available in the associated repository [2603.06198].

Source: https://www.emergentmind.com/topics/lit-ragbench