LiveRAG Benchmark Challenge

• LiveRAG Challenge is a competitive benchmark for retrieval-augmented generation that uses a fixed corpus (FineWeb-10BT) and a standardized generator (Falcon3-10B-Instruct).
• It focuses on evaluating retrieval methods, query reformulation, reranking, and evidence selection in a pseudo-live, time-constrained setting.
• The challenge fostered diverse methodological approaches, highlighted trade-offs between latency and quality, and evolved into a public benchmark enriched with psychometric metadata.

The LiveRAG Challenge was a SIGIR 2025 competitive benchmark for retrieval-augmented generation in which teams answered unseen, DataMorgana-generated questions under a strict live deadline while operating over a fixed external corpus, FineWeb-10BT, and a shared answer generator, Falcon3-10B-Instruct. By fixing the corpus and the final generation model, the challenge concentrated comparison on retrieval, query reformulation, reranking, evidence selection, and prompting rather than on frontier-model scale. Its public afterlife is the LiveRAG benchmark, a release of the challenge data enriched with ground-truth answers, supporting claims, and psychometric metadata for systematic post hoc evaluation (Carmel et al., 7 Jul 2025, Carmel et al., 18 Nov 2025).

1. Origins, scope, and competitive rationale

LiveRAG was organized to evaluate practical RAG systems in a setting that was simultaneously rigorous, fair, and pseudo-live. The organizers explicitly treated RAG evaluation as an open problem and designed the challenge to reduce overfitting and resource inequality by holding the corpus and final generator fixed, providing shared infrastructure, and requiring systems to answer fresh questions within a short window rather than on a static offline test set (Carmel et al., 7 Jul 2025).

The retrieval source was FineWeb-10BT, described in the challenge report as a randomly sampled subset of FineWeb containing about 15 million web documents derived from cleaned and de-duplicated CommonCrawl content. The report notes that, despite cleaning, the corpus still contains some toxic or offensive material and non-English pages, making retrieval harder than on tightly curated collections. The final answer generator was fixed to Falcon3-10B-Instruct; auxiliary tools for query rewriting, reranking, and prompt construction were permitted, but the competition was explicitly oriented toward moderate-resource pipelines rather than unrestricted model scaling (Carmel et al., 7 Jul 2025).

Participation was substantial. The challenge report records 70 team applications from 27 countries, with 40 teams allocated compute resources; the participating teams comprised more than 140 members from 16 countries, and about 77% of members were affiliated with academic institutions. This framing positioned LiveRAG not only as a leaderboard, but as a controlled comparison platform for RAG design choices (Carmel et al., 7 Jul 2025).

2. Corpus access, live protocol, and task format

The organizers supplied pre-built sparse and dense retrieval infrastructure over chunked FineWeb-10BT. Documents were segmented into non-overlapping, sentence-based chunks of up to 512 tokens. Two main indices were offered: an OpenSearch sparse index using BM25 with default parameters, and a Pinecone dense index built from E5-base-v2 embeddings with 768-dimensional vectors and Pinecone’s Slab architecture. Participants could build their own indices, but these provided baselines standardized the retrieval substrate across teams (Carmel et al., 7 Jul 2025).

The live event was divided into two sessions based on time zone, scheduled for 07:00–09:00 UTC and 15:00–17:00 UTC. The challenge report states that the evaluation window was two hours, corresponding to an average budget of about 14 seconds per question. The benchmark-release paper clarifies that each session received 500 unseen questions, with 105 shared between sessions for calibration and manual validation; after merging the two session sets and removing duplicates, the public benchmark contained 895 unique question-answer pairs (Carmel et al., 7 Jul 2025, Carmel et al., 18 Nov 2025).

The required submission format was unusually rich. For each question, teams had to return the final answer, the passages used for prompt augmentation, and the final prompt sent to Falcon3. This made it possible to assess not only end answers but also grounding behavior and evidence selection. A plausible implication is that the challenge rewarded pipelines that treated context curation as a first-class systems problem rather than as a byproduct of retrieval (Carmel et al., 7 Jul 2025).

3. Evaluation framework: correctness, faithfulness, and manual review

LiveRAG used a two-stage evaluation. The automated stage employed Claude-3.5-sonnet as an LLM judge to compute Correctness and Faithfulness. The manual stage reviewed the top 13 teams by automated Correctness—top 5 from Session 1 and top 8 from Session 2—using more than a dozen annotators and a rank-based aggregation protocol (Carmel et al., 7 Jul 2025).

Correctness was defined through two claim-based components: Coverage and Relatedness. Coverage measured how much of the vital content in the reference answer appeared in the system answer, with Direct claims weighted more heavily than Useful claims:

$$Cov(a,r)=\alpha \sum_{c \in D_r}\frac{NLI(a, c)}{|D_r|} + (1-\alpha)\sum_{c \in U_r}\frac{NLI(a, c)}{|U_r|}\ ,$$

where $\alpha = 0.7$ unless $U_r$ is empty. Relatedness measured the proportion of answer claims that were relevant to the question:

$Rel(a) = \frac{|D_a|}{|D_a| + |I_a|}.$

The final Correctness score was the harmonic mean of Coverage and Relatedness, and the challenge report notes that Correctness was scaled to $[-1,2]$ following the CRAG convention (Carmel et al., 7 Jul 2025).

Faithfulness measured whether answer claims were supported by the retrieved passages:

$$F(a, R) = \sum_{c \in C_a}\frac{NLI(c, R)}{|C_a|}\ ,$$

with $NLI(c,R)=\max_{r\in R} NLI(c,r)$. Faithfulness was scored on $[-1,1]$. Operationally, only the first 300 words of each response were evaluated, and only the first 10 submitted passages were considered for Faithfulness. This suggests that long answers and very deep evidence lists had bounded marginal value under the official metric (Carmel et al., 7 Jul 2025).

Manual review scored Coverage, Relatedness, and Quality on 0–2 Likert scales and aggregated results per question using Borda counts. The top four teams in automated Correctness were also the top four in manual ranking. Over the shared 105-question set, the challenge report gives correlations between automated Correctness and manual evaluation of 0.8826 for Borda, 0.8240 for Coverage, 0.8490 for Quality, and 0.6021 for Relatedness. The same report records that all participating teams outperformed the no-RAG baseline, validating the core premise that retrieval materially improved Falcon3-based QA in this setting (Carmel et al., 7 Jul 2025).

4. Public benchmark release and psychometric analysis

The public LiveRAG benchmark transformed the competition into a reusable research resource. It released 895 synthetic question-answer pairs derived from the two challenge sessions and augmented them with artifacts unavailable to competitors during the live event: the DataMorgana-generated reference answer, the supporting documents used to generate it, and answer claims labeled as Direct, Useful, or Useless for claim-level scoring. The dataset also exposes session labels, average correctness scores over participating systems, score standard deviations, and Item Response Theory parameters inferred from challenge responses (Carmel et al., 18 Nov 2025).

The question set is intentionally heterogeneous. The benchmark includes single-document and multi-document questions, and DataMorgana categories span answer type, answer style, premise, phrasing, linguistic variation, politeness, linguistic correctness, and user persona. Answer types include factoid, yes/no, definition, list, explanation, comparison, and multi-aspect. This matters because the benchmark was designed not merely to rank systems, but to differentiate system behavior across varied information needs and difficulty levels (Carmel et al., 18 Nov 2025).

A major contribution of the release is its psychometric modeling. The authors fit a two-parameter logistic IRT model, estimating per-question difficulty $b_i$ and discrimination $a_i$ from the correctness matrix of systems against items. The reported results indicate that difficulty and average correctness are strongly negatively correlated, with Pearson correlation $\alpha = 0.7$0, whereas discrimination and difficulty are only weakly negatively correlated, with Pearson $\alpha = 0.7$1. The latent skill estimates derived from challenge responses also aligned closely with official rankings, with Kendall’s tau of 0.766 for Session 1 and 0.999 for Session 2 (Carmel et al., 18 Nov 2025).

The release paper further characterizes which questions were hard. Multi-document items were substantially harder than single-document ones; comparison and multi-aspect questions were the toughest answer types; yes/no items were harder than expected; concise-answer questions were somewhat harder than detailed-answer or unspecified ones; search-style questions were harder than natural questions; distant paraphrases were harder than document-similar phrasing; typos increased difficulty; premise and politeness had little effect; and expert-persona questions were slightly easier than novice ones. In addition, LiveRAG is reported to have the highest lexical diversity among the compared benchmarks, with NGD of 3.062, the highest length entropy at 3.207, and a PoS compression ratio of 5.220. The release paper also notes that top LiveRAG teams using RAG on top of Falcon3-10B could beat GPT-4.1 without RAG, supporting the claim that retrieval, rather than base-model scale alone, was decisive on the long tail of difficult questions (Carmel et al., 18 Nov 2025).

5. Representative system families and leaderboard behavior

The challenge produced a broad methodological spectrum, from query-aware preprocessing and multi-agent planning to clustering, nugget extraction, and graph-enhanced retrieval. The table summarizes several representative submissions.

Submission	Distinctive design	Reported result
RMIT-ADM+S / GRAG (Ran et al., 17 Jun 2025)	Hypothetical answer generation before retrieval, hybrid RRF fusion, pointwise LLM reranking	Relevance 1.199, Faithfulness 0.477, top four finalists
PreQRAG / UDInfo (Martinez et al., 20 Jun 2025)	Single-document vs multi-document classification, question rewriting or decomposition, hybrid retrieval and reranking	Preliminary 2nd in Session 2; Correctness 1.200586, Faithfulness 0.623175
RAGtifier (Cofala et al., 17 Jun 2025)	Pinecone retrieval, BGE reranker, InstructRAG prompting, inverted context order	Correctness 1.13, Faithfulness 0.55, 4th place
HLTCOE (Duh et al., 27 Jun 2025)	GPT-Researcher workflow with ColBERT/PLAID-X retrieval, Qwen2.5 query generation, m2-bert snippet filtering	5th in automatic correctness; Correctness 1.070111, Faithfulness 0.340711
mRAG / CIIR (Salemi et al., 12 Jun 2025)	Coordinator-centered multi-agent RAG with self-training over sampled trajectories	Correctness 0.996, Faithfulness 0.418, 7th overall among top 20 teams
TopClustRAG (Bakagianni et al., 18 Jun 2025)	Hybrid retrieval followed by K-Means clustering, cluster-specific prompting, intermediate-answer synthesis	2nd in Faithfulness at 0.460062; 7th in Correctness at 0.685146

These systems collectively show that LiveRAG was not dominated by a single architectural doctrine. Some teams prioritized retrieval precision through preprocessing and reranking, as in PreQRAG and RAGtifier; others operationalized iterative search, as in HLTCOE’s GPT-Researcher pipeline and CIIR’s multi-agent mRAG; and others emphasized evidence structure, as in TopClustRAG’s cluster-conditioned synthesis (Martinez et al., 20 Jun 2025, Cofala et al., 17 Jun 2025, Duh et al., 27 Jun 2025, Salemi et al., 12 Jun 2025, Bakagianni et al., 18 Jun 2025).

Other submissions extended the design space further. “Graph-Enhanced RAG” adapted GeAR to the challenge with hybrid dense-sparse retrieval, proximal triple extraction, and online alignment to Wikidata, reporting preliminary automatic evaluation of 0.875714 Correctness and 0.529335 Faithfulness while also exposing semantic drift between FineWeb chunks and external triples as a central failure mode (Shen et al., 23 Jul 2025). UiS-IAI’s nugget-based pipeline treated atomic information nuggets as the basic unit of generation and reported that combining the original query with a few sub-query rewrites boosted recall, whereas increasing the number of documents for reranking and generation beyond a certain point reduced effectiveness without improving response quality (Łajewska et al., 27 Jun 2025).

6. Research lessons, limitations, and subsequent influence

Several submission papers converged on a common lesson: retrieval and context construction often mattered more than simply enlarging the final generator. HLTCOE report that switching from Falcon3-10B to a much larger Llama3.1-70B did not clearly improve output quality, and interpret this as evidence that retrieval and context construction mattered more than generator size in their LiveRAG pipeline (Duh et al., 27 Jun 2025). PreQRAG similarly argues that question-type awareness is crucial, especially because misclassifying a multi-document question as single-document can cause insufficient evidence retrieval (Martinez et al., 20 Jun 2025).

The challenge also made latency-quality trade-offs explicit. A hybrid-retrieval submission found that RankLLaMA reranking improved MAP from 0.523 to 0.797, a 52% relative improvement, but increased time from 1.74 seconds to 84.37 seconds per question; the same study reported that vocabulary alignment was the strongest predictor of performance on its development set, with document-similar phrasing improving cosine similarity from 0.562 to 0.762 (Fensore et al., 27 Jun 2025). RAGtifier likewise concluded that dense retrieval, BGE reranking, careful prompt design, and context ordering mattered more than elaborate multi-round reasoning under Falcon3 and two-hour live constraints (Cofala et al., 17 Jun 2025).

The benchmark release and challenge report also state clear limitations. LiveRAG’s questions are synthetic; the released reference answers are not the only valid answers; IRT parameters are learned from systems that all used Falcon3-based RAG setups, so some bias is possible; and the automated evaluation, while strongly correlated with manual review, still required manual calibration via the shared 105-question seed set and top-team adjudication (Carmel et al., 18 Nov 2025, Carmel et al., 7 Jul 2025). Another common misconception is that “live” meant unrestricted web retrieval. In fact, the challenge fixed the corpus to FineWeb-10BT and standardized final answer generation on Falcon3-10B-Instruct; the live aspect lay in pseudo-live evaluation on unseen questions under strict time constraints (Carmel et al., 7 Jul 2025).

LiveRAG’s influence extended beyond the original event. R2RAG in the NeurIPS 2025 MMU-RAG Competition is explicitly positioned as an extension of the team’s earlier LiveRAG system and adds dynamic routing between single-pass RAG and iterative agentic retrieval, with an evidence-sufficiency stopping rule designed for single-GPU deployment (Ran et al., 24 Feb 2026). RAGExplorer, though not a LiveRAG paper, addresses a directly adjacent problem: comparative diagnosis of multiple RAG configurations through a macro-to-micro visual analytics workflow, which is highly aligned with the debugging and systems-analysis demands that a challenge such as LiveRAG creates (Tian et al., 19 Jan 2026).

In this sense, LiveRAG functioned both as a competition and as an experimental regime. It standardized corpus and generator, exposed sharp trade-offs among recall, grounding, latency, and abstention, and then evolved into a public benchmark whose claim annotations and psychometric metadata support systematic study of why some RAG systems fail and others generalize across difficulty strata (Carmel et al., 7 Jul 2025, Carmel et al., 18 Nov 2025).