LexRAG: Benchmark for Legal RAG Systems

• LexRAG Benchmark is a comprehensive evaluation suite tailored for multi-turn legal RAG systems, emphasizing accurate retrieval and citation-based legal reasoning.
• It uses expert-curated multi-turn dialogues and a corpus of legal statutes to simulate realistic consultations and gauge retrieval performance (e.g., Recall@10 ≈33%).
• The integrated LexiT toolkit offers modular processing, retrieval, and generation pipelines for reproducible experiments and detailed multi-dimensional performance assessments.

A LexRAG Benchmark is a specialized evaluation suite for Retrieval-Augmented Generation (RAG) systems, with an explicit focus on multi-turn legal domain question-answering where document retrieval, context interpretation, and legal reasoning interact in a highly structured, citation-centric workflow. LexRAG provides multi-turn legal consultation dialogues paired with expert-curated retrieval and answer annotations, introduces domain-tailored RAG pipelines (the LexiT toolkit), and adopts both reference-based and advanced LLM-as-a-judge evaluation frameworks to diagnose retrieval precision and generation quality. LexRAG arises in the context of the broader RAG evaluation landscape, building upon granular metrics (e.g., TRACe, CCRS), modular design philosophies (as seen in XRAG, mmRAG), and addressing the unique complexities of legal consultation, including statute tracking, dialogue progression, and legal citation faithfulness.

1. Motivation for LexRAG and Domain Challenges

LexRAG was developed in response to the lack of rigorous benchmarks for evaluating RAG systems in high-stakes, multi-turn legal consultations (Li et al., 28 Feb 2025). The underlying motivation stems from several domain-specific requirements:

• Legal consultations involve progressive, multi-turn clarifications and require precise legal grounding, which standard single-turn or open-domain RAG benchmarks do not address.
• Legal responses must cite statutory authority, resolve complex anaphora, and offer context-aware interpretations, demanding a retrieval task that operates beyond simple keyword or semantic matching.
• LexRAG fills this gap by providing a realistic dataset and a methodology for both retrieval evaluation (does the system find the correct statute?) and generation (is the legal advice sound, clear, and correctly attributed?).

2. Dataset Structure and Annotation Protocols

The LexRAG dataset comprises 1,013 multi-turn dialogues, each containing five consecutive rounds, paired with a corpus of 17,228 legal articles from 222 statutes (Li et al., 28 Feb 2025). Dialogues are annotated by legal experts and include:

• Explicit answer texts
• Reference to the supporting legal articles (by statute/article)
• Annotated keywords for legal intent tracking

Each round in a dialogue simulates a practical consultation, where legal issues emerge and evolve. The retrieval task for round $t$ is, given the conversation history $H = \{q_1, r_1, ..., q_{t-1}, r_{t-1}, q_t\}$, to select an article subset $A_t$ from corpus $\mathcal{D}$ that supports a legally correct response.

Annotation protocols enforce reliable grounding:

• Answers must be supported by explicit articles, enabling citation-based evaluation
• Annotations include legal keyword identification for precise intent mapping
• Legal expert review ensures annotation quality and real-world legal correctness

This high annotation fidelity enables both document-level retrieval evaluation (Recall@k, citation accuracy) and answer-level assessment (faithfulness, completeness).

3. System Pipeline and LexiT Toolkit

LexRAG is paired with the LexiT toolkit—a modular implementation framework enabling reproducible RAG experiments in legal consultation settings (Li et al., 28 Feb 2025). LexiT includes:

• Processors: Transform dialogue context into retrieval queries (options include last query only, full dialogue context, query rewriting via LLMs, or concatenated queries).
• Retrievers: Support classical methods (e.g., BM25 via Pyserini) and dense embeddings (BGE, GTE).
• Generators: Interface with scalable generation backends (e.g., Hugging Face, vLLM) for legal response generation.

LexiT supports prompt engineering and pipeline customization for legal-specific context integration, and is open-sourced for reproducibility.

4. Evaluation Methodologies

LexRAG employs both reference-based and LLM-as-a-judge evaluation protocols:

Reference-based evaluation: Assess retrieval effectiveness by measuring inclusion of the gold reference articles ($|A_t \cap A^{ref}_t|$), typically via Recall@k. For response generation, compare generated answers against expert-written gold responses, focusing on factual consistency and legal correctness.

LLM-as-a-judge pipeline: An LLM is prompted to assign scores (1–10) to each response in five dimensions: factuality, user satisfaction, clarity, logical coherence, and completeness. This uses a chain-of-thought rationale (judge critiques response according to evidence and reference standard, then aggregates into a numeric score) (Li et al., 28 Feb 2025). The evaluation prompt explicitly discourages length bias, focusing on quality. This methodology closely tracks advanced frameworks such as CCRS (Muhamed, 25 Jun 2025), which formalize the multi-dimensional scoring as

$$\text{Metric Score} = \frac{\text{LLM}_\text{Judge}(r, q, C)}{100}$$

over a suite of metrics capturing coherence, relevance, information density, correctness, and recall.

Automated evaluation enables large-scale, fine-grained assessment with consistency near that of human annotators (validated in related LLM-as-a-judge works).

5. Experimental Findings and Model Limitations

Experiments using LexRAG reveal several characteristics of legal-domain RAG systems (Li et al., 28 Feb 2025):

• Retrieval remains challenging: even best dense retrievers with LLM query rewriting only reach Recall@10 ≈ 33%.
• Generative models show significant dependency on accurate retrieved context. With gold articles provided, top LLMs (Qwen-2.5-72B-Instruct, GLM-4) approach expert reference quality, but zero-shot generation (no provided articles) consistently falls short in completeness and citation.
• Dense retrieval outperforms lexical BM25 when appropriately tuned, but recall and precision remain well below human expert levels, underscoring the difficulty of multi-turn, context-sensitive legal retrieval.

A plausible implication is that further research is necessary for both retrieval technology (e.g., better query rewriting, anaphora disambiguation) and domain-adapted generation strategies.

6. Positioning in the RAG Benchmark Landscape

LexRAG extends the methodology and diagnostic philosophy found in general-purpose RAG benchmarks:

• The modular, explainable structure of TRACe (Relevance, Utilization, Completeness, Adherence) from RAGBench (Friel et al., 25 Jun 2024) helps inform LexRAG’s analytic decomposition, though LexRAG emphasizes citation and conversational context compatibility.
• XRAG’s four-phase analytical pipeline (pre-retrieval, retrieval, post-retrieval, generation) (Mao et al., 20 Dec 2024) inspires LexiT’s pipeline modularity and supports the diagnosis of dialog-aware retrieval failures and generation-specific deficits.
• Multi-dimensional, LLM-based grading such as that employed in CCRS (Muhamed, 25 Jun 2025) and LRAGE (Park et al., 2 Apr 2025) is operationalized in LexRAG’s automated judge, but tailored for legal and conversational settings.

Table: LexRAG Features in Comparison to Related Benchmarks

Benchmark	Domain/Focus	Retrieval Evaluation	Conversation	Citation Fidelity
RAGBench	General/Industry	Yes (TRACe)	Single-turn	Partial
LegalBench-RAG	Legal	Yes (Precision@k)	Single-turn	Explicit
XRAG	General	Yes (modular)	Yes	Not the focus
LexRAG	Legal	Yes (Recall@k, citation)	Multi-turn	Explicit/ref-based
LRAGE	Legal, Multilingual	Yes (rubric, GUI)	Single/multi	Partial

7. Practical Impact, Limitations, and Future Directions

LexRAG is positioned as a reference benchmark for developing legal consultation systems capable of robust multi-turn reasoning, article citation, and faithful answer generation:

• It enables diagnostic benchmarking of retrieval and generation faults over realistic consultation scenarios.
• Open-source resources (dataset, LexiT toolkit) foster reproducibility and extension, supporting research into multilingual or cross-jurisdictional legal QA.

Current limitations include modest retrieval recall, highlighting research challenges in corpus coverage, query understanding, and legal intent mapping. The integration of advanced evaluation methodologies such as those in CCRS (Muhamed, 25 Jun 2025) suggests opportunities for more nuanced, multi-faceted assessment. Potential future directions involve:

• Adapting LexRAG for cross-lingual and diverse legal systems
• Developing ensemble or hybrid evaluation strategies (e.g., chain-of-thought LLM juries or lightweight graders as in (Jeong, 17 Jun 2025))
• Extending benchmark scenarios to mirror emerging legal tasks (precedents, argumentation, evidentiary reasoning)

As legal AI continues to mature, LexRAG provides a robust foundation for both precise diagnostic research and credible, citation-grounded system evaluation in the legal domain.