LexiT Toolkit for Legal RAG Systems

• LexiT Toolkit is a modular framework for legal RAG that converts legal dialogue into queries using advanced query rewrite strategies to resolve ambiguity.
• It integrates classical lexical and dense retrieval methods with mainstream LLM toolkits to ensure precise legal article citations and reliable response generation.
• Its evaluation module employs an LLM-as-a-judge pipeline measuring factuality, clarity, and legal compliance to benchmark multi-turn dialogue performance.

The LexiT Toolkit is an open-source modular framework developed for the systematic construction, evaluation, and benchmarking of retrieval-augmented generation (RAG) systems in the legal domain. Purpose-built in conjunction with LexRAG—a benchmark corpus comprising multi-turn legal consultation dialogues and a comprehensive set of legal articles—LexiT enables researchers and practitioners to implement, evaluate, and refine RAG components that address the unique requirements of legal conversational AI. It encompasses all core RAG functionality: preprocessing and query formulation, document retrieval (both lexical and dense), response generation, and multi-faceted evaluation, including an LLM-as-a-judge pipeline designed for the legal context (Li et al., 28 Feb 2025).

1. System Architecture and Workflow

LexiT follows a modular design with distinct data, pipeline, and evaluation components to ensure extensibility and reproducibility. The toolkit processes legal consultations—single-turn and multi-turn—by integrating inputs from several corpora:

• Legal Articles (17,228 candidates from LexRAG corpus),
• Legal Books, and
• Legal Cases.

The pipeline comprises:

• Processor Module: Converts raw conversation history into queries. Strategies include taking only the last query, concatenating the full conversation, leveraging all previous queries, or a query rewrite strategy (invoking an LLM to reformulate context-dependent questions). The query rewrite method is pivotal for resolving reference ambiguity in nuanced legal exchanges.
• Retriever Module: Implements classical lexical matching algorithms (BM25, QLD, via Pyserini) and dense retrieval techniques (BGE, GTE models). The canonical BM25 scoring is formally given by

$$\text{Score}(q, d) = \sum_{t} \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1 + 1)}{f(t, d) + k_1 \cdot (1 - b + b \cdot |d|/\text{avgdl})}$$

This module enables direct empirical comparison of lexical versus neural retrieval for legal content.

• Generator Module: Employs mainstream LLM inference toolkits (vLLM, Huggingface, etc.), combining queries and retrieved documents to generate legally grounded responses.
• Evaluation Module: Separates metrics for retrieval (Recall, NDCG, Precision, MRR) and generation (ROUGE, BLEU, METEOR, BERTScore), with additional legal-specific assessment via an LLM-as-a-judge.

2. LexRAG Benchmark Integration

LexiT’s close integration with LexRAG ensures standardized and domain-relevant evaluation. It directly utilizes:

• The multi-turn dialogue samples and the legal article candidate pool from LexRAG.
• Preprocessing strategies customized for legal dialogues, such as advanced anaphora resolution and topic disambiguation.
• Workflow stages that closely align with LexRAG’s benchmark tasks: conversational knowledge retrieval and legally sound response generation.

This supports plug-and-play experimentation, allowing researchers to substitute individual RAG pipeline components for targeted ablation and comparative studies.

3. Legal-Domain Specialization

LexiT incorporates features essential for operational accuracy in legal contexts:

Feature	Legal Motivation	Toolkit Implementation
Citation-based grounding	Ensures answer accountability/verifiability	Generator enforces article refs
Multi-turn conversation	Supports evolving user context, anaphora	Full-context and query-rewrite
Modular design	Customizable RAG permutations	Independent pluggable modules

The toolkit’s design enforces the inclusion of explicit legal article citations in responses, critical for traceability and compliance. The query processing can ingest entire conversational histories or leverage LLM-mediated rewrites—an approach shown to enhance performance when prior user queries introduce ambiguous referents.

4. Mathematical Formulation and Algorithms

LexiT formalizes the conversational retrieval task as:

$$\text{Given dialogue history } H = \{q_1, r_1, \dots, q_t\}, \text{ retrieve legal articles } A_t = \{a_1, \dots, a_n\} \text{ from corpus } D.$$

This lays the groundwork for precise experimentation. The retrieval scoring, normalization, and generation steps align with accepted practices in IR and NLG, but LexiT’s legal focus necessitates stricter contextual and factual coherence compared to generic RAG systems.

5. LLM-as-a-Judge Evaluation Pipeline

LexiT’s evaluation pipeline leverages state-of-the-art LLMs (e.g., Qwen-2.5-72B-Instruct) to score generated answers pointwise across five axes:

• Factuality,
• User Satisfaction,
• Clarity,
• Logical Coherence,
• Completeness.

Assessment proceeds via chain-of-thought reasoning where the LLM compares each candidate response with benchmark references, scores each criterion, and aggregates results into a holistic rating (1–10 scale; expert benchmark at 8). The prompt template standardizes instructions, promoting legal-literate output handling. This method addresses the deficiencies of n-gram-based metrics for complex legal reasoning and is especially valuable for multi-turn dialogue scenarios.

6. Experimental Results and Insights

LexiT-facilitated experiments highlight critical system findings:

• Retrieval: Dense retrieval with query rewrite is performant; GTE-Qwen2-1.5B with rewritten queries achieves Recall@10 of 33.33%, the highest among tested configurations.
• Generation: Providing reference legal articles leads to superior LLM judge scores; retrieval-only contexts sometimes introduce irrelevant information.
• System Limits: Even frontier LLMs (Qwen-2.5-72B-Instruct) have room for improvement in interpreting and integrating retrieved content, especially for legal logic consistency.

These results strengthen the significance of preprocessing strategies and evaluation pipelines tailored for law. A plausible implication is that further gains in legal RAG quality are contingent on improvements in context modeling and explicit citation reasoning.

7. Significance and Research Directions

LexiT sets a precedent for reproducible, robust RAG experimentation in legal NLP:

• It supports scalable benchmarking and diagnosis of retrieval/generation errors unique to legal questioning.
• Citation-based and multi-turn dialogue management positions LexiT as a reference implementation for legal conversational agents.
• The modular framework enables granular ablation and rapid algorithmic prototyping.

This suggests substantial opportunity for optimizing query formulation and LLM evaluation schemas in future research. The inclusion of advanced evaluation (LLM-as-a-judge) reflects the increasing need for qualitative, expert-mimetic metrics in high-stakes domains. As legal conversational AI continues to evolve, LexiT provides foundational infrastructure for empirical analysis and system refinement, supported by ongoing benchmark development and toolkit releases (Li et al., 28 Feb 2025).