GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval

Abstract: Statutory retrieval is necessary for citation-grounded legal question answering, but remains underexplored for Greek. We introduce GreekBarRetrieval, a public retrieval benchmark derived from, and complementing GreekBarBench, which did not include retrieval. The new benchmark comprises 283 bar-exam questions, each accompanied by the facts of the case it refers to, and 6,308 candidate statutory articles to retrieve from. Questions and facts are stated in everyday language, but need to be mapped to the formal terminology of statutes and their abstract legal concepts. A further complication is that not all of the case facts are relevant to each question of a case. Experimenting with three BM25 variants and nine dense retrievers, we find that vanilla dense retrieval far outperforms vanilla sparse retrieval in Recall@100. However, LLM-based query reformulation helps BM25 close that gap, while also improving dense retrieval. With a ten-round ReAct-like LLM reformulation loop that we introduce, BM25 improves further in Recall@100 and obtains the best nDCG and MAP scores of all tested retrievers. Query reformulation also outperforms pseudo-relevance feedback, sparse-dense fusion, and English translation.

• The paper introduces a public benchmark of 283 Greek bar-exam queries, 6,308 candidate statutes, and 775 relevance judgments across five legal areas to isolate statutory retrieval.
• Dense retrieval outperforms vanilla BM25, with Gemini-001 reaching 0.77 Recall@100, while LLM query reformulation raises BM25 Recall@100 from 0.36 to 0.60.
• ReAct-BM25 achieves the strongest ranking quality, including 0.43 nDCG@10 and 0.67 Recall@100, but requires about 618,000 tokens per query, making reformulated dense retrieval the more practical default.

Greek statutory retrieval had no dedicated evaluation resource prior to this work. This paper introduces GreekBarRetrieval (2608.18752), a public benchmark for retrieving relevant Greek statutory articles given bar-exam questions and case facts, together with an extensive empirical study of sparse retrieval, dense retrieval, and LLM-based query reformulation in the Greek legal domain. The benchmark is derived from GreekBarBench [chlapanis-etal-2025-greekbarbench] and complements it: where GreekBarBench evaluates end-to-end legal answering with citations supplied as context (no true retrieval), GreekBarRetrieval isolates the retrieval step.

Benchmark construction and task characteristics

The benchmark is built from 310 Greek Bar examination questions spanning five legal areas (Civil, Criminal, Commercial, Public Law, and the Lawyers' Code). Each instance provides the facts of a case, a legal question about the case, and an official solution citing applicable statutes. The authors convert cited articles into SOURCE::ARTICLE identifiers (e.g., CrimC::42), drop questions whose citations are empty or reference articles outside the pool, and retain 283 queries against a retrieval pool of 6,308 candidate articles drawn from 23 legal sources. The result contains 775 relevance judgments covering 465 distinct articles (7.37% of the pool), with 2.74 gold articles per query on average; 56.9% of queries have multiple gold articles and 21.2% cite articles from more than one legal source.

The task is deliberately hard for three reasons. First, queries are phrased in everyday language while statutes use formal legal terminology and abstract concepts — a vocabulary mismatch aggravated by rich Greek morphology, which undermines exact term matching for sparse retrievers. Second, because bar exam files pair multiple distinct questions with identical case facts, each query concatenates a question with all case facts, some of which may be irrelevant to the specific legal issue asked. Third, answers frequently require combining articles distributed across several sources. A machine-translated English version of the benchmark is also released.

Experimental setup

The evaluation reports nDCG@10, nDCG@100, MAP@100, Recall@10, and Recall@100 via pytrec_eval, with Recall@100 as the primary coverage metric on the rationale that retrieved evidence must be available to downstream re-rankers or generator LLMs. Baselines comprise three BM25 variants differing in Greek-specific pre-processing (a Greek stemmer, spaCy lemmatization, and the gr-nlp-toolkit pipeline) and nine dense encoders ranging from Gemini-001 through Qwen3-8B/4B/0.6B, Euler-Legal-V1, Jina-v5-small, Arctic-v2, EmbGemma-300M, and Nomic-v1.5. Interventions include BM25 parameter tuning ($k_1$, $b$), TF-IDF term-expansion pseudo-relevance feedback (PRF), Rocchio-style positive-feedback PRF in embedding space, sparse-dense Reciprocal Rank Fusion (RRF), English translation via gpt-oss-120b, one-shot LLM query reformulation (keyword lists for BM25, prose rewrites for dense retrievers), and ReAct-BM25, a ten-round planner–observer loop inspired by ReAct in which retained articles from earlier rounds guide subsequent query generation. Final ReAct-BM25 rankings order retained articles by retention count across rounds, earliest retention round (to mitigate query drift), and best BM25 rank as tie-breaker.

Main findings

Vanilla dense retrieval dominates vanilla sparse retrieval. Gemini-001 reaches 0.77 Recall@100 versus 0.36 for BM25-GreekStemmer; among locally hosted models, Euler-Legal-V1 attains 0.68 and Qwen3-8B 0.67, with performance scaling within the Qwen3 family (0.67 / 0.62 / 0.45 for 8B / 4B / 0.6B). Nomic-v1.5 collapses to 0.08, plausibly because it was trained only on English data.

LLM query reformulation disproportionately benefits BM25. Reformulation lifts BM25 Recall@100 from 0.36 to 0.60 (+0.23) versus +0.06 for Qwen3-8B (0.67 → 0.73); all three sparse variants gain at least 0.20, while the seven Greek-capable dense encoders gain only 0.03–0.09. These differences are statistically significant after Holm correction. A qualitative example shows reformulation introducing the doctrinal term "breach of duty," moving a gold article from beyond rank 100 to rank 18 under BM25.

ReAct-BM25 achieves the best ranking quality at high cost. The ten-round loop raises BM25 Recall@100 further to 0.67 and yields the highest nDCG@10 (0.43), nDCG@100 (0.47), Recall@10 (0.52), and MAP@100 (0.37) of all tested systems. Its advantage over Reform-Qwen3-8B in ranking metrics is statistically significant, though not in Recall@100 (0.67 vs. 0.73). Performance improves monotonically over rounds (Recall@100: 0.45 → 0.67 across rounds 1–10), but analysis of candidate-pool recall shows the observer discards substantial relevant evidence: by round 10, candidate-pool recall reaches 0.77 versus 0.67 for observer-kept output, meaning observer filtering caps final recall.

Alternative interventions are largely ineffective. Parameter tuning adds at most +0.05 (oracle upper bound, tuned on the full benchmark); PRF changes Recall@100 by at most ±0.02; RRF fusion hurts five of eight dense encoders; and translation helps dense retrieval modestly (+0.04 for Qwen3-8B) but harms BM25-GreekStemmer (−0.11 when compared against the English BM25-spaCy variant).

Cost–quality trade-off

The systems occupy sharply different points in the cost space. Ten-round ReAct-BM25 requires 20 LLM calls and approximately 618k tokens per query (the observer accounts for 84% of tokens, since its prompt includes full article text), roughly 240× more tokens and ~154× higher estimated cost than Reform-Qwen3-8B, which needs two calls and about 2.9k tokens per query. Conversely, BM25-based systems avoid GPU indexing, vector databases, and re-indexing when embedding models change — an operational advantage that ReAct-BM25 inherits at inference-time expense. The paper's practical recommendation is therefore query reformulation followed by dense retrieval as the default, reserving iterative sparse retrieval for settings with small context budgets, mandated sparse infrastructure, or difficult queries.

Limitations

The paper concedes several constraints directly bearing on its conclusions. Gold labels derive solely from official bar-exam solutions and may omit genuinely relevant articles, so reported recall is a lower bound. With only 283 queries and 2.74 relevant articles per query, small differences warrant caution even when statistically significant. All LLM-dependent results come from a single generation path per query, so run-to-run variance cannot be assessed. The BM25 tuning sweep was optimized on the full benchmark and no development subset exists. The claim that reformulation transfers semantic-matching benefits to sparse retrieval rests on indirect evidence. No re-rankers were evaluated, although re-rankers coupled with sparse retrievers could erode ReAct-BM25's top-ranking advantage. Finally, retrieval is evaluated independently of end-to-end legal QA, leaving open whether these retrieval gains translate into more accurate, better-grounded answers.

Conclusion

GreekBarRetrieval establishes the first public retrieval benchmark for Greek statutory law, linking 283 bar-exam queries to gold statutes within a 6,308-article pool. The experimental results show that vanilla dense retrieval far exceeds vanilla BM25 on coverage, but that LLM query reformulation narrows the gap substantially for sparse retrieval while improving dense systems as well, outperforming parameter tuning, PRF, RRF fusion, and translation. The introduced ReAct-BM25 loop delivers the strongest top-rank quality among all tested methods at considerable computational cost. The benchmark and baselines provide a foundation for studying retrieval-augmented Greek legal question answering, particularly the interplay between query formulation quality, retrieval architecture, and downstream answer grounding.