JUÁ -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections

Abstract: Legal information retrieval in Portuguese remains difficult to evaluate systematically because available datasets differ widely in document type, query style, and relevance definition. We present JUÁ, a public benchmark for Brazilian legal retrieval designed to support more reproducible and comparable evaluation across heterogeneous legal collections. More broadly, JUÁ is intended not only as a benchmark, but as a continuous evaluation infrastructure for Brazilian legal IR, combining shared protocols, common ranking metrics, fixed splits when applicable, and a public leaderboard. The benchmark covers jurisprudence retrieval as well as broader legislative, regulatory, and question-driven legal search. We evaluate lexical, dense, and BM25-based reranking pipelines, including a domain-adapted Qwen embedding model fine-tuned on JUÁ-aligned supervision. Results show that the benchmark is sufficiently heterogeneous to distinguish retrieval paradigms and reveal substantial cross-dataset trade-offs. Domain adaptation yields its clearest gains on the supervision-aligned JUÁ-Juris subset, while BM25 remains highly competitive on other collections, especially in settings with strong lexical and institutional phrasing cues. Overall, JUÁ provides a practical evaluation framework for studying legal retrieval across multiple Brazilian legal domains under a common benchmark design.

• The paper introduces JUÁ, a heterogeneous benchmark combining five datasets for comprehensive evaluation of legal retrieval systems across jurisprudence, regulatory, and legislative domains.
• The paper evaluates diverse retrieval strategies—including BM25, dense retrievers, and neural rerankers—highlighting trade-offs between lexical matching and semantic generalization.
• The paper demonstrates that no single retrieval model excels universally, underscoring the need for hybrid and domain-adapted approaches to tackle complex legal IR challenges.

JUÁ: A Heterogeneous Benchmark for Brazilian Legal Information Retrieval

Motivation and Context

Legal Information Retrieval (LIR) in the Brazilian context is challenging due to heterogeneity in document types, query formulations, and relevance criteria in available datasets. The proliferation of statutes, case law, regulations, and institutional materials—all marked by idiosyncratic language and evolving legal standards—makes systematic and reproducible evaluation of retrieval methods non-trivial. Despite the recent introduction of resources such as JurisTCU, Ulysses-RFCorpus, and BR-TaxQA-R, there was no unified public benchmark for direct, protocol-aligned comparison of retrieval models in Brazilian legal corpora prior to the work described in "JUÁ -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections" (2604.06098).

JUÁ addresses this critical gap, introducing a multi-domain, extensible benchmark infrastructure that allows for direct, protocol-based comparison across core retrieval paradigms. Its inclusion of a public leaderboard and leaderboard-driven evaluation infrastructure further standardizes the evaluation process, promoting cumulative and transparent progress in Brazilian LIR.

Benchmark Design and Dataset Composition

JUÁ is constructed as a heterogeneous benchmark spanning multiple retrieval regimes. It combines five datasets:

• JUÁ-Juris: A new jurisprudential dataset derived from curated holdings of the Brazilian Federal Court of Accounts (TCU), constructed with stratified sampling on BM25-based retrieval difficulty and strict query–document alignment.
• JurisTCU: An adapted dataset from TCU’s jurisprudence with expert-verified relevance for realistic and synthetic queries, using only the EXCERTO field to increase task complexity.
• NormasTCU: Regulatory/normative retrieval over TCU’s internal/external acts, with both keyword and semantically rich queries and three-level graded relevance provided by domain experts.
• Ulysses-RFCorpus: Legislative retrieval reflecting real user needs in bill drafting, annotated via explicit relevance feedback from 54 legislative consultants, supporting user-grounded legislative query scenarios.
• BR-TaxQA: FAQ-style personal tax law QA using real questions and reference answers from Brazilian Federal Revenue, supporting both direct and linked answer retrieval with graded relevance.

By consolidating jurisprudence, legislative, regulatory, and question-answer retrieval, the benchmark exposes cross-domain trade-offs and measures generalization and domain adaptation in retrieval models.

Retrieval Models and Adaptation Protocols

Three main classes of retrieval strategies are evaluated:

• BM25: Classical sparse lexical matching, effective where institutional terminology and surface-form cues dominate.
• Dense retrievers: Qwen series (8B, 4B, fine-tuned), OpenAI, and KaLM Gemma3 12B models, evaluated in both zero-shot and fine-tuned settings.
• Rerankers: Post-processing via neural reranking (e.g., fine-tuned Qwen3-Embedding-4B reranker) over top-1000 BM25 candidates.

The most notable adaptation is the fine-tuning of Qwen3-Embedding-4B on a mixture including JUÁ-Juris, Ulysses (including synthetic queries), and SQuaD-pt for semantic regularization. Careful construction of training, validation, and negative sampling protocols is employed to maximize both in-domain and cross-domain generalization.

JUÁ’s protocol uses four principal metrics at cutoff-10 (NDCG@10, MRR@10, MAP@10, Recall@10), tracking both position-aware and coverage-oriented retrieval quality. Complementary metrics provide additional granularity, distinguishing between early-hit and breadth-of-coverage behaviors.

Empirical Findings and Metric Visualizations

No single retrieval paradigm dominates across all JUÁ datasets. BM25 is highly competitive on legislative and statutory retrieval with high lexical overlap, whereas domain-adapted dense retrievers show marked gains, particularly for supervision-aligned jurisprudence (JUÁ-Juris). In question-driven and semantically diverse settings (e.g., BR-TaxQA), dense models outperform sparse baselines by a substantial margin.

Figure 1: Model-level comparison of MRR@10 performance across all JUÁ benchmark datasets, highlighting early-hit ranking quality and the impact of domain adaptation.

Further analysis with MAP@10 and Recall@10 reveals that reranking typically improves top-10 breadth, though not universally: rerankers can be ineffective when BM25’s initial candidate pool is already near-optimal due to strong institutional lexical cues.

Figure 2: Model-level comparison of MAP@10 across JUÁ, emphasizing aggregate top-10 ranking quality and the consistency of fine-tuned dense retrieval.

Figure 3: Model-level comparison of Recall@10 across JUÁ, highlighting overall relevant material coverage within the top 10 retrieved documents.

Empirical discrepancies between datasets persist even within jurisprudential retrieval: fine-tuned dense retrievers excel on JUÁ-Juris (NDCG@10 up to 0.290; MRR@10 up to 0.230), but sparse BM25 remains strongest on JurisTCU. This confirms that “jurisprudence retrieval” is itself non-uniform, varying with query/document alignment and linguistic variability.

Qualitative error analysis reveals three broad regimes:

• Lexical specificity (short, institutionally anchored queries): BM25 dominates.
• Semantic abstraction/paraphrase (complex queries, paraphrastic/weak lexical overlap): Dense models/fine-tuned rerankers dominate.
• Complex/ambiguous queries: Both paradigms fail, reflecting limitations in current modeling.

Contradictory Claims and Implications

A dominant claim is that “strong overall retrieval performance does not imply uniform superiority across legal settings—top leaderboard models are frequently non-uniform and trade off across regimes”. In contrast to general-domain IR, domain-adapted dense retrieval is not universally superior and, in legislative/regulated settings with strong lexical controls, sparse methods remain non-trivial baselines (2604.06098).

Practical and Theoretical Implications

Practically, JUÁ enables protocol-aligned, reproducible comparison of retrieval strategies and paves the way for robust evaluation in LIR system deployment, including downstream tasks such as legal compliance, e-discovery, and retrieval-augmented generation. Its fine-grained granularity allows practitioners to align model selection with real operational regimes.

Theoretically, JUÁ supports investigation into domain adaptation, transfer learning, and the interaction of legal language structure with retrieval model architectures. It exposes the impact of supervision alignment and semantic regularization (e.g., the inclusion of SQuaD-pt) in dense model training.

The findings also illuminate the ongoing role of lexical retrieval infrastructures in specialized domains, where advancements in deep neural models do not automatically translate into dominant real-world performance—reinforcing the need for hybrid retrieval and pipeline optimization strategies.

Further, JUÁ serves as infrastructure for leaderboard-driven, cumulative evaluation, mirroring the role that benchmarks like BEIR and MTEB play in general IR, but specialized for legal Portuguese. This continuous infrastructure will be critical for tracking progress as LLMs and dense retrieval models become more refined.

Limitations and Future Directions

JUÁ is intentionally heterogeneous, and aggregate scores should not be over-interpreted as evidence of universal model strength. The benchmark combines datasets of varying annotation protocols and varying degrees of naturalness, introducing additional variance in measurement.

Train–test alignment effects—particularly for fine-tuned models (e.g., Qwen3-Embedding-4B (FT)) trained on JUÁ-Juris data—mean that adaptation gains should be cautiously interpreted; explicit ablation studies can further clarify the transferability of learned representations.

Benchmark curation choices (difficulty-aware sampling, removal of summary fields) increase discriminativity but may overstate task hardness relative to some real-world deployments.

Key future developments include expanding query diversity, further legal collection integration (across more institutions), and more transparent ablation of adaptation effects versus cross-domain transfer, to better model the spectrum of practical legal queries encountered in Brazilian institutions.

Conclusion

JUÁ establishes a robust, protocol-driven evaluation infrastructure for Brazilian legal information retrieval, enabling comprehensive comparison across retrieval paradigms and legal regimes (2604.06098). Its design foregrounds cross-domain robustness, reveals the persistence of strong lexical baselines, and empirically situates the conditions where dense adaptation and reranking are most beneficial. The benchmark's heterogeneity and leaderboard infrastructure are crucial for advancing both the state-of-the-art in Brazilian LIR and the wider theoretical understanding of retrieval in high-stakes, structurally complex legal domains.