LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks

Abstract: As LLMs are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal question remains challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.

• The paper introduces LexRubric, a benchmark using 12,337 atomic rubric items to assess open-ended Chinese legal tasks.
• It employs a six-dimensional framework—including Legal Accuracy, Completeness, and Ethics—to expose specific LLM failure modes.
• Experimental results reveal top models reach only a 75% normalized score, highlighting challenges in legal reasoning and safety.

LexRubric: Rubric-Guided Diagnostic Evaluation for Open-Ended Chinese Legal Tasks

Motivation and Benchmark Design

LexRubric addresses the limitations of existing legal AI evaluation resources by targeting open-ended and context-sensitive legal tasks in Chinese. Prior benchmarks primarily emphasized narrow tasks (e.g., judgment prediction, contract review) or standardized test-like QA settings, lacking granularity for diagnosing failure modes in LLM responses. LexRubric is constructed on two axes: (1) broad task coverage, sampling both layperson-oriented legal consultations and professional legal examination instances; and (2) fine-grained, multi-dimensional, expert-constructed rubrics that decompose assessment into atomic quality criteria.

Figure 1: The LexRubric framework, exhibiting the construction workflow and evaluation process, with model performance visualization.

Unlike metrics based on reference matching or single aggregate scores, LexRubric centers evaluation on 12,337 atomic rubric items, distributed across 649 benchmark queries and organized under a unified six-dimensional framework: Legal Accuracy, Reasoning/Logic, Completeness, Clarity/Structure, Task Compliance, and Ethics/Safety. Scoring explicitly weights each criterion, enabling both detailed diagnostic analysis and robust aggregate evaluation.

Comparative Analysis: LexRubric vs. Prior Benchmarks

LexRubric is distinguished from representative legal benchmarks by its dual emphasis on realism—via open-ended lay and professional questions—and diagnostic interpretability, through atomic criteria applicable across scenarios.

Figure 2: LexRubric's task and evaluative scope compared to prior benchmarks, highlighting its coverage of both layperson and professional scenarios and its multidimensional, rubric-centric approach.

Benchmarks such as CAIL2018, CUAD, and LexGLUE focus on specific sub-tasks with fixed answers, whereas systems like PLawBench and UCL-Bench integrate practical scenarios but rely on coarser, composite scoring. LexRubric achieves scenario diversity, atomic evaluation granularity, and systematic dimension separation, substantially improving the fidelity and transparency of LLM assessment in legal tasks.

Data Collection and Rubric Construction

The benchmark aggregates 649 questions: 473 real-world legal consultations (with after-the-fact expert curation and anonymization) and 176 judicial examination items co-designed with practitioners, covering 14 legal scenarios. Rubric development is guided by eight consensus annotation standards addressing core legal reasoning, user communication, procedural guidance, and safety. During annotation, each instance is assessed by three legal experts, with stringent atomicity, objectivity, and self-sufficiency enforced for each criterion.

Key design rules in rubric construction include the following: each rubric item must be atomic (targeting a single requirement), objective, binary (satisfied/not), legally valid, self-contained, mutually exclusive within instances, and tightly coupled to task relevance. Both positive and negative criteria are systematically weighted ($[-10, 10]$).

Rubric Validation and Refinement

Evaluation validity is enforced via an expert-in-the-loop rubric construction and consolidation process. Multiple experts independently author and review candidate rubrics per instance, with subsequent AI-assisted discriminative testing using models of varying capability levels and consensus-based quality control. Flagged rubric weaknesses are refined iteratively, including the application of domain-specific LLM agents for rubric repair, with final decisions vetted by legal annotators.

Experimental Results and Diagnostic Model Profiling

Eighteen LLMs (open/closed-source, general/legal-domain models) were evaluated within the LexRubric framework. The principal finding is that even top strong models (e.g., Kimi K2.6, Qwen3.6-Max-Preview) achieve only 75\% normalized score rates, evidencing the nontriviality of open-ended Chinese legal reasoning and safety tasks.

Notably, general-purpose models (Kimi, Qwen, GLM) outperform most legal-specialized LLMs, with the sole exception of LegalOne-8B, which benefits from domain adaptation. This demonstrates that specificity in legal context and annotation language is necessary for strong legal-domain performance—merely scaling legal-domain pretraining is insufficient.

Detailed dimension-wise analysis reveals:

• High scoring on Clarity/Structure and Task Compliance for most models.
• Distinct weaknesses in Legal Accuracy, Completeness, and (for most models) Ethics/Safety. For instance, Claude Sonnet 4.6 leads on the Safety axis, but lags behind on aggregate score.
• Consultation tasks (layperson queries) are systematically more challenging than judicial examination cases, impacting both reasoning and user intent extraction.

A hard subset analysis isolates 117 instances where all models score below 75\%, with the best model achieving only 51.3\%. Rank orderings shift substantially in this subset, suggesting that aggregate leaderboard positions do not fully characterize model robustness or error profiles.

Evaluation Reliability and Automation

LexRubric employs LLM-as-a-judge for automatic scoring, with the judge model (Qwen3.6-27B) demonstrating high agreement with human experts (Kendall $\tau_b$=0.800), and near-perfect rank consistency across alternative judge models. The explicit, atomic rubric design is crucial for enabling reliable automated evaluation and reducing judge-induced bias, even for negative criteria (Figure 3).

Figure 3: Case study showcasing application of the atomic rubric-based evaluation—each criterion triggers a binary assessment, with detailed rationale provided per item.

Data and Scenario Diversity

The benchmark includes broad legal coverage (criminal, labor, financial, IP, administrative law, etc.), supporting evaluation of both layperson queries and professional, highly structured exam prompts. Table-based data statistics and scenario distributions indicate substantial variation in question length, rubric density, and point allocations.

Figure 4: Example LexRubric items—illustrating both legal consultation (left) and judicial examination (right) settings and the diversity of user intents, structures, and rubric requirements.

Theoretical and Practical Implications

LexRubric advances legal LLM benchmarking by:

• Enabling precise, instance-level diagnostic profiling along multiple quality dimensions, facilitating fine-grained error analysis and model improvement strategies.
• Establishing a blueprint for rigorous, domain-informed, and scenario-diverse evaluation in high-stakes settings where language precision and safety are critical.
• Supporting research into scalable automated judgment and minimizing annotation and review costs, potentially generalizable to other professional domains (e.g., medicine, finance).

However, limitations exist regarding jurisdictional scope—the benchmark is confined to Chinese law and language, and result transference to other legal systems or languages is not guaranteed. LexRubric is ultimately an evaluation tool, not a substitute for professional oversight or deployment safety guarantees.

Conclusion

LexRubric introduces a detailed, rubric-guided and diagnostic evaluation paradigm for open-ended Chinese legal tasks, demonstrating nontrivial gaps in current LLM legal reasoning and safety capabilities. Its methodology—spanning atomic, dimension-structured rubrics, expert and model-in-the-loop validation, and comprehensive scenario coverage—substantiates its utility for both practical model selection and theoretical research into legal AI robustness and assessment. The resource and accompanying analysis can inform future benchmarks for high-stakes, open-domain AI applications and drive progress in legal language understanding and safe deployment of LLMs.