LexRubric: Legal Diagnostic Evaluation Benchmark

• LexRubric is a diagnostic benchmark that evaluates open-ended Chinese legal tasks via expert-written atomic criteria organized across six key dimensions.
• It incorporates 649 instances from legal consultations and judicial examinations, with 12,337 criteria designed to pinpoint model strengths and weaknesses.
• Its criterion-level approach enables precise diagnosis of models’ performance in legal accuracy, reasoning, completeness, clarity, task compliance, and safety.

LexRubric is a rubric-guided diagnostic benchmark for evaluating open-ended Chinese legal tasks. It contains 649 instances drawn from legal consultation and judicial examination, and pairs those instances with 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework. Rather than treating legal evaluation as answer matching or a single holistic score, LexRubric scores responses against many fine-grained legal and communicative requirements, enabling dimension-level diagnosis of where a model succeeds or fails in legal accuracy, reasoning, completeness, structure, task compliance, and safety (Chen et al., 8 Jun 2026).

1. Definition and evaluative rationale

LexRubric addresses a specific evaluative problem: open-ended legal responses are context-sensitive, high-stakes, and only weakly captured by closed-form benchmarks or coarse holistic scoring. Earlier legal benchmarks frequently emphasize multiple choice, classification, short-answer extraction, or broad aggregate scores, whereas legal consultation and professional legal analysis require coordinated handling of facts, applicable rules, legal reasoning, procedure, risk, communication style, and safety. LexRubric therefore treats legal evaluation as a rubric-guided diagnostic task rather than a simple correctness judgment (Chen et al., 8 Jun 2026).

Within the broader rubric literature, rubrics have been defined as structured sets of explicit criteria characterized by explicitness, structuredness, decomposability, and verifiability. LexRubric fits that formulation closely: its criteria are written in natural language, partitioned into atomic units, organized under stable dimensions, and designed for independent criterion-level checking rather than opaque scalar scoring (Chen et al., 7 Jun 2026).

A common misconception is that a legal benchmark of this kind is primarily a leaderboard. LexRubric is instead built to localize error sources. A response can be strong on clarity and task compliance yet weak on legal accuracy or completeness; conversely, it can contain correct legal rules while failing to disclose uncertainty or safety risks. This diagnostic structure is the benchmark’s main methodological contribution (Chen et al., 8 Jun 2026).

2. Corpus construction and task coverage

LexRubric is built from two complementary task sources. The first is legal consultation: the construction process begins from more than 50,000 real user queries about Chinese law, including questions from parties to disputes, lawyers, companies, and public bodies. Queries with valuable legal intent but poor wording are rewritten for clarity while preserving legal substance. After scoring for difficulty, completeness, value, relevance, and answerability, 622 consultation questions are selected; this consultation split includes 40 Chinese law items from OneMillion-Bench. The second source is judicial examination: 250 exam-style questions are written by legal experts in collaboration with a judicial exam platform (Chen et al., 8 Jun 2026).

After annotation, rubric refinement, and filtering, the final benchmark contains 649 instances: 473 consultation questions and 176 exam questions. The 14 legal scenarios are Criminal Offenses and Criminal Procedure (105); Labor, Employment, and Social Security (92); Creditor–Debtor, Guarantees, Enforcement, and Bankruptcy (79); Contract Transactions and Commercial Cooperation (66); Administrative Regulation, Tax, and Government Affairs (62); Corporate Governance, Equity, and Investment/Financing (59); Real Estate, Immovable Property, and Property Rights (50); Civil Procedure, Arbitration, Evidence, and Legal Profession (32); Marriage, Family, and Inheritance (23); Intellectual Property, Data Technology, and Cross-Border Compliance (20); Construction Projects and Tendering (18); Tort Liability, Personality Rights, and Personal Injury (16); Finance, Banking, Insurance, Asset Management, Securities, and Bills (16); and Consumer Rights, Product Quality, and Healthcare (11) (Chen et al., 8 Jun 2026).

The benchmark is deliberately long-context and document-like. Average question length is 818.82 for consultation, 1015.15 for exam, and 872.06 overall. The final rubric inventory contains 10,601 consultation items and 1,736 exam items, for an average of 19.01 rubric items per instance; consultation instances average 22.41 rubric items, while exam instances average 9.86. All user-identifying information in consultation logs is anonymized or obfuscated before release, and unsafe or out-of-scope content is filtered (Chen et al., 8 Jun 2026).

Annotation is costly and expert-intensive. Annotators are legal practitioners or PhD students who passed the National Unified Legal Profession Qualification Examination. Each instance undergoes three independent annotation rounds by different experts. Compensation is reported as \$44.1–\$73.5 per expert per instance; 872 instances were annotated before filtering, at a total cost of approximately \$154,000 (Chen et al., 8 Jun 2026).

3. Rubric architecture, atomicity, and scoring

LexRubric’s rubrics are written under eight cross-task “consensus standards”: emergency legal procedure guidance; information seeking; cross-jurisdiction adaptation; legal document handling; communication customization; responses under uncertainty; response depth and legal reasoning; and ethics and safety. These standards stabilize expectations across tasks, while each instance also receives question-specific criteria tied to its facts, statutes, procedures, or argumentative structure (Chen et al., 8 Jun 2026).

Each criterion must satisfy six design principles: it must be valid, task-relevant, mutually exclusive and relatively complete, atomic, binary, and self-contained. Atomicity is crucial. A criterion such as citing a legal provision, identifying applicable conditions, and applying those conditions to facts is not bundled into one item; each requirement is separated. This makes criterion-level judging feasible and allows diagnostic decomposition of response quality (Chen et al., 8 Jun 2026).

LexRubric formalizes the rubric for instance $x_i$ as

$$\mathcal{R}_i = \{(c_{ij}, p_{ij}, d_{ij})\}_{j=1}^{m_i},$$

where $c_{ij}$ is an atomic criterion, $p_{ij} \in [-10,10]$ is its integer point value, $d_{ij}$ is its assigned dimension, and $m_i$ is the number of rubric items for that instance. Given a model response $y_i$, the raw rubric score is

$$S(x_i,y_i) = \sum_{j=1}^{m_i} p_{ij} \cdot \mathbf{1}\{y_i \models c_{ij}\}.$$

To normalize across instances with different positive-point budgets, LexRubric defines the score rate

$$R_i = \frac{S_i}{\sum_{j=1}^{m_i} \max(p_{ij}, 0)},$$

and similarly defines a dimension score rate

$x_i$0

These normalized rates are reported as percentages and averaged across instances (Chen et al., 8 Jun 2026).

The six dimensions and their item counts are as follows.

Dimension	Rubric items
Legal Accuracy	3,947
Reasoning & Logic	4,151
Completeness	1,505
Clarity & Structure	593
Task Compliance	1,641
Ethics & Safety	500

The distribution shows that LexRubric is weighted toward legal substance and reasoning rather than surface style. That design choice matters empirically: many strong models achieve very high clarity and task compliance, but substantially lower legal accuracy and completeness (Chen et al., 8 Jun 2026).

4. Criterion-level judging and reliability

LexRubric uses LLM-as-a-judge, but not as a holistic grader. The primary judge is Qwen3.6-27B with temperature 0.0. For each criterion, the judge returns an explanation and a Boolean criteria_met; final scores are then computed by benchmark code from the rubric equations rather than assigned impressionistically. Model generations, by contrast, are produced with temperature 0.6 and maximum output length 16k, and each model receives the original legal query without a fixed response prompt template (Chen et al., 8 Jun 2026).

Reliability is tested directly against human experts. On a sample of 50 instances and six evaluated models, three legal experts independently score responses under the rubrics. Comparing model rankings induced by the Qwen3.6-27B judge with rankings induced by the average of human experts yields Kendall tau-b of 0.800, Spearman $x_i$1 of 0.886, and Pairwise Accuracy of 90.00%. This indicates strong judge–expert agreement at the model-ranking level (Chen et al., 8 Jun 2026).

Judge robustness across model choice is also unusually strong. When GLM-5.1, Kimi K2.6, and GPT-5.2 are used as alternative judges over all 18 evaluated models, their rankings agree closely with Qwen3.6-27B: Kendall tau-b is at least 0.974, Spearman is at least 0.996, and Pairwise Accuracy is at least 98.69%. The reported ranking discrepancies occur only among models whose overall scores differ by less than 0.1%, which suggests that LexRubric’s criterion-level procedure is comparatively stable to judge substitution (Chen et al., 8 Jun 2026).

This does not eliminate all concerns about LLM judging. The benchmark still depends on model-based rubric execution, and future systems could make execution more auditable by compiling rubrics into locked executable structures with evidence-anchored scoring and post-hoc calibration, as proposed in RULERS (Hong et al., 13 Jan 2026). LexRubric does not do this, but its criterion-wise Boolean design is compatible with that trajectory.

5. Empirical findings and model capability profiles

LexRubric evaluates 18 recent general and legal-domain LLMs. On all 649 instances, the best overall score rates are Kimi K2.6 at 75.21%, Qwen3.6-Max-Preview at 75.12%, Qwen3-Max at 74.35%, and Kimi K2.5 at 73.40%. GLM-5.1 reaches 69.96%, Claude Sonnet 4.6 reaches 69.18%, and Qwen3.5-397B-A17B reaches 68.03%. LegalOne-8B, a Chinese legal-domain model, reaches 65.14% and outperforms several large general models; by contrast, legal-domain models centered on US or English law perform poorly on Chinese legal tasks, including LawLLM-7B at 10.87%, SaulLM-54B-Instruct at 8.96%, and Saul-7B-Instruct at 5.67% (Chen et al., 8 Jun 2026).

The benchmark exposes sharply different capability profiles. For strong models, Clarity & Structure is often above 90%, and Task Compliance is typically in the low-to-mid 80s. Legal Accuracy and Completeness are lower and more variable, while Reasoning & Logic is intermediate. Ethics & Safety varies especially strongly: Claude Sonnet 4.6 is best on this dimension at 72.74%, while some otherwise strong models are markedly weaker. This makes clear that fluent and well-structured legal prose is not equivalent to safe or legally accurate advice (Chen et al., 8 Jun 2026).

Consultation is harder than exam-style reasoning for strong models. Kimi K2.6 scores 72.84% on consultation and 81.59% on exam; Qwen3-Max scores 71.88% versus 80.99%; Qwen3.6-Max-Preview scores 73.17% versus 80.35%. The plausible interpretation, directly suggested by the benchmark design, is that consultation requires longer-context fact extraction, user-intent inference, uncertainty management, and customized communication, whereas exam questions are more structured and closer to standard legal-reasoning distributions (Chen et al., 8 Jun 2026).

The hard subset further sharpens this picture. On 117 instances where all 18 models score below 75% in the full evaluation, the best overall score is only 51.30% for Qwen3.6-Max-Preview, followed by 48.91% for Kimi K2.6, 46.92% for Qwen3-Max, and 46.24% for Claude Sonnet 4.6. Open-ended legal question therefore remains challenging for current LLMs even when they perform competitively on aggregate averages (Chen et al., 8 Jun 2026).

Two misconceptions are directly rebutted by these results. First, legal-domain specialization alone is not sufficient; jurisdiction and language match matter decisively. Second, high clarity should not be read as high legal competence; models can be excellent at organization while still omitting liabilities, exceptions, procedural steps, or risk disclosures (Chen et al., 8 Jun 2026).

6. Relation to the broader rubric research program and open issues

LexRubric belongs to a wider shift from holistic evaluation toward structured criteria. In survey work on rubrics across evaluation, RL, and alignment, rubrics are described as a unifying framework that decomposes complex judgments into actionable, verifiable signals and increasingly functions at evaluative, training, and intrinsic levels of model development (Chen et al., 7 Jun 2026). LexRubric is a legal-domain instantiation of that shift: it fixes a domain, constructs expert atomic criteria, and uses them as the benchmark’s primary representation of quality (Chen et al., 8 Jun 2026).

At the same time, LexRubric sits within an active debate about how expert judgment should be elicited. JudgmentBench, another legal-domain benchmark, compares rubric scoring and pairwise comparative judgment on 30 real-world legal tasks and reports mean Spearman’s rank correlation of 0.908 for comparative judgment versus 0.150 for rubrics, with comparative judgments requiring less than half the annotation time (Yang et al., 24 May 2026). This suggests that rubric diagnostics and comparative preferences may be complementary rather than interchangeable: rubrics are strong at structured error localization, while pairwise judgments may better capture tacit holistic preferences in professional legal work.

Rubric quality itself is an independent problem. RIFT identifies eight failure modes—Subjective, Non-Atomic, Ungrounded, Misaligned or Rigid, Missing Criteria, Hackable, Low Signal, and Redundant Criteria—with overall 87% pairwise agreement and 0.64 average Cohen’s kappa among human annotators, and automated diagnostics reaching up to 0.86 F1 (Qi et al., 1 Apr 2026). LexRubric’s multi-stage legal-expert construction and discriminativeness filtering can be read as one response to those failure modes, but the benchmark does not yet expose an explicit rubric-diagnostic layer of that kind.

Several adjacent strands point to plausible future extensions. Feedback-to-Rubrics learns reusable natural-language rubrics from inline comments and iteratively refines them through comment-wise mismatches (Yoshida et al., 28 May 2026). Dynamic rubric generation for LLM-as-a-Judge produces dataset-specific and instance-specific rubrics without human annotation and improves them through meta-judge-guided DPO fine-tuning (Wang et al., 28 May 2026). Reliable-to-expressive safety judging treats judgment as rubric-following and uses dynamic rubrics plus curriculum training to stabilize behavior across rubric styles (Lim et al., 8 Jun 2026). These results suggest that future legal benchmarks or evaluators could combine LexRubric’s expert atomic supervision with dynamic rubric induction, rubric-following robustness, or continuously updated rubric banks.

LexRubric’s own limitations are explicit. It is specific to Chinese law; high performance on the benchmark does not guarantee safe unsupervised deployment; evaluation remains single-turn and benchmark-bound rather than fully interactive; and legal work in practice also depends on evolving statutes, organizational risk tolerances, and human review. Nonetheless, by grounding open-ended legal evaluation in 12,337 expert-written atomic criteria and validating criterion-level judging against legal experts, LexRubric establishes a concrete reference point for diagnostic legal evaluation in a domain where coarse scoring is often least adequate (Chen et al., 8 Jun 2026).