JUDGEBERT: Metric for French Legal Text Simplification
- JUDGEBERT is a supervised evaluation metric that determines if a simplified French legal text retains its legal meaning.
- It leverages the FrJUDGE dataset with detailed human annotations to capture legal nuances like exceptions and scope.
- The model uses a CamemBERT-based regression approach fine-tuned on legal sentence pairs, achieving high accuracy and robust error detection.
JUDGEBERT is a supervised learned evaluation metric for French legal text simplification, designed to estimate whether a simplified legal sentence preserves the legal meaning of its source. Introduced together with the FrJUDGE dataset, it treats legal meaning preservation as a distinct target from generic semantic similarity or ordinary simplification quality, with emphasis on whether a simplification retains legally relevant details, exceptions, and scope in French insurance and regulatory text from Quebec and Canada (Beauchemin et al., 23 Aug 2025).
1. Conceptual scope and task definition
JUDGEBERT was proposed for a narrow but legally consequential task: assessing whether a simplified French legal text preserves the legal meaning of the original. In this formulation, the target property is not generic paraphrase quality, topical similarity, or ordinary simplification adequacy. The underlying concern is that legal simplification can create a “legal gap” even when ordinary-language meaning appears close. The paper’s motivating example is that terms that are near-synonyms in everyday language, such as “automobile” and “vehicle,” are not legally interchangeable in Quebec law.
The intended use case is evaluation of legal automatic text simplification systems. The metric is therefore sentence-pair based: it compares a legal source sentence or short text block with an automatically generated simplification and predicts a legal meaning preservation score. The paper presents this as especially relevant for sensitive legal texts in which omissions, added content, altered scope, or weakened exceptions can materially change legal effect.
The central notion of preservation is operationalized in explicitly legal terms: a simplification should “convey the legal details and exceptions and does not misrepresent the law.” This makes JUDGEBERT domain-specific not merely because it is trained on legal text, but because the target variable itself is legal in nature rather than general semantic overlap (Beauchemin et al., 23 Aug 2025).
2. FrJUDGE dataset and annotation methodology
JUDGEBERT is trained and evaluated on FrJUDGE, described as the first legal meaning judgment dataset in any language. FrJUDGE contains 297 human-annotated French sentence pairs drawn from insurance forms used by the Bureau d’assurance du Canada and the Autorité des marchés financiers du Québec. The authors manually examined 312 text blocks; each block had to be 1 to 5 sentences long, could not be boilerplate such as titles, and had to be difficult enough to merit simplification, operationalized as a French Flesch-Kincaid grade level . Of these, 297 instances were retained for annotation and 15 additional instances were used only for annotator practice and then discarded.
The simplifications were generated automatically with GPT-4-turbo in zero-shot mode, specifically gpt-4-turbo-2024-04-09, with max new tokens 100, temperature 1.0, top-k/top-p reported as 0.9, and zero penalties. Each instance then received three annotations from a pool of five native French-speaking law students from Université Laval. The annotation scheme includes three layers. First, annotators assign a simplicity level with four labels: easier to read, equal to read, more difficult, and no simplification. Second, they assign a characterization from 18 classes, including categories such as definition, exclusions or restrictions, obligations of the insured, insurer’s rights, legal recourse, and effective date and renewal. Third, they assign the core Legal Meaning Preservation (LMP) score on a 1–10 Likert scale.
The LMP procedure is structured rather than impressionistic. Annotators first determine clause characterization, then place the pair into a preliminary bracket: 7–10 for “Accurate,” 2–6 for “Seems Imprecise,” and 1 for “Off-Track.” They then inspect four legal error classes—hallucinations, omissions, consistency issues, and confusions—subtracting one point for each detected error from the upper end of the bracket, with a floor at 1. This makes the score a legally guided judgment rather than a free-form similarity rating.
Inter-annotator agreement is reported as low for the core LMP signal, reflecting the difficulty of the task. For Legal Meaning Preservation, the paper reports 25.96% agreement, Krippendorff’s , and accuracy 18.48%. For final labels, the dataset uses majority vote for simplicity and characterization and average score for LMP. FrJUDGE also reports corpus statistics: 297 complex sentences and 297 simplified sentences, with average sentence length 18.83 tokens on the complex side and 14.24 on the simplified side, and lexical richness 1.2 versus 1.1 (Beauchemin et al., 23 Aug 2025).
3. Model design, representation, and training regime
Technically, JUDGEBERT is a sentence-pair regression model. Its backbone is CamemBERT-baseV2, specifically the Hugging Face model almanach/camembertv2-base, described in the paper as a RoBERTa architecture with 112 million parameters, 12 transformer layers, and hidden size 768. The specialization is task-specific fine-tuning on legal sentence pairs rather than additional legal-domain pretraining.
The model input is formed by concatenating the original legal text and the simplification with a [SEP] token:
The paper then describes a regression head on top of the encoder:
where is a scalar prediction intended to match the human LMP score on the 1–10 scale. The exact pooling operation, head dimensions, activation functions, optimizer, and explicit loss name are not specified in the paper.
Training is described concretely. Fine-tuning runs for at most 100 epochs with initial learning rate , batch size 16, patience 5 epochs, and linear learning rate decay. Results are reported under a 10-fold approach with random seeds [42, …, 51], each split using 60% train / 10% validation / 30% test. The paper studies two training conditions. JUDGEBERT is trained on the original 297 FrJUDGE triplets. JUDGEBERT-DA adds 594 sanity-check augmented examples, for a total of 891 triplets. The augmentation is designed to encode two edge conditions explicitly: identical sentences should score maximally, and unrelated legal sentences should score minimally (Beauchemin et al., 23 Aug 2025).
4. Evaluation results and sanity-check behavior
Evaluation uses Pearson correlation and RMSE against human LMP labels, together with two sanity checks built from a separate hold-out set of 297 identical and 297 unrelated legal sentence pairs. The unrelated pairs are intentionally difficult: they are drawn from the Quebec Automobile Insurance Act and the Quebec Road Safety Code and constrained to at most ROUGE-[1,2,L] = 0.25 and BLEU = 25, so lexical overlap is nontrivial even though legal meaning is unrelated.
The paper’s main JUDGEBERT results are summarized below.
| Variant | Training data | Pearson / RMSE |
|---|---|---|
| JUDGEBERT | 297 FrJUDGE triplets | / |
| JUDGEBERT-DA | 891 triplets | / |
Without augmentation, JUDGEBERT is already the strongest reported metric against human labels, outperforming baselines such as BERTScore, Coverage, LENS, MeaningBERT, QuestEval, SBERT, and SBERT-Multi on both correlation and RMSE. In that non-DA setting, however, it does not yet satisfy the sanity checks. In the augmented setting, JUDGEBERT-DA becomes the standout system: it achieves the best reported correlation with human judgments and is the only system reported to satisfy both sanity checks perfectly. The abstract formulates these two conditions as follows: for identical sentences, it always returns effectively 100%; for unrelated sentences, it returns effectively 0%.
A second practical result concerns overestimation risk. On the non-DA test set, the percentage of predictions that scored higher than the human label is reported as 82.22% for BERTScore, 82.22% for MeaningBERT, 76.67% for SBERT, and 77.78% for SBERT-Multi. By contrast, both JUDGEBERT and JUDGEBERT-DA are reported at 0.00% ± 0.00. The paper presents this as especially important in legal simplification, where overestimating preservation is riskier than underestimating it (Beauchemin et al., 23 Aug 2025).
5. Terminological positioning and neighboring research
Despite its name, JUDGEBERT is not a judicial outcome predictor, a court-document classifier, or a general-purpose LLM judge. It is closer to a domain-specific evaluation metric for sentence-pair legal meaning preservation. This distinction matters because nearby lines of research use similar naming intuitions but solve different problems.
A first comparison point is JuriBERT, which is “a new set of BERT models adapted to the French legal domain” and is trained as a masked-LLM for French legal text, with downstream evaluation on Court of Cassation pleadings and legal subject classification (Douka et al., 2021). JUDGEBERT is not a French legal-domain PLM of that type; it is a supervised regression metric over legal sentence pairs. A second comparison point is legal judgment prediction, as in the Swiss Federal Supreme Court benchmark built around multilingual BERT variants for approval-versus-dismissal prediction from case facts (Niklaus et al., 2021), or the Indian Supreme Court work comparing BERT, InLegalBERT, hierarchical transformers, and LLMs in a realistic prediction setting (Nigam et al., 2024). JUDGEBERT does not predict verdicts or case outcomes.
A third comparison point is the modern “judge model” literature. AdaJudge is a reward-model architecture for pairwise preference discrimination with adaptive refinement and adaptive multi-view pooling (Miao et al., 13 Jan 2026). BERT-as-a-Judge is an encoder-driven binary correctness evaluator over question-candidate-reference triplets in reference-based generative evaluation (Gisserot-Boukhlef et al., 10 Apr 2026). These systems are judges in the RLHF or benchmark-evaluation sense. JUDGEBERT instead evaluates whether two legal texts preserve the same legal meaning. This suggests that the name can be misleading outside its original context: the model is about legal preservation scoring, not courtroom outcome prediction and not generic LLM evaluation.
6. Limitations, release status, and significance
The paper is explicit about several limitations. FrJUDGE is small, with only 297 annotated instances for the original training condition. Its domain is narrow, covering mainly insurance-related French legal text. The texts are short and decontextualized, even though they come from authentic legal documents; the authors note that real legal interpretation often depends on whole-document context and underlying facts. Annotation subjectivity is substantial, as reflected in the low LMP agreement statistics. The paper also states that it did not test out-of-domain generalization and that JUDGEBERT may have overfit the training splits. A further caveat is comparative: most baselines are English-oriented, whereas JUDGEBERT uses French-specialized embeddings, so the results should not be read as proving that those baselines are irrelevant for English legal meaning preservation.
Within those limits, the paper’s significance is clear. JUDGEBERT operationalizes a legally specific notion of preservation that generic semantic metrics often miss: preservation of details and exceptions, avoidance of misrepresentation of law, and rejection of spurious similarity between legally unrelated clauses sharing vocabulary. A plausible implication is that it is best used as a domain-specific benchmarking or filtering component for French legal simplification systems rather than as a general legal semantic equivalence oracle.
The work also provides reproducibility resources. The paper gives a public repository for FrJUDGE/JUDGEBERT at https://github.com/GRAAL-Research/JUDGEBERT and states that the source texts were released under CC-BY 4.0 with permission. In practice, JUDGEBERT occupies a specific position in legal NLP: not a general legal foundation model, not an outcome predictor, and not a generic judge architecture, but a specialized learned metric for assessing whether simplification preserves legal meaning in French legal text (Beauchemin et al., 23 Aug 2025).