---
title: 'JUDGEBERT: Metric for French Legal Text Simplification'
url: https://www.emergentmind.com/topics/judgebert
type: topic
---

# JUDGEBERT: Metric for French Legal Text Simplification

JUDGEBERT is a supervised learned evaluation metric for French legal text simplification, designed to estimate whether a simplified legal sentence preserves the legal meaning of its source. Introduced together with the FrJUDGE dataset, it treats legal meaning preservation as a distinct target from generic semantic similarity or ordinary simplification quality, with emphasis on whether a simplification retains legally relevant details, exceptions, and scope in French insurance and regulatory text from Quebec and Canada [2508.16870].

## 1. Conceptual scope and task definition

JUDGEBERT was proposed for a narrow but legally consequential task: assessing whether a simplified French legal text preserves the legal meaning of the original. In this formulation, the target property is not generic paraphrase quality, topical similarity, or ordinary simplification adequacy. The underlying concern is that legal simplification can create a “legal gap” even when ordinary-language meaning appears close. The paper’s motivating example is that terms that are near-synonyms in everyday language, such as “automobile” and “vehicle,” are not legally interchangeable in Quebec law.

The intended use case is evaluation of legal automatic text simplification systems. The metric is therefore sentence-pair based: it compares a legal source sentence or short text block with an automatically generated simplification and predicts a legal meaning preservation score. The paper presents this as especially relevant for sensitive legal texts in which omissions, added content, altered scope, or weakened exceptions can materially change legal effect.

The central notion of preservation is operationalized in explicitly legal terms: a simplification should “convey the legal details and exceptions and does not misrepresent the law.” This makes JUDGEBERT domain-specific not merely because it is trained on legal text, but because the target variable itself is legal in nature rather than general semantic overlap [2508.16870].

## 2. FrJUDGE dataset and annotation methodology

JUDGEBERT is trained and evaluated on **FrJUDGE**, described as the first legal meaning judgment dataset in any language. FrJUDGE contains **297 human-annotated French sentence pairs** drawn from insurance forms used by the **Bureau d’assurance du Canada** and the **Autorité des marchés financiers du Québec**. The authors manually examined **312 text blocks**; each block had to be **1 to 5 sentences long**, could not be boilerplate such as titles, and had to be difficult enough to merit simplification, operationalized as a **French Flesch-Kincaid grade level \(\le 50\)**. Of these, **297 instances** were retained for annotation and **15 additional instances** were used only for annotator practice and then discarded.

The simplifications were generated automatically with **GPT-4-turbo** in zero-shot mode, specifically **gpt-4-turbo-2024-04-09**, with max new tokens **100**, temperature **1.0**, top-k/top-p reported as **0.9**, and zero penalties. Each instance then received three annotations from a pool of **five native French-speaking law students** from Université Laval. The annotation scheme includes three layers. First, annotators assign a **simplicity level** with four labels: easier to read, equal to read, more difficult, and no simplification. Second, they assign a **characterization** from **18 classes**, including categories such as definition, exclusions or restrictions, obligations of the insured, insurer’s rights, legal recourse, and effective date and renewal. Third, they assign the core **Legal Meaning Preservation (LMP)** score on a **1–10 Likert scale**.

The LMP procedure is structured rather than impressionistic. Annotators first determine clause characterization, then place the pair into a preliminary bracket: **7–10** for “Accurate,” **2–6** for “Seems Imprecise,” and **1** for “Off-Track.” They then inspect four legal error classes—**hallucinations**, **omissions**, **consistency issues**, and **confusions**—subtracting one point for each detected error from the upper end of the bracket, with a floor at 1. This makes the score a legally guided judgment rather than a free-form similarity rating.

Inter-annotator agreement is reported as low for the core LMP signal, reflecting the difficulty of the task. For **Legal Meaning Preservation**, the paper reports **25.96% agreement**, **Krippendorff’s \(\alpha = 0.10\)**, and **accuracy 18.48%**. For final labels, the dataset uses **majority vote** for simplicity and characterization and **average score** for LMP. FrJUDGE also reports corpus statistics: **297 complex sentences** and **297 simplified sentences**, with average sentence length **18.83 tokens** on the complex side and **14.24** on the simplified side, and lexical richness **1.2** versus **1.1** [2508.16870].

## 3. Model design, representation, and training regime

Technically, JUDGEBERT is a **sentence-pair regression model**. Its backbone is **CamemBERT-baseV2**, specifically the Hugging Face model `almanach/camembertv2-base`, described in the paper as a **RoBERTa architecture** with **112 million parameters**, **12 transformer layers**, and hidden size **768**. The specialization is task-specific fine-tuning on legal sentence pairs rather than additional legal-domain pretraining.

The model input is formed by concatenating the original legal text and the simplification with a `[SEP]` token:
$$
x = [\text{text}_1 \; \texttt{[SEP]} \; \text{text}_2].
$$
The paper then describes a regression head on top of the encoder:
$$
h = \mathrm{CamemBERT}(x), \qquad \hat{y} = W h + b,
$$
where \(\hat{y}\) is a scalar prediction intended to match the human LMP score on the **1–10** scale. The exact pooling operation, head dimensions, activation functions, optimizer, and explicit loss name are not specified in the paper.

Training is described concretely. Fine-tuning runs for **at most 100 epochs** with initial learning rate **\(5 \times 10^{-5}\)**, batch size **16**, patience **5 epochs**, and **linear learning rate decay**. Results are reported under a **10-fold approach** with random seeds **[42, …, 51]**, each split using **60% train / 10% validation / 30% test**. The paper studies two training conditions. **JUDGEBERT** is trained on the original **297** FrJUDGE triplets. **JUDGEBERT-DA** adds **594 sanity-check augmented examples**, for a total of **891 triplets**. The augmentation is designed to encode two edge conditions explicitly: identical sentences should score maximally, and unrelated legal sentences should score minimally [2508.16870].

## 4. Evaluation results and sanity-check behavior

Evaluation uses **Pearson correlation** and **RMSE** against human LMP labels, together with two sanity checks built from a separate hold-out set of **297 identical** and **297 unrelated** legal sentence pairs. The unrelated pairs are intentionally difficult: they are drawn from the Quebec Automobile Insurance Act and the Quebec Road Safety Code and constrained to at most **ROUGE-[1,2,L] = 0.25** and **BLEU = 25**, so lexical overlap is nontrivial even though legal meaning is unrelated.

The paper’s main JUDGEBERT results are summarized below.

| Variant | Training data | Pearson / RMSE |
|---|---|---|
| JUDGEBERT | 297 FrJUDGE triplets | \(0.74 \pm 0.02\) / \(1.72 \pm 0.10\) |
| JUDGEBERT-DA | 891 triplets | \(0.97 \pm 0.00\) / \(1.01 \pm 0.07\) |

Without augmentation, JUDGEBERT is already the strongest reported metric against human labels, outperforming baselines such as BERTScore, Coverage, LENS, MeaningBERT, QuestEval, SBERT, and SBERT-Multi on both correlation and RMSE. In that non-DA setting, however, it does **not** yet satisfy the sanity checks. In the augmented setting, **JUDGEBERT-DA** becomes the standout system: it achieves the best reported correlation with human judgments and is the only system reported to satisfy both sanity checks perfectly. The abstract formulates these two conditions as follows: for identical sentences, it always returns effectively **100%**; for unrelated sentences, it returns effectively **0%**.

A second practical result concerns overestimation risk. On the non-DA test set, the percentage of predictions that scored **higher than the human label** is reported as **82.22%** for BERTScore, **82.22%** for MeaningBERT, **76.67%** for SBERT, and **77.78%** for SBERT-Multi. By contrast, both **JUDGEBERT** and **JUDGEBERT-DA** are reported at **0.00% ± 0.00**. The paper presents this as especially important in legal simplification, where overestimating preservation is riskier than underestimating it [2508.16870].

## 5. Terminological positioning and neighboring research

Despite its name, JUDGEBERT is not a judicial outcome predictor, a court-document classifier, or a general-purpose LLM judge. It is closer to a domain-specific evaluation metric for **sentence-pair legal meaning preservation**. This distinction matters because nearby lines of research use similar naming intuitions but solve different problems.

A first comparison point is **JuriBERT**, which is “a new set of BERT models adapted to the French legal domain” and is trained as a masked-language model for French legal text, with downstream evaluation on Court of Cassation pleadings and legal subject classification [2110.01485]. JUDGEBERT is not a French legal-domain PLM of that type; it is a supervised regression metric over legal sentence pairs. A second comparison point is legal judgment prediction, as in the Swiss Federal Supreme Court benchmark built around multilingual BERT variants for approval-versus-dismissal prediction from case facts [2110.00806], or the Indian Supreme Court work comparing BERT, InLegalBERT, hierarchical transformers, and LLMs in a realistic prediction setting [2410.10542]. JUDGEBERT does not predict verdicts or case outcomes.

A third comparison point is the modern “judge model” literature. **AdaJudge** is a reward-model architecture for pairwise preference discrimination with adaptive refinement and adaptive multi-view pooling [2601.08097]. **BERT-as-a-Judge** is an encoder-driven binary correctness evaluator over question-candidate-reference triplets in reference-based generative evaluation [2604.09497]. These systems are judges in the RLHF or benchmark-evaluation sense. JUDGEBERT instead evaluates whether two legal texts preserve the same legal meaning. This suggests that the name can be misleading outside its original context: the model is about legal preservation scoring, not courtroom outcome prediction and not generic LLM evaluation.

## 6. Limitations, release status, and significance

The paper is explicit about several limitations. **FrJUDGE** is small, with only **297 annotated instances** for the original training condition. Its domain is narrow, covering mainly **insurance-related French legal text**. The texts are short and decontextualized, even though they come from authentic legal documents; the authors note that real legal interpretation often depends on whole-document context and underlying facts. Annotation subjectivity is substantial, as reflected in the low LMP agreement statistics. The paper also states that it did **not** test out-of-domain generalization and that JUDGEBERT may have overfit the training splits. A further caveat is comparative: most baselines are English-oriented, whereas JUDGEBERT uses French-specialized embeddings, so the results should not be read as proving that those baselines are irrelevant for English legal meaning preservation.

Within those limits, the paper’s significance is clear. JUDGEBERT operationalizes a legally specific notion of preservation that generic semantic metrics often miss: preservation of **details and exceptions**, avoidance of **misrepresentation of law**, and rejection of spurious similarity between legally unrelated clauses sharing vocabulary. A plausible implication is that it is best used as a domain-specific benchmarking or filtering component for French legal simplification systems rather than as a general legal semantic equivalence oracle.

The work also provides reproducibility resources. The paper gives a public repository for **FrJUDGE/JUDGEBERT** at `https://github.com/GRAAL-Research/JUDGEBERT` and states that the source texts were released under **CC-BY 4.0** with permission. In practice, JUDGEBERT occupies a specific position in legal NLP: not a general legal foundation model, not an outcome predictor, and not a generic judge architecture, but a specialized learned metric for assessing whether simplification preserves legal meaning in French legal text [2508.16870].

Source: https://www.emergentmind.com/topics/judgebert