Legal Fidelity Evaluation (LF-Eval)

• Legal Fidelity Evaluation is a multi-dimensional metric that assesses legal AI outputs for outcome correctness, legal reasoning, and trustworthiness.
• LF-Eval in KoBLEX uses an LLM-as-a-Judge method, scoring answers on criteria like relevance, legal consistency, and context fidelity to ensure proper statutory application.
• Empirical validation shows LF-Eval strongly correlates with expert judgments (Pearson r=84.90), outperforming traditional overlap-based metrics such as BLEU and ROUGE.

Legal Fidelity Evaluation (LF-Eval) denotes the assessment of whether a legal AI system’s output is faithful to the legal question, the governing legal materials, and the reasoning structure expected in legal practice. In its most specific sense, LF-Eval is an automatic metric introduced for provision-grounded, multi-hop legal question answering in KoBLEX, where it jointly considers the question, the generated answer, and the supporting legal provisions (Lee et al., 1 Sep 2025). In the broader legal-LLM literature, legal fidelity is treated as a multidimensional property that extends beyond surface-level accuracy to include outcome correctness, reasoning reliability, and trustworthiness, including fairness, robustness, and safety (Hu et al., 21 Jan 2026).

1. Conceptual foundations of legal fidelity

The contemporary literature does not reduce legal fidelity to a single scalar notion of correctness. A survey of LLM evaluation in legal applications organizes the problem into three dimensions: Outcome Correctness (Output Accuracy), Reasoning Reliability (Legal Reasoning), and Trustworthiness. The same survey argues that evaluation should cover the full pipeline of Result, Process, and Constraint, because legally correct answers may still be produced through flawed reasoning or under conditions that undermine trustworthiness (Hu et al., 21 Jan 2026).

Task-specific work operationalizes this general idea in different ways. In ContractEval, which studies clause-level legal risk identification in commercial contracts, the term “legal fidelity” is not explicitly defined mathematically, but the paper states that, in context, it refers to high recall, accurate extraction, low “laziness,” and conciseness (Liu et al., 5 Aug 2025). LexEval goes further and states that it is the first to explicitly define and test “legal fidelity,” characterizing it as the faithful application of legal knowledge and reasoning to real tasks while avoiding hallucinations, factual errors, or ethical lapses (Li et al., 2024).

A recurring theme is alignment with legal practice rather than generic NLP task structure. LAiW explicitly criticizes evaluations “defined by the experts of computer science” for lacking consistency with the logic of legal practice, and instead structures evaluation around legal syllogism: major premise, minor premise, and conclusion (Dai et al., 2023). This suggests that, in legal AI, fidelity is increasingly understood as fidelity not only to text or labels but also to professional legal method.

2. LF-Eval in provision-grounded legal question answering

In KoBLEX, LF-Eval is introduced as a purpose-built metric for open-ended and provision-grounded Question Answering (QA). KoBLEX contains 226 scenario-based QA instances and their supporting provisions, created using a hybrid LLM-human expert pipeline, and is designed to evaluate provision-grounded, multi-hop legal reasoning. The paper argues that existing legal benchmarks fail to evaluate this setting adequately and that token-level F1, BLEU, and related overlap metrics do not rigorously test whether an answer correctly applies relevant legal statutes or avoids hallucinations, omissions, and contradictions relative to law (Lee et al., 1 Sep 2025).

LF-Eval is defined as an LLM-as-a-Judge procedure specialized for legal QA. Its core design principle is holistic consideration of three inputs at once: the legal question, the generated answer, and the full set of supporting legal provisions. The evaluation prompt rates answers on a 1–10 scale for legal accuracy and fidelity and instructs the judge model to assess five criteria: Answer Relevance, Legal Consistency, Conclusion Accuracy, Context Fidelity, and Avoid Generic Responses. The prompt also asks for a detailed justification that references the statutes and the logic used in the evaluation (Lee et al., 1 Sep 2025).

This design addresses a specific failure mode of surface-overlap metrics: a response may look lexically similar to a reference answer while still omitting a required statutory condition, misapplying a provision, or substituting generic legal knowledge for the provided context. LF-Eval is intended to penalize exactly these failures (Lee et al., 1 Sep 2025).

3. Scoring procedure, validation, and empirical behavior

LF-Eval computes a score by sampling ten judgments from GPT-4o for each QA instance. In each run, the judge returns a score $s_i \in [1,10]$ together with probabilities $p(s_i)$ obtained using logprobs. The final score is the weighted average

$$\text{LF-Eval} = \frac{1}{10} \sum_{i=1}^{10} s_i \cdot p(s_i).$$

The procedure also retains the justification text for interpretability, so the metric is not merely scalar but accompanied by statute-referencing explanations (Lee et al., 1 Sep 2025).

Human validation in KoBLEX uses human legal experts (2 groups) who rated answers on the same 1–10 scale with criteria aligned to LF-Eval. The reported inter-annotator agreement is high ($r=73.97$). Against these human judgments, LF-Eval attains the strongest reported Pearson correlation among the tested metrics (Lee et al., 1 Sep 2025).

Metric	Pearson $r$ with humans
LF-Eval	84.90
Faithfulness	6.87
Token F1	61.25
BLEU	51.89
ROUGE-L	32.19

Within the same benchmark, LF-Eval also functions as a system-comparison metric. KoBLEX introduces Parametric provision-guided Selection Retrieval (ParSeR), a method that generates parametric provisions and applies a three-stage sequential retrieval process. The reported experiments state that, compared to standard retrieval with GPT-4o, ParSeR achieves +37.91 higher F1 and +30.81 higher LF-Eval, while delivering consistent performance across reasoning depths (Lee et al., 1 Sep 2025).

4. Alternative operationalizations of legal fidelity

LF-Eval in KoBLEX is one instantiation of a broader pattern: legal fidelity is evaluated differently depending on the legal task. ContractEval studies extraction fidelity in commercial contracts using the CUAD test set, which contains 4,128 samples from 102 contracts annotated for 41 risk-relevant clause categories. Models receive the full contract and a clause-category-specific question, and must either extract the exact sentence(s) or output “No related clause.” ContractEval evaluates three dimensions: Correctness of Risk Identification via $F_1$ and $F_2$, Output Effectiveness and Conciseness via Jaccard Similarity, and “Laziness” / Legal Fidelity via the False “No Related Clause” Rate (Liu et al., 5 Aug 2025).

ContractEval’s formulas make explicit how omission-sensitive legal fidelity can differ from standard IR-style scoring:

$$F_1 = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$

$$F_2 = 5 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{4 \cdot \mathrm{Precision} + \mathrm{Recall}}$$

$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$

The emphasis on $p(s_i)$0 reflects the paper’s claim that missing a risk is more serious than including excess text, and the false “No related clause” rate isolates a subset of false negatives that correspond to legally harmful omission (Liu et al., 5 Aug 2025).

Other work defines fidelity around different error types. In automated evaluation of 3-ply case-based legal arguments, faithfulness is defined as the absence of hallucination, where hallucination means generation of factors not present in the input case materials; the same framework also scores factor utilization and abstention when no factual basis exists for argument generation (Zhang et al., 31 May 2025). In patent drafting, Pat-DEVAL decomposes evaluation into Technical Content Fidelity (TCF), Data Precision (DP), Structural Coverage (SC), and Legal-Professional Compliance (LPC), and reports a Pearson correlation of 0.69 overall and 0.73 on LPC against patent expert judgments (Yoo et al., 1 Jan 2026). Across these formulations, “fidelity” consistently includes grounding and omission detection, but the governing unit of evaluation may be clauses, factors, propositions, answers, or long-form statutory compliance.

5. Rubric-based and practitioner-aligned frameworks

A major line of work treats legal fidelity as an explicitly rubric-governed property. LAiW organizes legal capabilities into Basic Information Retrieval (BIR), Legal Foundation Inference (LFI), and Complex Legal Application (CLA) so that evaluation follows the structure of legal syllogism rather than isolated end-task accuracy. Its reported findings state that LLMs may appear strong on complex application tasks while performing poorly on some basic retrieval and intermediate inference tasks, which the paper presents as a barrier to practical trust and acceptance by legal experts (Dai et al., 2023).

DISC-Law-Eval adopts a dual perspective: Objective Evaluation through multiple-choice legal exams and Subjective Evaluation through open-ended legal Q&A rated by GPT-3.5 on Accuracy (ACC), Completeness (CPL), and Clarity (CLR). The framework explicitly encourages answers structured according to legal syllogism—major premise, minor premise, conclusion—so that evaluation targets legal reasoning form as well as factual correctness (Yue et al., 2023). PLawBench extends this logic to realistic workflows, with 850 questions across 13 practical legal scenarios and approximately 12,500 rubric items, using six rubric dimensions: Issue & Fact Identification, Legal Reasoning, Legal Knowledge Application, Procedural & Strategic Awareness, Claim & Outcome Construction, and Professional Norms & Compliance (Shi et al., 23 Jan 2026).

Rubric-based decomposition also appears in narrower generation tasks. LP-Eval evaluates legal proposition generation in three steps: checking Formal Validity through the presence of stance, object, and specification; rating five substantive dimensions—Source Independence, Fact Independence, Conciseness, Generality, and Fidelity; and then assigning an overall quality score (Xu et al., 19 May 2026). LeMAJ similarly decomposes legal answers into Legal Data Points (LDPs) and tags each as <correct>, <incorrect>, <irrelevant>, or <missing>, reporting stronger correlation with human evaluation than common baselines and an 11% increase in inter-annotator agreement for correctness when reviewers used the LDP breakdown (Enguehard et al., 8 Oct 2025). A plausible implication is that legal fidelity evaluation benefits from segmentation into legally meaningful units rather than holistic scoring alone.

6. Limits of surface metrics and emerging directions

A common misconception is that legal fidelity can be adequately captured by overlap-based measures such as ROUGE, BLEU, or token F1. The survey literature explicitly rejects this position, noting that standard metrics focus on surface similarity and may miss subtle but critical legal differences (Hu et al., 21 Jan 2026). KoBLEX reports that LF-Eval correlates more strongly with human judgments than Faithfulness, Token F1, BLEU, or ROUGE-L (Lee et al., 1 Sep 2025). A cross-system study of Chinese and US case judgments reaches a similar conclusion: ROUGE and BLEU are useful but insufficient alone, and human legal evaluation better captures “reasoning fidelity” than lexical similarity (Wang et al., 2024).

The same survey identifies additional unresolved problems: rubric reliance, cross-jurisdictional variance, inadequate coverage of privacy preservation, toxicity, and safety, and the difficulty of automating reliable evaluation for open-ended legal reasoning (Hu et al., 21 Jan 2026). Recent work pushes the field toward more explicit causal, relevance-sensitive, and evidence-path-aware evaluation. A legal-relevance-sensitive evaluation framework distinguishes “should-not-change” perturbations, where outputs should remain stable, from “should-change” perturbations, where outputs should change only when legally material facts are altered. It formalizes these with Invariance and Change Alignment metrics and argues that legal trustworthiness requires calibrated sensitivity to legally material changes (Linze et al., 26 May 2026):

$p(s_i)$1

$p(s_i)$2

An even stronger departure from judge-based scoring appears in GroundEval, a judge-free framework for evaluating stateful agents against grounded, time-bounded, and access-controlled evidence. GroundEval scores not only final answers but also the recorded retrieval trajectory, with three tracks—Silence, Perspective, and Counterfactual—that target absence checking, temporal/permission fidelity, and causal correctness. Its compliance-adjusted score is

$p(s_i)$3

where violations include access, subsystem, and temporal boundary crossings (Flynt, 22 Jun 2026). This suggests a broader shift in legal fidelity evaluation: from answer-only plausibility toward explicit verification of whether the system used the right evidence, under the right permissions, for the right legal reason.