JustEva: Fairness Toolkit for Legal AI

• The paper introduces JustEva, a toolkit that audits fairness in legal inference by quantifying inconsistency, bias, and imbalanced inaccuracy.
• It employs counterfactual perturbations and fixed-effects regression to isolate the influence of extra-legal factors on LLM outputs.
• Empirical evaluation across multiple models reveals significant prediction instability and systematic treatment disparities in judicial AI.

Searching arXiv for the specified paper to ground the article and confirm bibliographic details. Search query: arXiv (Xue et al., 15 Sep 2025) JustEva A Toolkit to Evaluate LLM Fairness in Legal Knowledge Inference JustEva is a comprehensive, open-source evaluation toolkit for auditing fairness in legal knowledge inference by LLMs. It is designed for judicial contexts in which LLM outputs may be influenced by extra-legal attributes despite the high-stakes and black-box character of AI-assisted legal decision-making. The toolkit implements an end-to-end workflow: generating structured LLM outputs on a legally grounded dataset, analyzing them with three fairness metrics—inconsistency, bias, and imbalanced inaccuracy—applying statistical inference to distinguish systematic effects from chance, and presenting results through informative visualizations (Xue et al., 15 Sep 2025).

1. Problem setting and conceptual scope

JustEva is motivated by the integration of LLMs into legal practice, including use by attorneys and even judges. In this setting, fairness failures may undermine due process and social justice because legal decisions are contextual, ambiguous, and high-stakes. The central concern is that extra-legal attributes such as gender, crime type, location, or procedural minutiae may unduly affect an LLM’s recommendation or prediction, thereby replicating or amplifying existing judicial biases. A structured, transparent, and scalable evaluation framework is therefore required to measure sensitivity to extra-legal factors, systematic effects on outcomes, and groupwise disparities in prediction errors (Xue et al., 15 Sep 2025).

Within this framework, JustEva evaluates fairness along three dimensions. Inconsistency captures output instability under controlled perturbations of extra-legal labels. Bias measures systematic treatment disparity in predicted sentencing outcomes after controlling for case-specific heterogeneity. Imbalanced inaccuracy measures whether absolute prediction errors are systematically larger for some label values than for others. Taken together, these dimensions define a legal-task-specific fairness audit rather than a generic performance benchmark.

A common misconception is that fairness evaluation in legal AI can be reduced to demographic parity or a small number of protected attributes. JustEva is broader: its label system spans demographic and non-demographic factors, substantive and procedural dimensions, and representative examples include defendant gender, crime type, judge characteristics, trial format, location, and judicial procedure. This suggests that the toolkit treats unfairness as a multi-factor phenomenon embedded in judicial process rather than as a single-axis demographic issue.

2. Dataset foundation and extra-legal label system

The toolkit adopts JudiFair, a dataset containing 177,100 case facts derived from real Chinese judicial documents and annotated with 65 extra-legal labels. The labels were curated by legal experts and are grounded in legal theory and empirical research. The paper does not provide the exact, full list or detailed definitions of all 65 labels; only representative examples and category descriptions are given (Xue et al., 15 Sep 2025).

A central design feature is the inclusion of counterfactual case variants. For each label, the dataset includes matched variants differing only in that label while preserving other facts. The paper gives the example of swapping defendant gender. These matched variants enable what the paper describes as causal-like probes of LLM behavior, because they isolate the effect of a single extra-legal factor on the model’s prediction while holding the rest of the case constant.

The same label system serves two roles. First, it drives counterfactual perturbation analysis for inconsistency measurement. Second, it enters fixed-effects regressions as indicator variables, denoted as “Treated” terms, for detecting systematic differences in predicted sentence length and in prediction-error magnitudes. This dual use links descriptive instability, inferential disparity, and error asymmetry within one evaluation framework.

The scope of the label system also distinguishes JustEva from general-domain fairness toolkits. Relative to Aequitas, AI Fairness 360, Fairlearn, and FairPy, JustEva is positioned as a legal-domain toolkit with a broader, legally grounded set of extra-legal labels and with counterfactual variants specifically designed for judicial fairness analysis. Unlike prior legal fairness benchmarks focused mostly on demographic attributes or limited label sets, it supplies both a dataset and a methodological workflow tailored to judicial contexts.

3. Fairness metrics and statistical inference

JustEva implements three complementary fairness metrics. The first is inconsistency, defined as the extent to which an LLM’s prediction changes when only one extra-legal label value is perturbed while all other case facts are held fixed. For each label $l$, the toolkit computes the proportion of changed predictions, denoted $p_l$, weights that proportion by the effective sample size $w_l$, and averages across labels:

$\begin{equation} \label{equatoin:Inconsistency} Inconsistency = \frac{\sum_{l=1}^{N} w_l \cdot p_l}{\sum_{l=1}^{N} w_l} \end{equation}$

Scores are further averaged across models to summarize inconsistency by label and overall (Xue et al., 15 Sep 2025).

The second metric is bias, operationalized through a fixed-effects regression on log-transformed sentence length in months. The dependent variable is $\mathrm{Ln(Sentence)}$, chosen for approximate normality and noted as consistent with empirical legal literature. Document ID fixed effects control for case-specific heterogeneity, and robust standard errors clustered at the document level address intra-case correlation. The regression is:

${\small \begin{equation} \label{eq:regression} Ln(Sentence) = \gamma + \sum_{j=1}^{J} \alpha_j \cdot \text{Treated}_{j} + \sum_{i=1}^{I} \beta_i \cdot \text{ID}_{i} + \varepsilon \end{equation} }$

Statistical significance of the coefficients $\alpha_j$ is interpreted as evidence that changing a label value systematically affects predicted sentencing outcomes.

The third metric is imbalanced inaccuracy, defined as disparity in the magnitude of prediction errors across label values. It is estimated by replacing the dependent variable with the absolute difference between predicted and true sentence length:

${\small \begin{equation} \label{eq:regression2} Abs\_Dif = \gamma + \sum_{j=1}^{J} \alpha_j \cdot \text{Treated}_{j} + \sum_{i=1}^{I} \beta_i \cdot \text{ID}_{i} + \varepsilon \end{equation} }$

Significant $\alpha_j$ coefficients in this model indicate systematic disparities in error magnitude.

To address multiple comparisons across labels and models, the paper applies a Bernoulli test that treats each significant label-level detection as a Bernoulli trial with threshold $\tau$ and computes the probability of observing at least $p_l$0 significant results by chance:

$p_l$1

If this probability is small, the paper deems the observed fairness deficiency systematic rather than incidental. The procedure is applied per model and to pooled counts across all LLMs. The paper does not specify the exact value of $p_l$2 or how it is chosen. It also does not detail confidence intervals, effect sizes, bootstrapping, permutation tests, logistic regression, or nonparametric alternatives. The inference pipeline is centered on high-dimensional fixed-effects linear regression using Stata’s reghdfe via PyStata, together with clustered robust standard errors and the aggregate Bernoulli assessment.

4. End-to-end workflow and software architecture

JustEva supports two experiment types. The first generates structured LLM outputs from the provided dataset or from user data. The second performs statistical analysis and inference on those outputs through regression and related tests. This organization creates a complete evaluation workflow in which model querying, data handling, inferential analysis, and visualization are integrated rather than treated as separate scripts or manual stages (Xue et al., 15 Sep 2025).

The front end is built with Vue 3 and Vite. It provides configuration for either API-based or local LLM access, automatic generation of model predictions on case data, manual upload of data, and an interactive results panel. Visualizations are implemented with Chart.js and include bar charts and pie charts summarizing inconsistency, counts of biased labels, counts of imbalanced inaccurate labels, and aggregate Bernoulli $p_l$3-values. A pop-up results window reports metric-specific statistics and totals such as the number of significant labels.

The back end uses Python, integrates with the OpenRouter API for querying multiple LLMs, and uses PyStata to execute Stata commands for fixed-effects regressions and statistical testing. API configuration includes API URL, model name, temperature, and provider. Results are post-processed into JSON for visualization and Excel tables for review.

The statistical analysis workflow is described as follows: run fixed-effects regressions on $p_l$4 to detect bias and on $p_l$5 for imbalanced inaccuracy; compute inconsistency using label-wise perturbation proportions $p_l$6 and weights $p_l$7; apply clustered robust standard errors and document fixed effects; and use the Bernoulli test to summarize aggregate significance across labels and across models. The paper does not specify prompt templates, raw output schemas, detailed preprocessing steps beyond conversion to JSON and Excel, specific random seeds, or environment details.

From a methodological perspective, this architecture makes fairness auditing reproducible at the level of workflow structure even where some lower-level implementation details are unspecified. A plausible implication is that the toolkit is meant less as a benchmark-only artifact than as a deployable audit environment for comparative model evaluation.

5. Empirical application and observed fairness deficiencies

The paper reports an empirical application to three LLMs: GLM 4, Qwen2.5 72B Instruct, and Gemini Flash 1.5, all evaluated with temperature = 0. The reported results are summarized below (Xue et al., 15 Sep 2025).

Model	Inconsistency	Bias / Imbalanced inaccuracy
GLM 4	0.142	Bias No. = 27, Bias p (5%) = 0.00; Imbalanced Inacc. No. = 19, Inacc. p (5%) = 0.00
Qwen2.5 72B Instruct	0.140	Bias No. = 30, Bias p (5%) = 0.00; Imbalanced Inacc. No. = 29, Inacc. p (5%) = 0.00
Gemini Flash 1.5	0.134	Bias No. = 30, Bias p (5%) = 0.00; Imbalanced Inacc. No. = 35, Inacc. p (5%) = 0.00

The paper interprets these results as showing that, on average, more than 10% of predictions vary per label across counterfactual variants, which it characterizes as substantial instability for judicial applications. For both bias and imbalanced inaccuracy, the Bernoulli $p_l$8-values are reported as effectively 0.00 at the 5% threshold, and the paper notes significance at the 1% level ($p_l$9). These findings are presented as evidence of systematic fairness deficiencies rather than random fluctuations.

The substantive conclusion drawn in the paper is that current LLMs lack fair and trustworthy legal tools. More specifically, the results indicate three concurrent failure modes: sensitivity to extra-legal perturbations, systematic treatment disparity in predicted outcomes, and systematic disparity in error magnitude across extra-legal label values. Because these deficiencies appear across multiple models and many labels, the paper frames them as structural problems in current LLM legal inference rather than isolated model-specific anomalies.

An important interpretive point is that the reported deficiencies arise under a controlled evaluation setting grounded in matched counterfactual variants and fixed-effects regression. This suggests that the toolkit is not merely detecting superficial output variance, but is attempting to separate case-specific legal signal from extra-legal influence with explicit inferential controls.

6. Availability, practical use, and limitations

The toolkit repository is reported at https://github.com/KYSpring/ai_fairness_demo, and the paper also mentions a video demo hosted via a linked Google Drive URL. The paper describes JustEva as deployable, but it does not provide step-by-step installation commands, dependency versions, contribution guidelines, or detailed documentation. The license is also not specified (Xue et al., 15 Sep 2025).

For practical use, the described procedure is to configure API and model parameters in the front end, use Auto-Generate to produce LLM predictions on the JudiFair dataset or upload custom data via Manual Upload, initiate evaluation, and then inspect inconsistency, bias, and imbalanced inaccuracy through the visualization interface. Statistical analysis is executed through PyStata and Stata’s reghdfe, and outputs are saved to JSON and Excel. The paper recommends temperature = 0, evaluating all labels and counterfactual variants, relying on document fixed effects and clustered standard errors, and applying the Bernoulli test for multiple comparisons.

The paper also states several limitations. The full taxonomy and detailed definitions of the 65 labels are not listed. The dataset is built from Chinese judicial documents, and generalizability to other jurisdictions and languages is not established. Prompt templates, seeding, and model-specific configuration details are not provided. The statistical methods are concentrated on fixed-effects linear regression and the Bernoulli aggregate test; alternative inference frameworks such as logistic models, bootstrapping, and permutation tests are not described.

Proposed future directions include expanding the label set, supporting multilingual legal contexts, and incorporating counterfactual explanation techniques to better identify and mitigate the influence of unfair attributes. More broadly, JustEva provides a methodological foundation for legal-domain fairness auditing in which dataset design, perturbation analysis, regression-based inference, and visualization are treated as components of a single evaluative system.