LEXam: A Legal Reasoning Benchmark and Logic Tool

• LEXam is a dual-purpose resource that combines a legal reasoning benchmark built from 340 real law exams with a logic embedding tool for non-classical reasoning.
• The benchmark assesses long-form legal reasoning with tasks ranging from open-ended analysis to multiple-choice evaluations of issue spotting, rule recall, and rule application.
• The logic embedding tool aids in translating non-classical logics into classical higher-order logic, supporting rapid prototyping in automated theorem proving.

Searching arXiv for papers on LEXam and related uses of the term. LEXam is principally the name of a legal-reasoning benchmark introduced in 2025 as “Benchmarking Legal Reasoning on 340 Law Exams” (Fan et al., 19 May 2025). Built from real law school examinations, it was designed to evaluate whether LLMs can perform not only answer selection but also long-form legal reasoning, including issue spotting, rule recall, and rule application. In later legal-AI literature, LEXam became a recurrent evaluation substrate for work on LLM-as-a-Judge methodology, prompt optimization, multilingual legal reasoning, adversarial robustness, and critiques of access-to-justice claims. An unrelated earlier arXiv paper also used the name LEXam for an extensible logic embedding tool for non-classical reasoning, so the term has two distinct technical meanings across research communities (Steen, 2022).

1. Formation and scope of the legal benchmark

The legal benchmark LEXam was assembled from 340 law exams collected from the University of Zurich’s law school website. The source material spans 2016–2023, initially covers 116 courses, and was consolidated by legal experts into 78 legal subdomains. It is bilingual, covering English and German, and it draws on Swiss law, European law, and international law. The benchmark includes both long-form and objective-style tasks, reflecting the paper’s aim to test legal reasoning under realistic exam conditions rather than through only short-form labels or retrieval tasks (Fan et al., 19 May 2025).

The dataset is reported as comprising 4,886 total questions, with 2,841 open-ended questions and a multiple-choice subset whose size is described inconsistently in the paper: the abstract reports 2,045 MCQs, while the dataset section reports 1,660 MCQs after conversion and unification. The open-ended portion is split into 300 development questions and 2,541 test questions, corresponding to 10.6% and 89.4% respectively. Each item is enriched with metadata including Area, Course, Jurisdiction, Language, and Year; the MCQ portion also records whether an option such as “None of the statements” exists and how many statements are present (Fan et al., 19 May 2025).

Aspect	Reported description
Source	340 law exams from the University of Zurich
Temporal span	2016–2023
Languages	English and German
Jurisdictional coverage	Swiss, European, and international law
Total questions	4,886
Open-ended questions	2,841
MCQ count	2,045 in abstract; 1,660 in dataset section
Open-ended split	300 dev, 2,541 test

The benchmark’s stated motivation is that prior legal NLP benchmarks often score only final outputs, are easier than real exams, are usually monolingual or narrow, and do not reflect actual exam-style legal reasoning. LEXam was therefore positioned as a benchmark for long-form reasoning under exam conditions rather than merely for legal-text classification or answer extraction (Fan et al., 19 May 2025).

2. Task structure and evaluation design

LEXam divides legal reasoning into two main task families. The first consists of open-ended legal questions accompanied by reference answers and explicit guidance describing the expected reasoning chain, often in terms of issue spotting, rule recall, and rule application. The second consists of multiple-choice questions, intended for direct outcome-based evaluation. This dual design permits evaluation of both process-sensitive reasoning and answer selection (Fan et al., 19 May 2025).

The MCQ subset is not treated as a trivial exam transcription. Three authors manually parse the original items into a question stem and statements, after which the benchmark generates MCQs with 2 to 5 statements, 1 correct answer, and 3 distractors. The distractors are sampled from logically possible incorrect combinations, which yields a random guessing baseline of about 25% accuracy. To unify formats, true/false questions are also converted into MCQ form. The paper additionally defines an auxiliary robustness set of 385 questions, each with 5 statements, for which the number of answer choices varies across 4, 8, 16, and 32 (Fan et al., 19 May 2025).

For the open-ended portion, LEXam adopts an LLM-as-a-Judge protocol rather than lexical-overlap metrics such as BLEU, ROUGE, BERTScore, or AlignScore. The judge is GPT-4o, using a specialized prompt written by two legally trained authors and iteratively refined in a pilot study. The paper states that the chief calibration issue was how to penalize answers that introduce plausible but incorrect legal information not present in the reference. To validate the judge, 3 human legal experts with Swiss law backgrounds annotate 50 randomly sampled question-answer pairs blindly and independently. The reported agreement statistics are Pearson $r = 0.70$, quadratic weighted $\kappa = 0.49$, and MAE = 1.95 (Fan et al., 19 May 2025).

The paper also applies the Alternative Annotator Test in a leave-one-out design with $m = 3$ experts, $n = 50$ items, and strict setting $\varepsilon = 0$. It reports: for Legal Annotator 1, LLM wins 33, human wins 17, raw $p=0.0110$, BY $p=0.0303$, $\rho=0.66$; for Legal Annotator 2, LLM wins 29, human wins 21, raw $p=0.1310$, BY $p=0.2402$, $\kappa = 0.49$0; for Legal Annotator 3, LLM wins 37, human wins 13, raw $\kappa = 0.49$1, BY $\kappa = 0.49$2, $\kappa = 0.49$3; and the overall winning rate is $\kappa = 0.49$4. With relaxed replacement bonus $\kappa = 0.49$5, GPT-4o passes all three tests with $\kappa = 0.49$6 (Fan et al., 19 May 2025).

3. Baseline findings and the benchmark’s difficulty profile

LEXam’s main experimental results present open-ended legal reasoning as substantially more difficult than multiple-choice selection. On the open-ended portion, the best reported scores are Gemini-2.5-Pro: 82.2, Claude-3.7-Sonnet: 77.6, DeepSeek-R1: 68.4, GPT-4.1: 68.2, GPT-4o: 66.2, and DeepSeek-V3: 60.0. Among smaller models, Gemma-3-12B-it is reported as a standout with 50.9. On the MCQ portion, the top accuracies are Claude-3.7-Sonnet: 57.2%, Gemini-2.5-Pro: 55.7%, GPT-4.1: 54.4%, GPT-4o: 53.1%, and DeepSeek-R1: 52.4% (Fan et al., 19 May 2025).

The benchmark identifies several consistent sources of difficulty. The harder cases are open-ended questions rather than MCQs, questions requiring structured multi-step reasoning, German questions relative to English, Swiss-law questions relative to generic or international ones, and criminal law and private law relative to public or interdisciplinary topics. Within the MCQ regime, performance also drops as the number of statements increases. The robustness subset sharpens this point: when the number of answer choices rises from 4 to 32, accuracy declines markedly, for example Gemini-2.5-Pro: 68.6 → 35.6, Claude-3.7-Sonnet: 60.9 → 33.0, DeepSeek-R1: 57.5 → 24.9, GPT-4.1: 58.0 → 26.3, and GPT-4o: 53.7 → 21.8 (Fan et al., 19 May 2025).

These results support the benchmark’s central claim that contemporary models can produce plausible legal text yet still struggle with the structured, multi-step reasoning demanded by long-form legal analysis. A plausible implication is that LEXam’s main value lies less in ranking models by a single scalar and more in exposing where reasoning breaks down: process length, multilingual variation, jurisdictional specificity, and distractor sensitivity all materially affect measured performance.

4. LEXam as a methodological testbed

Subsequent work used LEXam to study prompt optimization under LLM-as-a-Judge evaluation. One such paper treats LEXam as a free-text legal QA benchmark built from Swiss university law examination questions, comprising 2,841 open-ended questions in English and German. It uses the benchmark’s 300-question development set and 2,541-question test set, further partitioning the development set into 60 questions for gradient generation and 240 questions for prompt selection and validation. The study adapts ProTeGi to LEXam’s 0–1 continuous Judge Scores, using the baseline LEXam task prompt as the reference for identifying bad examples. Across four task models—Qwen3-32B, Qwen3-235B, gpt-oss-20B, and gpt-oss-120B—automatic optimization improves over the original human-centered prompt baseline, with reported gains from $\kappa = 0.49$7 to $\kappa = 0.49$8. Optimization with the more lenient judge Qwen3-32B is reported to be both larger and more consistent than optimization with the stricter DeepSeek-V3, and prompts optimized with lenient feedback transfer better to strict judges than the reverse (Elganayni et al., 22 Apr 2026).

A separate multilingual study uses LEXam as one of seven datasets in a broader evaluation pipeline. For LEXam specifically, it evaluates Gemini 1.5 Flash and Gemini 2.5 Flash (Preview 05/20) on 500 data points per language for the multiple-choice subset and 400 data points per language for the open-ended subset, using only English and German. Its reported baseline results are: for Gemini 1.5 Flash, LEXam MC English 0.48, LEXam MC German 0.40, LEXam Open English 3.71, LEXam Open German 2.77; for Gemini 2.5 Flash, LEXam MC English 0.74, LEXam MC German 0.51, LEXam Open English 4.14, LEXam Open German 3.62. The same paper applies a word substitution attack and, for Gemini 2.5 Flash, a character insertion attack. Under character insertion, the paper reports that English multiple-choice performance remains unchanged, open-ended English drops by 0.12, German multiple-choice drops by 0.04, and open-ended German drops by 0.43 (Ioannou et al., 26 Sep 2025).

Taken together, these studies establish LEXam as more than a static leaderboard benchmark. It functions as an experimental platform for testing evaluator disposition, prompt sensitivity, multilingual generalization, and adversarial degradation.

5. Critiques of what LEXam measures

A major critique in the 2026 legal-AI literature is that LEXam is a “lawyer-mediated” benchmark. On this account, the benchmark measures legal reasoning only after the input has already been cleaned up by legal experts: facts are ordered, irrelevant details removed, and procedural posture clarified. The critique is that this setup measures an upper bound on model performance, whereas access to justice for self-represented litigants depends on a lower bound defined by noisy, incomplete, and narratively disordered inputs. To illustrate the gap, one paper samples 100 English multiple-choice questions from LEXam, each with four answer choices, keeps the original LEXam task instructions untouched, and evaluates GPT-4.1-mini, GPT-4.1-nano, and GPT-4o-mini using raw accuracy. It then applies three typo perturbations—one-character deletions, one-character swaps, and keyboard-adjacent swaps—at frequencies of every 2 words, every 3 words, and every 4 words, as well as two forms of context dilution: a “padding sandwich” with 10 or 30 irrelevant sentences per filler block, and an interleaved condition inserting two filler sentences between each question sentence and each answer choice. The paper reports that all models show general degradation, rankings can shift between clean and distorted inputs, and degradation is not always monotonic as typo density increases (Lou et al., 16 Jun 2026).

The same critique generalizes beyond robustness to the meaning of legal evaluation. The paper argues that LEXam measures “the model’s capacity to perform legal reasoning on inputs pre-processed with lawyering,” making it meaningful for clear legal questions but not for whether a model can help a pro se litigant formulate a legally usable claim from an unstructured account of events. It therefore characterizes LEXam as a good proxy for lawyer-assisted deployment but not for pro se use. The broader recommendation is not to abandon legal benchmarks, but to redesign them so they directly measure robustness to noisy narratives, buried salient facts, omissions, surface-level errors such as typos, and behaviors related to long-context attention, abstention, and hallucination under missing or unclear facts (Lou et al., 16 Jun 2026).

A related but distinct critique arises in work on doctrinal legal reasoning under the EU AI Act. That paper describes LEXam as one of the closest current efforts to doctrinal evaluation because it includes multiple-choice and open-ended questions and uses LLM-as-a-Judge scoring for free-text answers. Even so, it argues that LEXam remains insufficient for EU doctrinal legal reasoning because law-school exams are pedagogical and course-specific, do not properly test coherence across a legal system, suppress contestability through dependence on reference answers, and are primarily Swiss-law based. In that framing, LEXam is a useful step toward doctrinal evaluation, but not the benchmark needed to operationalize “appropriate accuracy” for high-risk judicial AI under the EU AI Act (Finck, 16 Jun 2026).

6. Homonymous use in automated theorem proving

An unrelated 2022 paper uses LEXam as the name of an extensible logic embedding tool for lightweight non-classical reasoning (Steen, 2022). In that work, LEXam is both a library of shallow embeddings for non-classical logics and a stand-alone executable that parses an input problem, reads its logic specification, selects the corresponding embedding, and outputs a classical higher-order logic problem in TPTP THF syntax. The purpose is to avoid building a dedicated prover for each non-classical logic and instead reuse existing higher-order automated theorem provers through semantic embedding into HOL (Steen, 2022).

This theorem-proving LEXam supports several logic families, including many quantified normal multi-modal logics, various hybrid logics, public announcement logic (PAL), Carmo and Jones’ dyadic deontic logic, and Åqvist’s dyadic deontic logic E. Its input layer uses the TPTP-based non-classical extensions TFN and THN, with explicit logic specifications such as logic annotations, and its architecture separates a library module from an application module so that new embeddings can be added without changing the external front-end. The paper describes integration into Leo-III, where the prover invokes the LEXam embedding API and then performs proof search on the translated THF problem (Steen, 2022).

The tool is presented as useful where specialized ATP systems are unavailable, particularly for rapid logic prototyping, experimentation, educational settings, and low-threshold student exercises. At the same time, the paper notes that shallow embeddings are not necessarily optimal for every logic, that benchmark and competitor availability is limited for many supported logics, and that the supported range remains “quite limited” relative to the full landscape of non-classical logics. This homonymous use is conceptually and technically separate from the legal benchmark, but it is part of the arXiv record attached to the name LEXam (Steen, 2022).