---
title: 'LEXam: A Legal Reasoning Benchmark and Logic Tool'
url: https://www.emergentmind.com/topics/lexam
type: topic
---

# LEXam: A Legal Reasoning Benchmark and Logic Tool

Searching arXiv for recent papers on LEXam and related uses of the term.
LEXam is principally the name of a legal-reasoning benchmark introduced in 2025 as “Benchmarking Legal Reasoning on 340 Law Exams” [2505.12864]. Built from real law school examinations, it was designed to evaluate whether large language models can perform not only answer selection but also long-form legal reasoning, including issue spotting, rule recall, and rule application. In later legal-AI literature, LEXam became a recurrent evaluation substrate for work on LLM-as-a-Judge methodology, prompt optimization, multilingual legal reasoning, adversarial robustness, and critiques of access-to-justice claims. An unrelated earlier arXiv paper also used the name LEXam for an extensible logic embedding tool for non-classical reasoning, so the term has two distinct technical meanings across research communities [2203.12352].

## 1. Formation and scope of the legal benchmark

The legal benchmark LEXam was assembled from **340 law exams** collected from the **University of Zurich’s law school website**. The source material spans **2016–2023**, initially covers **116 courses**, and was consolidated by legal experts into **78 legal subdomains**. It is bilingual, covering **English** and **German**, and it draws on **Swiss law**, **European law**, and **international law**. The benchmark includes both long-form and objective-style tasks, reflecting the paper’s aim to test legal reasoning under realistic exam conditions rather than through only short-form labels or retrieval tasks [2505.12864].

The dataset is reported as comprising **4,886 total questions**, with **2,841 open-ended questions** and a multiple-choice subset whose size is described inconsistently in the paper: the abstract reports **2,045 MCQs**, while the dataset section reports **1,660 MCQs** after conversion and unification. The open-ended portion is split into **300 development questions** and **2,541 test questions**, corresponding to **10.6%** and **89.4%** respectively. Each item is enriched with metadata including **Area**, **Course**, **Jurisdiction**, **Language**, and **Year**; the MCQ portion also records whether an option such as “None of the statements” exists and how many statements are present [2505.12864].

| Aspect | Reported description |
|---|---|
| Source | 340 law exams from the University of Zurich |
| Temporal span | 2016–2023 |
| Languages | English and German |
| Jurisdictional coverage | Swiss, European, and international law |
| Total questions | 4,886 |
| Open-ended questions | 2,841 |
| MCQ count | 2,045 in abstract; 1,660 in dataset section |
| Open-ended split | 300 dev, 2,541 test |

The benchmark’s stated motivation is that prior legal NLP benchmarks often score only final outputs, are easier than real exams, are usually monolingual or narrow, and do not reflect actual exam-style legal reasoning. LEXam was therefore positioned as a benchmark for long-form reasoning under exam conditions rather than merely for legal-text classification or answer extraction [2505.12864].

## 2. Task structure and evaluation design

LEXam divides legal reasoning into two main task families. The first consists of **open-ended legal questions** accompanied by **reference answers** and **explicit guidance** describing the expected reasoning chain, often in terms of **issue spotting**, **rule recall**, and **rule application**. The second consists of **multiple-choice questions**, intended for direct outcome-based evaluation. This dual design permits evaluation of both process-sensitive reasoning and answer selection [2505.12864].

The MCQ subset is not treated as a trivial exam transcription. Three authors manually parse the original items into a question stem and statements, after which the benchmark generates MCQs with **2 to 5 statements**, **1 correct answer**, and **3 distractors**. The distractors are sampled from logically possible incorrect combinations, which yields a **random guessing** baseline of about **25% accuracy**. To unify formats, true/false questions are also converted into MCQ form. The paper additionally defines an auxiliary robustness set of **385 questions**, each with **5 statements**, for which the number of answer choices varies across **4, 8, 16, and 32** [2505.12864].

For the open-ended portion, LEXam adopts an **LLM-as-a-Judge** protocol rather than lexical-overlap metrics such as BLEU, ROUGE, BERTScore, or AlignScore. The judge is **GPT-4o**, using a specialized prompt written by **two legally trained authors** and iteratively refined in a pilot study. The paper states that the chief calibration issue was how to penalize answers that introduce plausible but incorrect legal information not present in the reference. To validate the judge, **3 human legal experts** with Swiss law backgrounds annotate **50 randomly sampled question-answer pairs** blindly and independently. The reported agreement statistics are **Pearson \(r = 0.70\)**, **quadratic weighted \(\kappa = 0.49\)**, and **MAE = 1.95** [2505.12864].

The paper also applies the **Alternative Annotator Test** in a **leave-one-out design** with \(m = 3\) experts, \(n = 50\) items, and strict setting \(\varepsilon = 0\). It reports: for Legal Annotator 1, **LLM wins 33, human wins 17, raw \(p=0.0110\), BY \(p=0.0303\), \(\rho=0.66\)**; for Legal Annotator 2, **LLM wins 29, human wins 21, raw \(p=0.1310\), BY \(p=0.2402\), \(\rho=0.58\)**; for Legal Annotator 3, **LLM wins 37, human wins 13, raw \(p=0.0002\), BY \(p=0.0010\), \(\rho=0.74\)**; and the **overall winning rate** is **\(\omega = 0.67\)**. With relaxed replacement bonus \(\varepsilon=0.15\), GPT-4o passes all three tests with **\(\omega=1.00\)** [2505.12864].

## 3. Baseline findings and the benchmark’s difficulty profile

LEXam’s main experimental results present open-ended legal reasoning as substantially more difficult than multiple-choice selection. On the open-ended portion, the best reported scores are **Gemini-2.5-Pro: 82.2**, **Claude-3.7-Sonnet: 77.6**, **DeepSeek-R1: 68.4**, **GPT-4.1: 68.2**, **GPT-4o: 66.2**, and **DeepSeek-V3: 60.0**. Among smaller models, **Gemma-3-12B-it** is reported as a standout with **50.9**. On the MCQ portion, the top accuracies are **Claude-3.7-Sonnet: 57.2%**, **Gemini-2.5-Pro: 55.7%**, **GPT-4.1: 54.4%**, **GPT-4o: 53.1%**, and **DeepSeek-R1: 52.4%** [2505.12864].

The benchmark identifies several consistent sources of difficulty. The harder cases are **open-ended questions rather than MCQs**, questions requiring **structured multi-step reasoning**, **German** questions relative to English, **Swiss-law** questions relative to generic or international ones, and **criminal law** and **private law** relative to public or interdisciplinary topics. Within the MCQ regime, performance also drops as the number of statements increases. The robustness subset sharpens this point: when the number of answer choices rises from **4** to **32**, accuracy declines markedly, for example **Gemini-2.5-Pro: 68.6 → 35.6**, **Claude-3.7-Sonnet: 60.9 → 33.0**, **DeepSeek-R1: 57.5 → 24.9**, **GPT-4.1: 58.0 → 26.3**, and **GPT-4o: 53.7 → 21.8** [2505.12864].

These results support the benchmark’s central claim that contemporary models can produce plausible legal text yet still struggle with the structured, multi-step reasoning demanded by long-form legal analysis. A plausible implication is that LEXam’s main value lies less in ranking models by a single scalar and more in exposing where reasoning breaks down: process length, multilingual variation, jurisdictional specificity, and distractor sensitivity all materially affect measured performance.

## 4. LEXam as a methodological testbed

Subsequent work used LEXam to study prompt optimization under **LLM-as-a-Judge** evaluation. One such paper treats LEXam as a free-text legal QA benchmark built from **Swiss university law examination questions**, comprising **2,841 open-ended questions** in **English and German**. It uses the benchmark’s **300-question development set** and **2,541-question test set**, further partitioning the development set into **60 questions** for gradient generation and **240 questions** for prompt selection and validation. The study adapts **ProTeGi** to LEXam’s **0–1 continuous Judge Scores**, using the baseline LEXam task prompt as the reference for identifying bad examples. Across four task models—**Qwen3-32B**, **Qwen3-235B**, **gpt-oss-20B**, and **gpt-oss-120B**—automatic optimization improves over the original human-centered prompt baseline, with reported gains from **\(+2.79\%\)** to **\(+6.74\%\)**. Optimization with the more lenient judge **Qwen3-32B** is reported to be both larger and more consistent than optimization with the stricter **DeepSeek-V3**, and prompts optimized with lenient feedback transfer better to strict judges than the reverse [2604.20726].

A separate multilingual study uses LEXam as one of seven datasets in a broader evaluation pipeline. For LEXam specifically, it evaluates **Gemini 1.5 Flash** and **Gemini 2.5 Flash (Preview 05/20)** on **500 data points per language** for the multiple-choice subset and **400 data points per language** for the open-ended subset, using only **English and German**. Its reported baseline results are: for **Gemini 1.5 Flash**, **LEXam MC English 0.48**, **LEXam MC German 0.40**, **LEXam Open English 3.71**, **LEXam Open German 2.77**; for **Gemini 2.5 Flash**, **LEXam MC English 0.74**, **LEXam MC German 0.51**, **LEXam Open English 4.14**, **LEXam Open German 3.62**. The same paper applies a **word substitution attack** and, for Gemini 2.5 Flash, a **character insertion attack**. Under character insertion, the paper reports that English multiple-choice performance remains unchanged, open-ended English drops by **0.12**, German multiple-choice drops by **0.04**, and open-ended German drops by **0.43** [2509.22472].

Taken together, these studies establish LEXam as more than a static leaderboard benchmark. It functions as an experimental platform for testing evaluator disposition, prompt sensitivity, multilingual generalization, and adversarial degradation.

## 5. Critiques of what LEXam measures

A major critique in the 2026 legal-AI literature is that LEXam is a **“lawyer-mediated”** benchmark. On this account, the benchmark measures legal reasoning only after the input has already been cleaned up by legal experts: facts are ordered, irrelevant details removed, and procedural posture clarified. The critique is that this setup measures an **upper bound** on model performance, whereas access to justice for self-represented litigants depends on a **lower bound** defined by noisy, incomplete, and narratively disordered inputs. To illustrate the gap, one paper samples **100 English multiple-choice questions** from LEXam, each with **four answer choices**, keeps the original LEXam task instructions untouched, and evaluates **GPT-4.1-mini**, **GPT-4.1-nano**, and **GPT-4o-mini** using raw accuracy. It then applies three typo perturbations—**one-character deletions**, **one-character swaps**, and **keyboard-adjacent swaps**—at frequencies of **every 2 words**, **every 3 words**, and **every 4 words**, as well as two forms of context dilution: a **“padding sandwich”** with **10** or **30** irrelevant sentences per filler block, and an **interleaved** condition inserting two filler sentences between each question sentence and each answer choice. The paper reports that all models show general degradation, rankings can shift between clean and distorted inputs, and degradation is not always monotonic as typo density increases [2606.23716].

The same critique generalizes beyond robustness to the meaning of legal evaluation. The paper argues that LEXam measures “the model’s capacity to perform legal reasoning on inputs pre-processed with lawyering,” making it meaningful for clear legal questions but not for whether a model can help a pro se litigant formulate a legally usable claim from an unstructured account of events. It therefore characterizes LEXam as a good proxy for lawyer-assisted deployment but not for pro se use. The broader recommendation is not to abandon legal benchmarks, but to redesign them so they directly measure robustness to **noisy narratives**, **buried salient facts**, **omissions**, **surface-level errors such as typos**, and behaviors related to **long-context attention**, **abstention**, and **hallucination** under missing or unclear facts [2606.23716].

A related but distinct critique arises in work on doctrinal legal reasoning under the EU AI Act. That paper describes LEXam as one of the closest current efforts to doctrinal evaluation because it includes **multiple-choice** and **open-ended** questions and uses **LLM-as-a-Judge** scoring for free-text answers. Even so, it argues that LEXam remains insufficient for **EU doctrinal legal reasoning** because law-school exams are **pedagogical and course-specific**, do not properly test **coherence** across a legal system, suppress **contestability** through dependence on reference answers, and are primarily **Swiss-law based**. In that framing, LEXam is a useful step toward doctrinal evaluation, but not the benchmark needed to operationalize “appropriate accuracy” for high-risk judicial AI under the EU AI Act [2606.18158].

## 6. Homonymous use in automated theorem proving

An unrelated 2022 paper uses **LEXam** as the name of an **extensible logic embedding tool** for **lightweight non-classical reasoning** [2203.12352]. In that work, LEXam is both a **library** of shallow embeddings for non-classical logics and a **stand-alone executable** that parses an input problem, reads its logic specification, selects the corresponding embedding, and outputs a **classical higher-order logic** problem in **TPTP THF** syntax. The purpose is to avoid building a dedicated prover for each non-classical logic and instead reuse existing higher-order automated theorem provers through semantic embedding into HOL [2203.12352].

This theorem-proving LEXam supports several logic families, including **many quantified normal multi-modal logics**, **various hybrid logics**, **public announcement logic (PAL)**, **Carmo and Jones’ dyadic deontic logic**, and **Åqvist’s dyadic deontic logic E**. Its input layer uses the TPTP-based non-classical extensions **TFN** and **THN**, with explicit logic specifications such as `logic` annotations, and its architecture separates a **library module** from an **application module** so that new embeddings can be added without changing the external front-end. The paper describes integration into **Leo-III**, where the prover invokes the LEXam embedding API and then performs proof search on the translated THF problem [2203.12352].

The tool is presented as useful where specialized ATP systems are unavailable, particularly for rapid logic prototyping, experimentation, educational settings, and low-threshold student exercises. At the same time, the paper notes that shallow embeddings are not necessarily optimal for every logic, that benchmark and competitor availability is limited for many supported logics, and that the supported range remains “quite limited” relative to the full landscape of non-classical logics. This homonymous use is conceptually and technically separate from the legal benchmark, but it is part of the arXiv record attached to the name LEXam [2203.12352].

Source: https://www.emergentmind.com/topics/lexam