AI Achieves a Perfect LSAT Score

Abstract: This paper reports the first documented instance of a LLM achieving a perfect score on an officially disclosed Law School Admission Test (LSAT). Controlled experiments on eight reasoning models show that varying the prompt, shuffling answer choices, and sampling multiple responses have no meaningful effect as drivers of performance. Ablating the thinking phase that models generate before answering, however, lowers frontier accuracy by up to 8 percentage points, predominantly in logical reasoning. Distilled models produce full thinking traces in the same format yet plateau far below frontier performance. A pilot process reward model fine-tuned via QLoRA on official LSAT explanations narrows this gap through Best-of-5 selection, with gains again predominantly in logical reasoning. The gatekeeper of elite legal education since 1948, the LSAT has not merely been passed but answered without a single error by models that reason. The upper bound of the cognitive capacities it has tested is no longer exclusive to human cognition.

• The paper demonstrates that frontier AI models achieve 100% accuracy on the LSAT by using internal multi-step reasoning traces.
• Disabling the thinking phase leads to a 3–8 percentage point drop in accuracy, underscoring the role of the chain-of-thought process.
• Process reward modeling significantly boosts performance by selecting the best reasoning traces, outperforming majority voting in error reduction.

AI Models Attain Perfect LSAT Performance: Empirical Analysis and Implications

LSAT as a Benchmark for Logical Reasoning in AI

The Law School Admission Test (LSAT) is recognized as a rigorous benchmark for evaluating analytic reasoning in linguistic contexts. Unlike other standardized exams that incorporate domain knowledge, the LSAT focuses principally on abstract logical reasoning, inference, and argument evaluation, rendering it an ideal target for quantifying the reasoning capabilities of LLMs.

Benchmarking efforts illustrate rapid advancement. GPT-3.5 scored 149 and GPT-4 scored 163—below the median observed in elite law school cohorts. The paper documents frontier LLMs achieving, and in at least one case exceeding, the perfect LSAT score of 180. Kimi K2 Thinking and DeepSeek-R1 achieved 100% accuracy on the Official Test (April 2025 disclosed LSAT), marking the first such instance for a LLM (2604.10034).

Modern Model Architectures and Thinking Traces

Frontier models are distinguished from earlier generations by the emergence of explicit reasoning traces at inference time. The transition from prompt-based chain-of-thought to training-based and finally inference-time internal deliberation is central to this leap [wei2022chain, openai2024reasoning, deepseek2025r1]. Models internally generate multi-step thinking phases that are sometimes observable via API (full trace or summarized), and ablation studies confirm these traces materially improve logical reasoning accuracy.

Of note, frontier open/closed API models (GPT-5, Claude Opus 4, Gemini 2.5 Pro, DeepSeek-R1, Kimi K2, QwQ-32B) differ in trace visibility, persistence across turns, and modularity between thinking and response generation. Analyses reveal that all top-performing models generate internal reasoning traces regardless of exposure to users.

Methodological Rigor: Prompt Sensitivity, Position Bias, and Self-Consistency

Evaluation across multiple experimental conditions, including prompt minimalism, structured prompting, and constrained prompting, reveals prompt sensitivity is negligible at the frontier. Accuracy remains invariant (97–100%) regardless of how the model is instructed. This finding contradicts earlier literature which documents high prompt sensitivity in non-reasoning LLMs [zhao2021calibrate, lu2022fantastically].

Position bias, a documented artifact in multiple-choice evaluation [pezeshkpour2023large], is not present in reasoning models at the frontier. Accuracy is unchanged between original and shuffled answer ordering, and answer selection distributions do not differ significantly from uniform.

Self-consistency sampling, previously a technique for boosting performance via majority voting among sampled traces [wang2023selfconsistency], yields negligible gains for frontier models. Frontier errors are highly deterministic; majority voting fails to recover errors, as incorrect answers are reproduced across samples.

Ablation Studies: Importance of the Thinking Phase

Disabling the reasoning phase in frontier models reduces LSAT accuracy by 3–8 percentage points—an effect concentrated in logical reasoning, not reading comprehension. This establishes a direct causal role for the chain-of-thought process in enabling elite logical reasoning performance, a result with nontrivial theoretical implications [feng2024towards].

Small distilled models, trained to mimic frontier traces, produce correctly formatted thinking but plateau well below frontier accuracy. This demonstrates the distinction between reasoning form and reasoning rigor; merely producing traces does not guarantee effective logical inference.

Process Reward Modeling for Reasoning Trace Selection

A process reward model (PRM) was fine-tuned via QLoRA on official LSAT explanations to score reasoning traces for rigor at the small scale. Inference-time selection of traces scored Best-of-5 by the PRM resulted in significant accuracy improvement (+11.7 percentage points relative to pass@1, +7.8 percentage points over self-consistency), with gains concentrated in logical reasoning. This demonstrates that reasoning rigor is externally accessible and amenable to optimization [uesato2022solving, lightman2023lets].

Notably, PRM selection outperformed majority voting in settings where errors are correlated, further validating the value of reward modeling over frequency-based sampling in multi-turn reasoning contexts.

Strong Numerical Findings and Contradictory Claims

• Frontier models achieved up to 100% accuracy on the Official LSAT—surpassing the human median.
• Prompt phrasing, answer ordering, and majority voting have no meaningful effect on frontier accuracy. This contradicts previously established views about sensitivity and sampling efficacy in LLMs.
• Disabling the thinking phase reduces accuracy by up to 8 percentage points, predominantly in logical reasoning.
• Process reward modeling substantially boosts small-model accuracy, confirming that reasoning rigor—not mere trace generation—determines performance.

Practical and Theoretical Implications

Practically, LSAT performance by AI eliminates the prospect of the exam as a unique filter for human analytic ability. Law school admissions and legal education must now reckon with the reality that reasoning models can emulate—and possibly surpass—the logic performance markers historically used for human gating.

Theoretically, the findings challenge assumptions about the upper bound of cognitive capacities tested by standardized reasoning benchmarks. Frontier models can now solve problems previously considered unreachable for bounded-depth transformers unless chain-of-thought reasoning is incorporated [feng2024towards]. Moreover, the paper’s results motivate research in automated verification of reasoning rigor, improved process supervision at scale, and fine-grained trace selection for both frontier and small models.

Future developments will likely focus on process reward modeling for frontier models, scaling candidate pools for Best-of-N selection, and extending reasoning supervision into fine-tuning protocols. As model errors at the frontier become rarer and more deterministic, external verification and trace selection may represent the primary avenue for further accuracy improvement.

Conclusion

A reasoning model has attained perfect performance on the LSAT, a test designed as the benchmark for human logical reasoning aptitude. The ceiling for logical inference, as operationalized in the legal profession, is now accessible to AI. Forward paths include deeper exploration of process-based reward modeling, scaling reasoning trace verification, and reevaluating the role of standardized reasoning benchmarks in the era of machine reasoners (2604.10034).