Reasoning Models Ace the CFA Exams (2512.08270v1)

Abstract: Previous research has reported that LLMs demonstrate poor performance on the Chartered Financial Analyst (CFA) exams. However, recent reasoning models have achieved strong results on graduate-level academic and professional examinations across various disciplines. In this paper, we evaluate state-of-the-art reasoning models on a set of mock CFA exams consisting of 980 questions across three Level I exams, two Level II exams, and three Level III exams. Using the same pass/fail criteria from prior studies, we find that most models clear all three levels. The models that pass, ordered by overall performance, are Gemini 3.0 Pro, Gemini 2.5 Pro, GPT-5, Grok 4, Claude Opus 4.1, and DeepSeek-V3.1. Specifically, Gemini 3.0 Pro achieves a record score of 97.6% on Level I. Performance is also strong on Level II, led by GPT-5 at 94.3%. On Level III, Gemini 2.5 Pro attains the highest score with 86.4% on multiple-choice questions while Gemini 3.0 Pro achieves 92.0% on constructed-response questions.

• The paper demonstrates that recent reasoning models achieve near-perfect accuracy across CFA exam Levels I–III, setting new performance baselines.
• It employs a comprehensive, up-to-date dataset reflecting the 2024–2025 CFA curriculum to ensure robust and relevant benchmarking.
• The study highlights that while LLMs excel in quantitative tasks, challenges persist in ethical reasoning and nuanced judgment.

Evaluation of State-of-the-Art Reasoning Models on the CFA Exams

Introduction

This paper evaluates the proficiency of leading LLMs on the CFA exams—a tiered, high-stakes assessment designed to test both technical knowledge and reasoning in professional finance. Previous work established that even advanced models such as GPT-4 struggled with the CFA’s multi-level structure, particularly in the more complex, synthesis-oriented higher tiers. In contrast to earlier studies, this paper deploys the most recent generation of reasoning models, benchmarks them across Levels I–III, and analyzes their strengths and persistent weaknesses, using a new dataset built to reflect the latest official exam standards.

Dataset and Benchmark Construction

The authors construct a comprehensive dataset using official and leading third-party CFA practice materials, reflecting the 2024–2025 curriculum updates. This ensures that:

• Level I contains 540 traditional MCQs focused on foundational knowledge.
• Level II consists of 176 vignette-based MCQs targeting application and analysis.
• Level III offers 264 items, split between MCQs and constructed-response questions (CRQs), structured as per the specialization pathways introduced in the 2025 curriculum.

This design addresses limitations in prior work due to curriculum drift and risk of benchmark contamination by using up-to-date, proprietary sources. It also enables robust comparisons by maintaining topic and format distributions aligned with actual CFA exam requirements.

Figure 1: Sample mock CFA exam questions by level, demonstrating the progression from independent MCQs (Level I) to vignette-based and constructed-response formats (Levels II and III).

Experimental Methodology

The evaluation encompasses both baseline (ChatGPT, GPT-4/o) and contemporary (GPT-5, Gemini 3.0 Pro, Grok 4, Claude Opus 4.1, DeepSeek-V3.1) models. Models are prompted under two conditions:

• Zero-Shot (ZS): Direct answer without reasoning steps.
• Chain-of-Thought (CoT): Step-by-step explicit reasoning prior to the final answer.

Evaluation metrics follow established pass/fail thresholds, ensuring direct comparability to prior studies. Automated scoring with a rubric-aligned LLM is used for CRQs, with transparent articulation of its limitations (such as possible verbosity bias).

Numerical Results and Analysis

Overall Performance

The central empirical findings are as follows:

• All recent reasoning models pass every CFA level under both prompt conditions.
• Gemini 3.0 Pro achieves 97.6% accuracy on Level I (record), and 86.6–92.0% on Level III constructed-response.
• GPT-5 reaches 94.3% accuracy on Level II, outperforming all predecessors.
• Constructed-response accuracy with Gemini 3.0 Pro under CoT attains 92.0%, a substantial margin above both baselines and the prior SOTA.

This marks a dramatic shift from previous studies, where no single model consistently cleared all three levels, especially Level III.

Prompting Effects

Older architectures show sizable improvements from CoT prompting (up to +14pp for GPT-4 on Level I), confirming the importance of structured reasoning aids. However, for newer models, CoT provides inconsistent or even negative effects on multiple-choice tasks, suggesting that these architectures internally instantiate effective reasoning chains even in ZS mode. In contrast, for open-ended CRQs, CoT retains clear value, with strong performance boosts observed.

Topic-Specific Performance and Error Analysis

The strongest models nearly eliminate errors in quantitative domains (Quantitative Methods, Economics), contrary to earlier work identifying these as weakness areas. The residual errors concentrate in Ethical and Professional Standards, with error rates of 17–21% on Level II, affirming that these standards remain comparatively resistant to LLM reasoning.

Item-level analyses reveal persistent difficulties in nuanced ethical application, rigorous evidence assessment, and complex calculation fidelity, even for the highest-scoring models.

(Figure 2)

Figure 2: Example of a Rule Application error, where even advanced models misapply ethical standards in a vignette context.

(Figure 3)

Figure 3: Concept Oversimplification error example illustrating a common simplification in model reasoning on Level III.

(Figure 4)

Figure 4: Calculation Error example, exposing a scenario where a leading model miscalculates base values for carried interest in a private markets vignette.

In addition, model upgrades sometimes expose trade-offs; Gemini 3.0 Pro improves substantially on CRQ synthesis but slightly regresses on Level III MCQ compared to Gemini 2.5 Pro.

Limitations

Constructed-response grading relies on automated LLM evaluation, introducing potential measurement errors, particularly verbosity bias and limited detection of subtle logical flaws. Potential training data contamination remains a non-negligible threat. For Level III, reliance on third-party practice material may not capture subtle stylistic differences of the actual exam. Human expert adjudication is needed for a true upper-bound estimate of LLM proficiency.

Implications and Future Directions

These results indicate that SOTA LLMs now match and, in several topics, exceed the knowledge and procedural competence of professional financial analysts, at least within the codified domains of CFA Levels I and II. The clear and rising capability on CFA Level III CRQs indicates progress toward handling the open-ended, integrative reasoning pivotal for senior financial roles.

From a practical perspective, the deployment of such LLMs for professional training, exam preparation, and potentially even real-world financial analysis is now supported by robust, empirical evidence. Theoretically, this raises questions about the boundaries of current architectures in domains demanding contextual nuance, ethical judgment, and long-tail, non-codifiable expertise.

Further work should prioritize:

• Human-in-the-loop validation for constructed-response scoring;
• Development of contamination-resistant, dynamically generated finance benchmarks;
• Targeted model improvement for ethical and judgment-driven topics;
• Analysis of model behavior on adversarially constructed exam items.

Conclusion

This study establishes new unified baselines for LLM performance on finance professional benchmarks. Recent reasoning models—particularly Gemini 3.0 Pro and GPT-5—demonstrate near-perfect mastery of CFA Level I and II and unprecedented proficiency on Level III synthesized tasks. While persistent weaknesses in ethical judgment and specific calculation tasks remain, these results clarify the frontier of LLM capabilities and set the agenda for future advances in domain-specific, high-stakes professional reasoning.