KMMLU-Pro: Korean Professional Benchmark
- KMMLU-Pro is a professional-grade benchmark suite designed to evaluate large language models on high-stakes Korean licensure exams across diverse domains.
- The dataset includes 2,822 government-sourced multiple-choice questions, curated through OCR parsing and meticulous manual review to ensure high fidelity and relevance.
- Evaluation protocols use zero-shot, dual-language testing, exposing performance gaps in domains like law and tax/accounting and guiding LLM certification for regulated professions.
KMMLU-Pro is a professional-grade benchmark suite designed for evaluating LLMs on real-world, high-stakes Korean professional licensure examinations. Developed to address the growing necessity for domain-specific and institutionally grounded LLM assessment, KMMLU-Pro focuses on measuring the breadth and depth of knowledge that corresponds to legally mandated proficiencies across critical professional fields in Korea, including law, medicine, accounting, and value estimation. Featuring 2,822 multiple-choice questions directly sourced from government-issued Korean National Professional Licensure (KNPL) exams, the dataset establishes a comprehensive evaluation standard for LLM deployment in regulated sectors (Hong et al., 11 Jul 2025).
1. Motivation and Development Objectives
KMMLU-Pro was introduced to extend the scope of benchmark evaluation for LLMs beyond the technical qualification focus of existing datasets such as KMMLU and KMMLU-Redux. While KMMLU-Redux refined noisy academic-technical data from the Korean National Technical Qualification (KNTQ) exams, it lacks the specialization and regulatory rigor necessary for evaluating professional expertise. KMMLU-Pro, by sourcing questions exclusively from KNPL exams—spanning professions where certification determines legal practice—aims to provide a stringent, practically relevant testbed reflecting the knowledge thresholds enforced in industrial and regulatory environments. This approach is intended to facilitate pre-deployment vetting of LLMs in high-responsibility professions such as legal advisory, clinical decision-making, and financial auditing, where model accuracy and domain compliance are paramount (Hong et al., 11 Jul 2025).
2. Data Acquisition and Curation Workflow
KMMLU-Pro's dataset was assembled through the direct acquisition of official KNPL exam PDFs across 14 professional credentials from authoritative government online repositories. The extraction and processing pipeline involved OCR parsing with GPT-4o, followed by rigorous manual conversion of tables and images into standardized text conforming to LaTeX formatting guidelines. Human annotators reviewed all parsed items for OCR errors, ensured the exclusion of questions with multiple correct answers or untranscribable visual data, and validated ground-truth correctness based on the official answer keys. To ensure contamination-freeness, n-gram based overlap checks were performed against major datasets (FineWeb2, KMMLU), confirming no cross-corpus duplication. Only the latest annual exam cycles were retained to minimize future data disclosures and safeguard benchmark longevity. The final release contains 2,822 consistently formatted, high-fidelity multiple-choice items covering 100% of the selected professional licensure domains (Hong et al., 11 Jul 2025).
3. Domain Taxonomy and Content Scope
KMMLU-Pro comprehensively covers four major professional sectors, detailed into 14 specific licensure domains:
| Major Domain | Licensure Exams (Count) | Questions |
|---|---|---|
| Law | Certified Judicial Scrivener (198), Lawyer—Bar Exam (150), Certified Public Labor Attorney (239), Certified Patent Attorney (109) | 696 |
| Tax & Accounting | Certified Public Accountant—CPA (208), Certified Tax Accountant (238), Certified Customs Broker (159) | 605 |
| Value Estimation | Certified Damage Adjuster—CDA (120), Certified Appraiser (196) | 316 |
| Medicine & Allied Health | Doctor of Korean Medicine (288), Dentist (252), Pharmacist (271), Herb Pharmacist (244), Physician (M.D.) (150) | 1,205 |
All items are aligned to the structure and cognitive demands of official qualifying exams. Representative questions include contextually specific legal, accounting, and biomedical knowledge, as illustrated by items on the Korean Patent Act, deferred tax liability mechanisms for CPAs, and cytochrome P450 metabolism in drug pharmacokinetics for pharmacists.
4. Evaluation Protocols and Metrics
LLMs are evaluated under a zero-shot, multiple-choice regime, with each question posed in both Korean and English; the higher of the two scores per model/domain is reported. The principal evaluation metrics are:
- Accuracy:
- Licensure Pass Count:
Pass criteria for each exam mirror the official KNPL standards: a minimum of 40% per subject and an overall average of 60% (with a 54.22% cutoff for the Bar Exam). The number of licensure fields in which a given model meets these thresholds is tabulated.
Decoding settings are differentiated for reasoning-augmented and generic models, with no additional in-context examples provided.
5. Comparative Performance and Examination Outcomes
Empirical results show a stratified landscape of model proficiency across both open- and closed-weight LLMs:
| Model | Accuracy | Licenses Passed (out of 14) |
|---|---|---|
| DeepSeek R1 (671B, w/ thinking) | 71.33% | 7 |
| Qwen3-235B-A22B (235B, w/ thinking) | 68.22% | 6 |
| Llama-4-Maverick-17B-128E (w/o CoT) | 68.10% | 4 |
| GPT-4.1 | 72.99% | 10 |
| Claude 3.7 Sonnet (w/ thinking) | 77.70% | 12 |
| o1 (OpenAI) | 78.09% | 10 |
Notably, state-of-the-art closed models such as Claude 3.7 Sonnet and o1 significantly surpass the 60% pass mark, with Claude 3.7 passing 12 of 14 credentials. Open-weight reasoning-enhanced models (e.g., DeepSeek R1, Qwen3) approach these results, particularly in medicine. However, persistent deficits remain in law and tax/accounting domains, where no evaluated model achieves universal subject-level success, and difficult exams such as the Certified Judicial Scrivener remain below 50% for most systems.
6. Domain Analysis and Benchmark Distinctions
KMMLU-Pro reveals pronounced domain-specific variability in LLM performance. Models—regardless of weight class or access status—consistently surpass pass marks in clinical and biomedical domains, reflecting the abundance and universality of medical content in pretraining corpora. Conversely, legal and tax/accounting items, grounded in Korean-specific statutes and regulatory practice, drive significant error rates, highlighting limitations in cross-lingual transfer and data coverage. In direct comparison to translation-centric benchmarks such as MMMLU, KMMLU-Pro demonstrates a marked performance gap on law questions, emphasizing the inadequacy of translated academic tasks for evaluating localized, professional knowledge (Hong et al., 11 Jul 2025). This suggests a critical need for institutionally localized datasets for robust model certification prior to deployment in regional professional settings.
7. Access, Licensing, and Community Adoption
The full KMMLU-Pro dataset, including standardized evaluation splits and licensure pass-criteria scripts, is publicly released under a CC BY-NC-ND 4.0 license through Hugging Face (https://huggingface.co/datasets/LGAI-EXAONE/KMMLU-Pro). This ensures broad accessibility for research communities focusing on LLM safety, professional reliability, and cross-lingual competence. The structure and rigor of KMMLU-Pro make it suitable for model comparison, transfer learning studies, and the development of safety-critical LLM applications within the Korean professional context (Hong et al., 11 Jul 2025).