KMMLU-Redux: A Curated Korean LLM Benchmark

• The paper presents a meticulous curation process that removes duplicate, contaminated, and ill-posed questions to ensure reliable LLM assessment.
• KMMLU-Redux is built from high-stakes Korean National Technical Qualification exams, applying model-based filtering and expert review to retain only the most challenging items.
• Empirical results reveal a 5–10% accuracy drop on Redux compared to the original, demonstrating its effectiveness in mitigating artificial score inflation.

KMMLU-Redux is a curated, compact Korean-language benchmark designed to address critical weaknesses identified in the original KMMLU dataset for evaluating LLMs on domain-specific, expert-level multiple-choice questions relevant to high-stakes Korean industrial and professional contexts. KMMLU-Redux systematically removes problematic items, including duplicated, contaminated, ill-posed, or error-prone questions, in order to provide a robust, reliable, and challenging testbed for model evaluation, aligning with both industrial certification standards and best practices in benchmark construction (Hong et al., 11 Jul 2025).

1. Origins and Motivation

KMMLU, introduced in 2024, was constructed from 533 publicly available Korean exams, spanning 45 domains (including STEM, applied sciences, humanities, and professional specializations), with a test split of 35,030 questions (Son et al., 2024). However, post-release analysis revealed a series of shortcomings:

• Duplication: 5.36% exact-match duplicate questions within the test set, and 5.46% duplicate between train and test splits.
• Dataset Errors: 7.66% of test items suffered from issues such as leaked answers (i.e., ground-truth option present verbatim in the prompt), ill-posed questions (missing required context), notational mistakes, and low clarity.
• Contamination: 1.88% of the test set was found to appear in FineWeb2 and common web corpora via n-gram matching, threatening evaluation validity.

These flaws risked significant inflation of model scores and unreliable assessment of genuine LLM proficiency in Korean. The motivation behind KMMLU-Redux was to increase reliability by eliminating these confounds, thereby setting a new empirical standard for LLM evaluation in Korea (Hong et al., 11 Jul 2025).

2. Construction and Cleaning Methodology

2.1 Source Selection

KMMLU-Redux selectively focuses on the Korean National Technical Qualification (KNTQ) exams, which are recognized as high-stakes industrial standards requiring a bachelor’s degree or a minimum of nine years’ relevant work experience. The dataset aggregates questions from the 100 most recent KNTQ exams, each mapped to one of 14 industrial domains according to the Korean Standard Industrial Classification.

2.2 Difficulty Filtering

To remove trivial or over-exposed items, the full question pool underwent model-based filtering: seven small open-weight LLMs were tasked with each question. If four or more models answered a question correctly, it was designated as “easy” and subsequently excluded. This process removed 38.6% of candidate questions, ensuring that KMMLU-Redux retained only the most challenging, discriminative items (Hong et al., 11 Jul 2025).

2.3 Denoising and Decontamination

Manual expert review excised problematic questions exhibiting any of the following:

• Leaked Answers: Ground-truth solution appeared verbatim in the prompt.
• Ill-posed Items: Missing external tables, diagrams, or critical information.
• Notation Errors: For example, inconsistent or incorrect use of mathematical notation (“m2” vs “m²”).
• Poor Clarity/Grammar: Ambiguous wording, malformed expressions.

Cross-referencing with FineWeb2 and the original KMMLU further eliminated duplication and web-exposed contamination.

3. Dataset Composition and Statistics

KMMLU-Redux comprises a total of 2,587 questions, all in a 4-choice multiple-choice format, distributed across 14 core industrial domains:

Domain	# Questions
Mechanical Engineering	312
Electrical & Electronics	301
Information & Communication	254
Construction	243
Architecture	182
Chemistry	175
Food & Processing	172
Management & Office	165
Materials	158
Agriculture, Forestry & Fisheries	148
Culture, Arts, Design & Broadcasting	137
Environment & Energy	126
Mining & Resources	117
Safety Management	97

Questions maintain fidelity to the original KNTQ formats, e.g.:

“다음 중 X 공정에서 반드시 고려해야 할 안전 수칙은 무엇인가? A) … B) … C) … D) … Answer: ⟨LETTER⟩”

The compactness of the dataset (relative to the original KMMLU) coupled with strong domain coverage facilitates rigorous, efficient benchmarking (Hong et al., 11 Jul 2025).

4. Evaluation Protocol and Metrics

Evaluation is carried out under a zero-shot Chain-of-Thought (CoT) protocol in Korean (with English prompts for non-Korean models). For non-reasoning models, greedy decoding is used; for reasoning-capable models, temperature is set to 0.6 and top-p to 0.95. Model outputs are parsed via regex for final answer extraction.

The primary metric is exact-match accuracy: $$\mathrm{Accuracy} = \frac{N_{\mathrm{correct}}}{N_{\mathrm{total}}}$$ Spearman’s ρ correlation is computed to assess the consistency of model rankings between KMMLU-Redux and the original KMMLU (Hong et al., 11 Jul 2025).

5. Empirical Results and Comparative Analysis

Notable findings from benchmark evaluations include:

• All tested models exhibited a 5–10 percentage point decrease in accuracy on KMMLU-Redux versus the original, confirming the increased challenge after error remediation.
• Despite heightened difficulty, model ranking orderings were largely preserved (Spearman’s ρ = 0.995), implying that Redux retains the discriminative ordering of LLM capabilities but removes “phantom gains” due to noise or flawed items.
• Item-level analysis demonstrated that “leaked-answer” questions artificially inflated model accuracy by up to +15 points, while ill-posed, clarity, and notation errors suppressed model scores by 5–10 points.
• Hardest domains persist as Mining & Resources and Architecture, where leading models did not exceed 60% accuracy.

Sample performance excerpt: | Model | Acc (%) | |---------------------------------|---------| | o1 (OpenAI) | 81.14 | | Claude 3.7 Sonnet (w/ thinking) | 79.36 | | o3 | 79.92 | | DeepSeek R1 (671B) | 78.51 | | Qwen3-235B-A22B (w/ thinking) | 74.49 | | Phi-4 (14B) | 49.75 | | Aya Expanse 32B | 33.05 |

Top performance verifies that Redux is robust even for state-of-the-art models, with meaningful score stratification (Hong et al., 11 Jul 2025).

6. Applications and Broader Implications

KMMLU-Redux functions as a practical, reliable yardstick for assessing LLM industrial expertise in Korean, particularly for applications in manufacturing, electronics, energy, and allied sectors. Model evaluations on KMMLU-Redux enable organizations to vet LLM candidates specifically on credentialed, real-exam questions before advancing to high-stakes deployments.

The public release encourages ongoing benchmarking and enables the development of automated error-detection methodology, analogous to practices emerging in MMLU-Redux for English (Gema et al., 2024). Regular dataset refreshes (e.g., annual KNTQ updates) and expansion into multimodal and open-ended formats are recommended as next steps.

7. Relationship to Preceding Benchmarks and Future Directions

KMMLU-Redux crystallizes best practices in benchmark curation, building on the methodology and remediation philosophy exemplified by MMLU-Redux (Gema et al., 2024). Unlike translated or synthetic datasets, it prioritizes native Korean exam items and strict quality control, thereby circumventing cultural or linguistic artifacts inherent in earlier approaches.

A plausible implication is that Redux-style audits, with documented expert review, rigorous decontamination, and difficulty calibration, should be codified as standard protocol in all future high-stakes LLM benchmarks—especially for low-resource or culturally specific languages. Further extensions to multimodal evaluation and human-authored frontier domains are anticipated to maintain relevance with evolving industrial standards.

8. Citations

• "From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation" (Hong et al., 11 Jul 2025)
• "KMMLU: Measuring Massive Multitask Language Understanding in Korean" (Son et al., 2024)
• "Are We Done with MMLU?" (Gema et al., 2024)