---
title: 'MedBench-IT: Italian Medical Exam Benchmark'
url: https://www.emergentmind.com/topics/medbench-it
type: topic
---

# MedBench-IT: Italian Medical Exam Benchmark

MedBench-IT is a benchmark for evaluating large language models on Italian medical university entrance examination questions. It is introduced as the first comprehensive benchmark for this setting and is built from original Italian, expert-authored preparatory materials rather than translated items, with the stated aim of measuring performance in a high-stakes educational domain where linguistic nativeness, subject knowledge, exam robustness, and practical deployability all matter [2509.07135].

## 1. Definition, naming, and scope

MedBench-IT is a native Italian, domain-specific, exam-style benchmark aligned with the structure of Italian medical admissions preparation. Its target problem is not professional clinical decision support in the narrow sense, but evaluation on Italian medical university entrance examinations across six school-level subjects: Biology, Chemistry, Logic, General Culture, Mathematics, and Physics. The benchmark is therefore medically themed and education-oriented, rather than a benchmark of physician-grade clinical competence [2509.07135].

The name can be misleading because it resembles the Chinese MedBench family. The Chinese "MedBench" papers describe cloud-based or large-scale benchmarking systems for Chinese medical LLMs, multimodal models, and agents, but they do not define MedBench-IT as a subtrack, module, or official abbreviation. In that sense, MedBench-IT is a distinct benchmark that shares only partial naming continuity with those Chinese efforts, not an Italian branch of MedBench v4 or an "IT" component inside it [2407.10990][2511.14439].

This distinction is methodologically important. MedBench-IT evaluates text-only, 5-option multiple-choice examination questions in Italian, whereas the Chinese MedBench systems are framed around Chinese medical practice, broader clinical workflows, and, in later versions, multimodal and agentic evaluation. A plausible implication is that MedBench-IT belongs more naturally to specialized educational benchmarking than to hospital-readiness auditing.

## 2. Corpus construction and organization

The benchmark is built from questions provided by Edizioni Simone, described as a leading Italian publisher of medical entrance exam preparatory materials. The starting corpus contained 43,525 questions. Preprocessing removed image-reliant questions, excluded English subject questions, stripped XML/HTML markup, and standardized every item to a text-only format with a question stem, five answer options, and a single correct answer. After preprocessing, the authors selected a stratified sample of 17,410 questions, preserving the original subject and difficulty proportions [2509.07135].

The benchmark is organized as a single evaluation corpus rather than a train/dev/test resource. The paper does not define training, validation, or held-out test splits; all reported experiments are run on the full benchmark. Every question is annotated with subject and difficulty metadata, and the format is uniformly 5-way multiple choice with exactly one correct answer. Because the source material is proprietary, the dataset itself cannot be publicly redistributed; access is instead framed around data-sharing agreements facilitated through the commercial partnership [2509.07135].

| Subject | Questions | Share |
|---|---:|---:|
| Biology | 4,888 | 28.1% |
| Chemistry | 3,992 | 22.9% |
| Logic | 3,014 | 17.3% |
| General Culture | 2,292 | 13.2% |
| Mathematics | 1,679 | 9.6% |
| Physics | 1,545 | 8.9% |

| Difficulty level | Questions | Share |
|---|---:|---:|
| Level 1 (Base) | 8,032 | 46.1% |
| Level 2 (Intermediate) | 7,153 | 41.1% |
| Level 3 (Advanced) | 2,225 | 12.8% |

The distribution is intentionally non-uniform. Biology and Chemistry dominate the subject mix, while Level 1 and Level 2 questions account for most of the corpus. This suggests that MedBench-IT is designed to mirror the empirical distribution of Italian exam-preparation material rather than impose an artificially balanced sampling scheme.

## 3. Evaluation protocol and scoring

The benchmark evaluates both proprietary/API-based systems and locally run open models, with explicit attention to practical deployability. Proprietary models include o1-preview, GPT-4o, GPT-4 Turbo, GPT-4o mini, GPT-3.5 Turbo, Claude 3.5 Sonnet, Claude 3.5 Haiku, DeepSeek-R1, and DeepSeek Chat. Open or local evaluation focuses especially on sub-30B models that can fit in less than 40GB VRAM at half precision, including Qwen 2.5 instruct models from 0.5B to 14B, Gemma 2 instruct models, Llama 3 / 3.1 instruct models, Phi-4, DeepSeek distilled variants, OLMo 2, Aya Expanse 8B, and Minerva-family models. Proprietary systems were accessed through official APIs; open models were run locally using standard libraries such as vLLM; experiments were conducted between December 2024 and January 2025 [2509.07135].

Prompting is zero-shot and entirely in Italian. Two prompts are used: a standard prompt that asks for direct answer selection in the format `Risposta: [numero]`, and a reasoning-eliciting prompt that asks the model to explain its reasoning before providing the final answer in the format `Ragionamento: ...` followed by `Risposta: [numero]`. For the main evaluations, unless otherwise specified, all models used `temperature = 0` to encourage deterministic outputs. Only the final numerical answer is scored, even when the model produces a rationale [2509.07135].

The main metric is accuracy. The paper gives the following formula, while noting that the PDF text appears to contain a typographical brace omission:

$$
\text{Accuracy} = \frac{\text{Number of Correct Answers}{\text{Total Number of Questions} \times 100\%
$$

The intended meaning is percentage correct over all evaluated questions. Accuracy is reported overall, by subject, and by difficulty level. The paper does not describe a more complex answer parser, fallback heuristics, or partial-credit mechanism. Because all questions are 5-way multiple-choice with one correct option, evaluation is exact-match over the predicted answer index [2509.07135].

## 4. Empirical performance profile

The overall leaderboard is led by proprietary or very large API-exposed models. DeepSeek-R1 achieves 91.9% under the standard prompt and 91.8% under the reasoning prompt, followed by o1-preview at 89.1% and 90.7%, Claude 3.5 Sonnet at 87.8% and 88.3%, DeepSeek Chat at 86.1% and 87.3%, and GPT-4o at 83.9% and 86.8%. Among local models below 30B parameters, the strongest reported systems are Phi-4 (14B), with 76.8% standard and 67.9% reasoning, and Qwen 2.5 14B, with 72.6% standard and 76.9% reasoning [2509.07135].

| Model | Standard | Reasoning |
|---|---:|---:|
| DeepSeek-R1 | 91.9% | 91.8% |
| o1-preview | 89.1% | 90.7% |
| Claude 3.5 Sonnet | 87.8% | 88.3% |
| GPT-4o | 83.9% | 86.8% |
| Phi-4 (14B) | 76.8% | 67.9% |
| Qwen 2.5 14B | 72.6% | 76.9% |

A central empirical pattern is that Logic and Mathematics are the hardest subjects across the board, while Biology, Chemistry, and General Culture are the easiest. For GPT-4o under standard prompting, Biology reaches 92.4% whereas Logic falls to 64.9%, a difference of roughly 27.5 points. Similar subject asymmetries appear for both proprietary and local systems, although stronger models degrade less sharply. This indicates that MedBench-IT does not merely reward factual recall; it also probes reasoning-heavy domains where answer selection depends on more structured inference [2509.07135].

The benchmark also foregrounds resource-constrained deployment. Top proprietary/API systems cluster roughly between 84% and 92%, while the strongest local systems reach the low-to-mid 70s. The paper treats this as a practically important result: sub-30B local models can be meaningfully competitive in an Italian educational setting, even if they do not match the best large proprietary reasoners.

## 5. Robustness, prompt sensitivity, and auxiliary analyses

MedBench-IT goes beyond leaderboard accuracy by explicitly analyzing reproducibility, ordering bias, reasoning-prompt sensitivity, and readability effects. For reproducibility, GPT-4o was evaluated twice on the full 17,410-question corpus with identical parameters but at temperature 1. The result was 88.86% response consistency, meaning 11.14% of answers changed across runs. Consistency varied substantially by subject, reaching 96.8% in Biology and 73.6% in Logic; by difficulty, it was 89.8% for Level 1, 88.1% for Level 2, and 88.0% for Level 3. The two runs did not differ significantly under McNemar’s test, with \(p > 0.05\), and the paper interprets this as ordinary stochastic variation rather than systematic instability [2509.07135].

Ordering bias was tested by comparing the original dataset with a version in which answer options were shuffled. For GPT-4o, accuracy changed from 83.9% to 83.5%, a decrease of 0.4 points that was not significant under McNemar’s test. For Claude 3.5 Haiku, accuracy changed from 80.4% to 79.5%, a decrease of 0.9 points that was significant with \(p < 0.001\). The reported interpretation is that positional bias is generally small on this benchmark, but can still be statistically detectable for some models [2509.07135].

Reasoning-prompt evaluation yields a deliberately nontrivial picture. Explicit reasoning does not universally improve performance. It helps some top models and many mid-sized models, including o1-preview, GPT-4o, Qwen 2.5 14B, Gemma 2 9B, and Llama 3.1 8B, but harms others, including GPT-4 Turbo, Claude 3.5 Haiku, Phi-4, Gemma 2 2B, and Qwen 2.5 0.5B. The most extreme failure case is Aya Expanse 8B, which falls from 46.7% to 0.1% under the reasoning prompt. The paper also notes a qualitative regularity: models tend to produce shorter explanations when correct than when incorrect, suggesting that verbosity may correlate with uncertainty [2509.07135].

The benchmark further analyzes readability using the Italian Flesch Reading Ease score, referred to as the Formula di Flesch-Vacca and computed with `textstat`. Logistic regression finds a statistically significant but small inverse relationship between readability and performance: for the standard prompt, the reported odds ratio is approximately 0.997 per point increase with \(p < 0.001\); for the reasoning prompt, it is approximately 0.999 with \(p < 0.001\). The stated conclusion is that readability has only a minor effect relative to subject matter and reasoning demand.

## 6. Position within medical benchmarking and stated limitations

MedBench-IT occupies a distinctive position within medical AI evaluation. Unlike the Chinese MedBench systems, which are framed as cloud-based or clinically aligned infrastructures for Chinese medical LLMs, multimodal models, and agents, MedBench-IT is a native Italian benchmark centered on admission-style multiple-choice examination questions. Unlike MedPerf, which is explicitly a federated evaluation platform and governance model for privacy-preserving multi-site medical AI benchmarking, MedBench-IT is a benchmark corpus and evaluation methodology rather than a distributed execution framework [2407.10990][2511.14439][2110.01406].

This positioning has direct interpretive consequences. MedBench-IT should not be read as a benchmark of professional medical competence. The paper explicitly states that the dataset consists of preparatory materials rather than advanced medical-school or clinical reasoning tasks, and therefore should not be interpreted as a test of physician-level performance. It is also not suitable for retrieval-augmented generation assessment because the benchmark does not include supporting context documents, and it does not test multimodal reasoning because image-based questions were removed during preprocessing [2509.07135].

The paper is explicit about additional limitations. The benchmark is entirely multiple-choice, so it does not fully evaluate open-ended explanation or synthesis. Only zero-shot prompts were tested; few-shot configurations, especially with full chain-of-thought exemplars, are left for future work. Contamination from pretraining corpora cannot be entirely ruled out, although the authors regard it as unlikely given the source material. Reproducibility and ordering-bias analyses were conducted on only a limited subset of models. Finally, the proprietary nature of the underlying Edizioni Simone dataset creates reproducibility barriers and reduces openness relative to fully public benchmarks [2509.07135].

Future work proposed in the paper includes extending the benchmark to more advanced medical examinations, conducting deeper qualitative error analysis, moving beyond multiple-choice evaluation formats, maintaining a continuously updated leaderboard, and potentially allowing evaluation of additional submitted models. A live leaderboard hosted by OnePix Academy is planned as an ongoing evaluation platform. In that sense, MedBench-IT is best understood as an Italian-native, medically themed educational benchmark whose principal contribution lies not only in scale, but in showing that non-English, domain-specific evaluation can be combined with robustness-oriented analysis and realistic deployment constraints in a unified benchmark design [2509.07135].

Source: https://www.emergentmind.com/topics/medbench-it