---
title: 'MCQG-SRefine: A Framework for Expert MCQ Generation'
url: https://www.emergentmind.com/topics/mcqg-srefine-framework
type: topic
---

# MCQG-SRefine: A Framework for Expert MCQ Generation

MCQG-SRefine is a framework for generating and evaluating multiple-choice questions (MCQs) from medical case inputs, optimized for USMLE-style assessments. The system addresses the known deficiencies of large language models (LLMs) in expert-level MCQ generation—specifically outdated knowledge, hallucination, prompt sensitivity, and inconsistent question quality—by combining expert-guided prompt engineering, iterative self-critique and correction, and a reference-free LLM-based automatic evaluation metric [2410.13191].

## 1. Motivation and Problem Definition

Automatic generation of professional-level MCQs from natural language case texts, especially for domains such as medical licensing exams, presents challenges due to required domain expertise, multi-hop reasoning, and compliance with style guidelines. State-of-the-art LLMs (e.g., GPT-4) exhibit issues of hallucination, context drift, outdated factual grounding, and substantial prompt sensitivity, resulting in poor quality and inadequate difficulty calibration of generated MCQs. MCQG-SRefine is designed to address these shortcomings by tightly grounding the generation in up-to-date medical case data, enforcing style and content constraints via expert-curated prompts, and refining outputs through automated, iterative feedback loops, all while automating the evaluation step traditionally reserved for human experts [2410.13191].

## 2. System Architecture

The MCQG-SRefine framework comprises four principal modules: expert-driven prompt engineering, iterative self-critique and self-correction, a comparison feedback loop, and an automated evaluation metric (LLM-as-Judge).

### 2.1 Expert-Driven Prompt Engineering

Each MCQ component—context ($c$), question ($q$), answer ($a$), and distractors ($d$)—is independently initialized using a few-shot prompt template. The templates draw on 3–5 USMLE exemplar questions retrieved via a ColBERT semantic retriever, given an input clinical note ($n$), target topics ($t$), and key concepts ($k$):

\[
P_X: \mathrm{USMLE}\ [X]\ \mathrm{generation:}\ \text{Few-shot examples:} \{R(n, t, k) \to (c_i, q_i, a_i, d_i)\}\ \text{Generate}\ X^0\ \text{accordingly:}
\]

Each template enforces:
- Stylistic and content standards based on NBME guidelines
- Prohibition of answer leakage within context
- Consistency between generated content and specified topics/keypoints

### 2.2 Iterative Self-Critique and Self-Correction

The initial draft $u^0 = (c^0, q^0, a^0, d^0)$ is iteratively improved through a loop consisting of answering, critiquing, and correcting operations. The loop operates as follows:

``` 
Input: medical case n, topics t, keypoints k
u⁰ ← INIT(n,t,k)
for r in 1..R_max do
  (âᵣ, rᵣ) ← ANSWER(uᵣ⁻¹)
  fᵣ ← CRITIQUE(uᵣ⁻¹, âᵣ, rᵣ)
  if CritiqueScore(fᵣ) ≥ τ then break
  uᵣ ← CORRECT(uᵣ⁻¹, fᵣ)
end for
return u* := uᵣ
```

This loop evaluates the draft components ($c, q, a, d, r$) with a 30-aspect guide (e.g., context relevance, question clarity, distractor plausibility, reasoning flow). Each aspect is scored 1–5, and the overall CritiqueScore is computed as

\[
\mathrm{CritiqueScore}(f) = \frac{1}{|\mathcal{A}|} \sum_{c \in \{c,q,a,d,r\}} \sum_{i \in \mathrm{aspects}(c)} s_{c,i}
\]

where $s_{c,i} \in \{1, \ldots, 5\}$ and $\mathcal{A}$ is the set of all aspects. The loop terminates when $\mathrm{CritiqueScore} \geq 0.9$ of maximum or after four rounds.

### 2.3 Comparison Feedback Loop

To select among candidate drafts generated during refinement, MCQG-SRefine employs a comparison prompt to the LLM. The comparison function

\[
F_{\mathrm{cmp}}(I, R_a, R_b) \longrightarrow \{\text{A better, B better, Tie}\}
\]

guides the LLM to compare two questions on USMLE rubrics and state a preference, allowing for variant selection based on both CritiqueScore and LLM comparative judgment.

## 3. LLM-as-Judge: Automated Evaluation Metric

Human expert evaluation of MCQ quality is resource-intensive. MCQG-SRefine replaces this with an LLM-as-Judge system, implemented in both rating and comparison modes.

### 3.1 Rating Mode

A subset $S^*$ of ten aspects (from the original 30), derived via an all-combo greedy search for maximal expert-alignment, is used for score aggregation:

\[
\operatorname{Score}_{\mathrm{Judge}}(u) = \frac{1}{|S^*|}\sum_{i\in S^*} s_i, \quad s_i \in [1, 5]
\]

### 3.2 Comparison Mode

The LLM chooses a preference between two candidate questions, analogously to $F_{\mathrm{cmp}}$ in the feedback loop.

### 3.3 Correlation with Expert Judgment

Filtering to 10 key aspects improves agreement with a senior physician (“Expert X”) from Cohen’s $\kappa = 0.226$ (slight) to $\kappa = 0.539$ (moderate), and raw agreement to 80.0%.

## 4. Formal Workflow and Mathematical Definitions

- **Stopping criteria:** CritiqueScore $\geq 0.9 \cdot \mathrm{max}$ or $r > 4$.
- **Correction:** $u_r = C(u_{r-1}, f_r)$, where $C$ uses free-text feedback and scores to revise each component, leveraging additional few-shot exemplars in the re-prompt.
- **Aspect selection for evaluation:** Greedy and all-combo search algorithms are applied to maximize LLM-expert alignment during metric construction.

## 5. Empirical Evaluation and Findings

Experimental results on USMLE-style MCQ generation demonstrate:

| Metric                                  | MCQG-SRefine           | GPT-4 Baseline      |
|------------------------------------------|------------------------|---------------------|
| Human preference win rate                | 72.5%                  | 17.5%               |
| Human evaluator Cohen's $\kappa$         | 0.667                  | —                   |
| LLM-as-Judge win rate (comparison mode)  | ~80%                   | ~20%                |
| Difficulty: "Easy" questions             | $-80\%$                | —                   |
| Difficulty: "Medium" multiplication      | $\times 2.25$          | —                   |
| Difficulty: "Hard" multiplication        | $\times 4$             | —                   |
| LLM-as-Judge overall score (0–1 scale)   | 0.917                  | 0.864               |

~30% of best questions were obtained in round 1; the remainder were distributed across rounds 2–4. This suggests that iterative self-refinement contributes significantly to output quality.

## 6. Context, Limitations, and Implications

MCQG-SRefine demonstrates the efficacy of integrating expert-derived prompts, in-loop LLM self-critique, and automatic LLM-based adjudication for producing domain-expert-level MCQs, with both machine and human metrics showing substantial improvements over vanilla LLM baselines. The methodology generalizes to other high-stakes testing domains requiring multi-hop reasoning and stringent content control. A plausible implication is the reduction of reliance on human-in-the-loop evaluation via strong LLM-as-Judge concordance with human experts, though performance remains capped by the underlying LLM’s medical reasoning and alignment quality [2410.13191].

Source: https://www.emergentmind.com/topics/mcqg-srefine-framework