MCQG-SRefine: A Framework for Expert MCQ Generation

• MCQG-SRefine is a framework for generating expert-level MCQs by combining expert-driven prompt engineering, iterative self-correction, and automated LLM evaluation.
• It enforces USMLE-style content constraints and mitigates LLM issues like hallucination and outdated knowledge through a refined multi-module process.
• Empirical evaluations show enhanced human preference rates, improved difficulty calibration, and stronger alignment with expert assessments compared to baseline models.

MCQG-SRefine is a framework for generating and evaluating multiple-choice questions (MCQs) from medical case inputs, optimized for USMLE-style assessments. The system addresses the known deficiencies of LLMs in expert-level MCQ generation—specifically outdated knowledge, hallucination, prompt sensitivity, and inconsistent question quality—by combining expert-guided prompt engineering, iterative self-critique and correction, and a reference-free LLM-based automatic evaluation metric (Yao et al., 2024).

1. Motivation and Problem Definition

Automatic generation of professional-level MCQs from natural language case texts, especially for domains such as medical licensing exams, presents challenges due to required domain expertise, multi-hop reasoning, and compliance with style guidelines. State-of-the-art LLMs (e.g., GPT-4) exhibit issues of hallucination, context drift, outdated factual grounding, and substantial prompt sensitivity, resulting in poor quality and inadequate difficulty calibration of generated MCQs. MCQG-SRefine is designed to address these shortcomings by tightly grounding the generation in up-to-date medical case data, enforcing style and content constraints via expert-curated prompts, and refining outputs through automated, iterative feedback loops, all while automating the evaluation step traditionally reserved for human experts (Yao et al., 2024).

2. System Architecture

The MCQG-SRefine framework comprises four principal modules: expert-driven prompt engineering, iterative self-critique and self-correction, a comparison feedback loop, and an automated evaluation metric (LLM-as-Judge).

2.1 Expert-Driven Prompt Engineering

Each MCQ component—context ($c$), question ($q$), answer ($a$), and distractors ($d$)—is independently initialized using a few-shot prompt template. The templates draw on 3–5 USMLE exemplar questions retrieved via a ColBERT semantic retriever, given an input clinical note ($n$), target topics ($t$), and key concepts ($k$):

$$P_X: \mathrm{USMLE}\ [X]\ \mathrm{generation:}\ \text{Few-shot examples:} \{R(n, t, k) \to (c_i, q_i, a_i, d_i)\}\ \text{Generate}\ X^0\ \text{accordingly:}$$

Each template enforces:

• Stylistic and content standards based on NBME guidelines
• Prohibition of answer leakage within context
• Consistency between generated content and specified topics/keypoints

2.2 Iterative Self-Critique and Self-Correction

The initial draft $u^0 = (c^0, q^0, a^0, d^0)$ is iteratively improved through a loop consisting of answering, critiquing, and correcting operations. The loop operates as follows:

Input: medical case n, topics t, keypoints k
u⁰ ← INIT(n,t,k)
for r in 1..R_max do
  (âᵣ, rᵣ) ← ANSWER(uᵣ⁻¹)
  fᵣ ← CRITIQUE(uᵣ⁻¹, âᵣ, rᵣ)
  if CritiqueScore(fᵣ) ≥ τ then break
  uᵣ ← CORRECT(uᵣ⁻¹, fᵣ)
end for
return u* := uᵣ

This loop evaluates the draft components ($c, q, a, d, r$) with a 30-aspect guide (e.g., context relevance, question clarity, distractor plausibility, reasoning flow). Each aspect is scored 1–5, and the overall CritiqueScore is computed as

$q$0

where $q$1 and $q$2 is the set of all aspects. The loop terminates when $q$3 of maximum or after four rounds.

2.3 Comparison Feedback Loop

To select among candidate drafts generated during refinement, MCQG-SRefine employs a comparison prompt to the LLM. The comparison function

$q$4

guides the LLM to compare two questions on USMLE rubrics and state a preference, allowing for variant selection based on both CritiqueScore and LLM comparative judgment.

3. LLM-as-Judge: Automated Evaluation Metric

Human expert evaluation of MCQ quality is resource-intensive. MCQG-SRefine replaces this with an LLM-as-Judge system, implemented in both rating and comparison modes.

3.1 Rating Mode

A subset $q$5 of ten aspects (from the original 30), derived via an all-combo greedy search for maximal expert-alignment, is used for score aggregation:

$q$6

3.2 Comparison Mode

The LLM chooses a preference between two candidate questions, analogously to $q$7 in the feedback loop.

3.3 Correlation with Expert Judgment

Filtering to 10 key aspects improves agreement with a senior physician (“Expert X”) from Cohen’s $q$8 (slight) to $q$9 (moderate), and raw agreement to 80.0%.

4. Formal Workflow and Mathematical Definitions

• Stopping criteria: CritiqueScore $a$0 or $a$1.
• Correction: $a$2, where $a$3 uses free-text feedback and scores to revise each component, leveraging additional few-shot exemplars in the re-prompt.
• Aspect selection for evaluation: Greedy and all-combo search algorithms are applied to maximize LLM-expert alignment during metric construction.

5. Empirical Evaluation and Findings

Experimental results on USMLE-style MCQ generation demonstrate:

Metric	MCQG-SRefine	GPT-4 Baseline
Human preference win rate	72.5%	17.5%
Human evaluator Cohen's $a$4	0.667	—
LLM-as-Judge win rate (comparison mode)	~80%	~20%
Difficulty: "Easy" questions	$a$5	—
Difficulty: "Medium" multiplication	$a$6	—
Difficulty: "Hard" multiplication	$a$7	—
LLM-as-Judge overall score (0–1 scale)	0.917	0.864

~30% of best questions were obtained in round 1; the remainder were distributed across rounds 2–4. This suggests that iterative self-refinement contributes significantly to output quality.

6. Context, Limitations, and Implications

MCQG-SRefine demonstrates the efficacy of integrating expert-derived prompts, in-loop LLM self-critique, and automatic LLM-based adjudication for producing domain-expert-level MCQs, with both machine and human metrics showing substantial improvements over vanilla LLM baselines. The methodology generalizes to other high-stakes testing domains requiring multi-hop reasoning and stringent content control. A plausible implication is the reduction of reliance on human-in-the-loop evaluation via strong LLM-as-Judge concordance with human experts, though performance remains capped by the underlying LLM’s medical reasoning and alignment quality (Yao et al., 2024).