SID-Instruct: Benchmarking Socratic STEM Dialogue

• SID-Instruct is a closed-loop benchmarking pipeline that systematically evaluates interdisciplinary Socratic dialogues in STEM education.
• It provides over 10,000 annotated dialogue turns across diverse lessons, using a detailed nine-field pedagogical schema and quantitative metrics.
• The framework reveals gaps in LLMs' abilities for higher-order reasoning and interdisciplinary guidance, driving improvements in intelligent tutoring systems.

SID-Instruct refers to a closed-loop benchmarking and evaluation pipeline for guided instructional dialogue in interdisciplinary STEM education, as introduced in the SID benchmark framework "SID: Benchmarking Guided Instruction Capabilities in STEM Education with a Socratic Interdisciplinary Dialogues Dataset" (Jiang et al., 6 Aug 2025). Unlike algorithmic frameworks for instruction generation in LLM alignment (e.g., SeDi-Instruct (Kim et al., 7 Feb 2025), Self-Instruct (Wang et al., 2022)), SID-Instruct targets the simulation, annotation, and rigorous assessment of pedagogical dialogues between LLMs operating as teacher and student agents within multi-turn Socratic protocols. The system provides both annotated datasets (10,000+ Socratic turns over 1,920 dialogues) and a formal evaluation methodology to quantify higher-order guidance, knowledge integration, and interdisciplinarity.

1. Framework Overview and Objectives

SID-Instruct is designed to systematically assess the ability of LLMs to conduct, structure, and scaffold complex, interdisciplinary problem-solving conversations in STEM education through a Socratic dialogue paradigm. The pipeline comprises three primary stages:

1. Dialogue generation via LLM-simulated teacher–student interaction across diverse STEM tasks.
2. Rich pedagogical annotation using a nine-field schema capturing intents, strategies, cognitive states, and discipline shifts.
3. Evaluation using both objective behavioral indicators and subjective, rubric-based assessments by strong LLM-judge models.

The core research objective is to expose the pedagogical and interdisciplinary limitations of current LLMs and to drive development toward models capable of orchestrating authentic knowledge integration and cognitive progression in student-like agents.

2. Composition and Structure of the Dataset

The SID-Instruct dataset is constructed as follows:

• 48 interdisciplinary lesson plans spanning physics, chemistry, biology, geography, history, information technology, Chinese, and art, with high-frequency cross-bridges (e.g., Physics–Chemistry, Geo–Biology, IT–Physics).
• For each task: 20 simulated student profiles × 2 inquiry lines = 1,920 dialogues.
• Each dialogue comprises ≥5 Socratic turns, with teacher agents required to scaffold from discipline-specific recall (level L1) to open-ended, inferential reasoning (level L3). Teachers always frame turns as questions, integrating at least two disciplines per conversational round.
• Total: Over 10,000 dialogue turns, with pedagogical annotation and toxicity-control thresholds maintained (all moderation API scores < 0.1, final annotation Cohen’s κ > 0.81).

3. Pedagogical Annotation Schema

Each turn in SID-Instruct is annotated along nine pedagogical axes:

1. Speaker (Teacher or Student)
2. Utterance (verbatim text)
3. Teacher intent (e.g., Introduce concept, Check understanding, Guide reasoning, Trigger transfer, Summarize)
4. Teaching strategy (e.g., Follow-up question, Hint, Analogy, Contextualization, Break-down, Encouragement, Corrective feedback)
5. Discipline (primary subject(s) addressed)
6. Discipline transfer (binary marker for explicit cross-disciplinary engagement)
7. Student cognitive state (e.g., Clear, Vague, Difficulty, Irrelevant, Incorrect, Higher-order)
8. Teacher guidance level (L1: closed, L2: explanatory, L3: open/inferential)
9. Cognitive level (Bloom taxonomy: Remember, Understand, Apply, Analyze, Evaluate, Create)

Annotation is conducted through LLM pre-labeling (Qwen3-32B), dual human adjudication, and conflict resolution by senior experts.

4. Automated and Rubric-Based Evaluation Metrics

SID-Instruct introduces a two-tiered evaluation:

• Objective behavioral indicators: metrics derived from annotation, formalized as:
  • Strategy Density (SD): $\mathrm{SD} = \frac{\mathit{Strat}_t}{T}$
  • Strategy Variety (SV): $\mathrm{SV} = \frac{\mathit{Uniques}}{M}$
  • Interdisciplinary Knowledge Transfer (IKT): $\mathrm{IKT} = \frac{\mathit{Trans}_t}{T}$
  • Bloom Progression (BP): $$\mathrm{BP} = \frac{\max_i(\ell_i)-\min_i(\ell_i)}{5}$$
  • Structure Completeness (SC): $$\mathrm{SC} = \frac{I_{\mathrm{cov}}}{I_{\mathrm{req}}}$$
  • L3 Guidance Rate (L3GR): $\mathrm{L3GR} = \frac{\mathit{L3}}{T}$
  • Cognitive Correction Count (3C): $$\mathrm{3C} = \frac{\mathit{Corr}}{\max(\mathit{Err},1)}$$
  • Composite Score: $$\text{TotalScore} = 0.5(\text{process}) + 0.35(\text{cognition}) + 0.15(\text{interdisc})$$
• Subjective rubrics: five holistic dimensions, scored 1–5 by an LLM-based judge (DeepSeek-V3):
  • Interdisciplinary Scaffolding Response Generation (X-SRG)
  • Multi-disciplinary Reasoning Chain Completeness (M-RCC)
  • Misconception Spotting & Repair (X-MSR)
  • Interdisciplinary Reasoning Alignment (IRA)
  • Transition Coherence & Fluency (TCF)

Outcomes are reported as means and standard deviations across randomized evaluation samples, with strong correlation to expert human judgment (Pearson r > 0.85).

5. Evaluation Protocol and Baselines

SID-Instruct evaluates teacher-role LLMs (e.g., GPT-4o, QwQ-32B, Qwen-2.5-14B-Instruct, SocraticLM, EduChat-R1, InnoSpark) across all dialogues:

• Teacher agents interact with a fixed simulated student (GPT-4o reference), adhering to Socratic constraints (question-only, enforced discipline integration, progressive complexity).
• Maximum tokens per response are set at 1,000; stochastic decoding temperature is 0.7.
• All outputs are subject to annotation and metric computation.

Representative objective scores (TotalScore, out of 1.0): QwQ-32B 0.728, GPT-4o 0.707, InnoSpark 0.692, Qwen-2.5 0.605, revealing consistent gaps in interdisciplinary transfer (IKT < 41%) and cognitive progression (BP < 56%).

6. Key Findings and Implications

Experimental results highlight several key phenomena:

• Leading LLMs (QwQ-32B, GPT-4o, InnoSpark) achieve high fluency and structure on subjective rubrics (AvgScore 4.44–4.93/5) but struggle to guide true interdisciplinary linkage and higher-order cognition as measured by BP and IKT.
• Case studies demonstrate that even high-scoring models frequently respond with linear, single-discipline reasoning, miss opportunities to leverage misconceptions as scaffold points, or break Socratic structure by premature explanation or correction.
• The combined objective (metrics) and subjective (rubric) evaluation methodology exposes a nontrivial gap between linguistic surface quality and authentic pedagogical guidance.

7. Practical Applications and Extensions

SID-Instruct provides both methodological tooling and training resources for:

• Integrating LLM pedagogical agents into intelligent tutoring systems and STEM education platforms.
• Dashboards quantifying strategy density, interdisciplinarity, and guidance progression for teacher training.
• Extending the framework for multimodal instructional settings (e.g., incorporation of diagrams, code) by domain adaptation.
• Mechanistic studies on how LLM prompt engineering affects scaffolding efficacy, error leverage, and self-explanation.

SID-Instruct defines the current best-practice for evaluation of LLMs in guided, Socratic, interdisciplinary instruction and establishes a benchmark for future progress in pedagogically-aware artificial teaching assistants (Jiang et al., 6 Aug 2025).