ELMES: LLM Education Assessment

• ELMES is an open-source evaluation framework designed to assess large language models in educational scenarios using modular, scenario-driven workflows.
• It employs multi-agent dialogue orchestration and a hybrid evaluation engine that integrates objective metrics with subjective pedagogical scoring.
• The system’s modular architecture enables rapid construction of authentic classroom tasks and efficient benchmarking of teaching capabilities.

ELMES is an open-source evaluation framework purpose-built for the automated, reproducible assessment of LLMs in diverse educational scenarios. The system is designed to address deficiencies in contemporary LLM evaluation—specifically, the lack of pedagogically relevant metrics and flexible, fine-grained tools for probing educational functionality beyond generic intelligence. ELMES exposes a modular, scenario-driven workflow that enables rapid construction of multi-agent learning dialogues and objective measurement of subjective teaching qualities, substantially reducing the technical barrier for educational AI research and benchmarking (Wei et al., 27 Jul 2025).

1. Motivation and Design Rationale

LLMs are increasingly deployed in educational contexts ranging from automated tutoring to lesson-plan generation, but robust, context-appropriate evaluation remains undeveloped. Existing benchmarks focus on generalized intelligence or language understanding rather than pedagogically salient constructs: instructional ability, dialogue-based guidance, personalization, and curriculum coherence. ELMES was conceived to systematically fill this evaluation gap by enabling fine-grained, scenario-dependent assessment with explicit measurement of “soft” teaching skills alongside accuracy and content mastery.

ELMES operationalizes a rigorous evaluation process based on realistic educational workflows, with scenario configuration decoupled from execution and scoring logic. This architecture allows educators and researchers to efficiently define, run, and visualize experiments tailored to authentic classroom-like tasks (Wei et al., 27 Jul 2025).

2. Modular System Architecture

The software is segmented into five loosely coupled modules, each handling a critical system function and exposed as an independent Python package:

• Scenario Manager: Ingests a single declarative YAML or JSON configuration, expanding abstract “Tasks” into a suite of concrete test cases. Loads agent, model, task, direction, and evaluation rubric definitions.
• Multi-Agent Dialogue Orchestrator: Constructs a directed acyclic interaction graph between named conversational agents (e.g., teacher ↔ student) as specified in the “directions” field, leveraging the LangGraph engine for concurrent, multi-turn simulation. Logs full dialogue traces into SQLite.
• Hybrid Evaluation Engine: Interleaves objective metric computation (explicit keyword matches, length constraints) with subjective scoring via an LLM-as-a-Judge paradigm. Rubrics are mapped to formal Pydantic schemas, accessible to function-calling LLMs or via prompt-injected JSON Schema for broader model compatibility.
• Data Aggregation Module: Compiles per-case judgments and meta-data to CSV/JSON and computes scenario-level aggregates (mean per dimension, overall mean).
• Visualization Module: Graphs agent workflows and generates multidimensional comparison plots for interpretable, comparative results.

This strict modularity permits targeted extension and independent lifecycle management of core capabilities (Wei et al., 27 Jul 2025).

3. Scenario Configuration and Dialogue Construction

All experiment logic is governed by a single editable configuration file. Users define:

• Agents: Each with a role, model provider (e.g., GPT-4o-mini), and role-specific prompt templates.
• Models: Details for LLM backends (type, credentials).
• Tasks: Input variable sets and task selection modes (e.g., union, cross product).
• Directions: Flow control in dialogue (start nodes, turn routing, conditional halting).
• Evaluation: Metrics and rubric (type, description, format), and scoring LLM (can differ from dialogue agents).

Each {variable} in agent or task definitions is instantiated per test case, composing unique learning contexts (e.g., “Explain Pythagorean theorem to a 9-year-old who loves drawing”). The directions field encodes both simple and conditional multi-agent dialogue flows.

Sample configuration files encapsulate full scenario logic and can be modified or expanded for new research questions without Python coding (Wei et al., 27 Jul 2025).

4. Hybrid Evaluation and Rubric Implementation

ELMES introduces a hybrid scoring paradigm:

• Objective Metrics: Quantified by deterministic rules (e.g., checking for presence of key concepts, response length).
• Subjective (Pedagogical) Metrics: Judged by a strong LLM prompted with the dialogue and a formal evaluation rubric.

Mathematically, for each metric $i$, the formal scoring function is

$$R_i = f_i(\text{generated\_text},\,\text{criteria}_i) \in \{1,2,3,4,5\}.$$

Aggregates are computed as: $$\text{AvgDimension}_i = \frac{1}{N}\sum_{j=1}^N R_{i}^{(j)}, \quad \text{OverallAvg} = \frac{1}{M}\sum_{i=1}^M\left(\frac{1}{N}\sum_{j=1}^N R_{i}^{(j)}\right)$$ where $N$ is the number of test cases and $M$ the number of metrics.

Each dialogue or generation is scored holistically using a system-prompted judge LLM (e.g., Gemini-2.5-Pro), which receives rubric definitions and is instructed to output structured scores matching the scenario schema (Wei et al., 27 Jul 2025).

5. Supported Educational Scenarios and Metrics

ELMES ships with four rigorously defined evaluation scenarios, each operationalizing distinct educational competencies:

Scenario	Input Modality	Dialogue Structure	Key Metrics (examples)
Knowledge Explanation	(knowledge point, persona, stage)	1-turn user → teacher → user	Role Adherence, Emotional Support, Knowledge Mastery, Teaching Method, Personalization
Guided Problem-Solving	(problem, persona)	Up to 6 Socratic teacher-student turns, keyword/turn-based ends	Accuracy, Guidance, Role Consistency, Metacognition, Personalization, Inclusivity
Lesson Plan Generation	(theme, disciplines, grade, time)	Single-turn generation	Core Concept Connectivity, Interdisciplinary Logic, Consistency, Standardization
Contextualized Question	(knowledge, difficulty, format)	Single-turn output	Value Alignment, Context Quality, Pedagogical Utility, Solution Quality

Metric sets are constructed in collaboration with education experts, supporting high-resolution analysis. Each scenario emulates authentic classroom, planning, or assessment tasks (Wei et al., 27 Jul 2025).

6. Benchmarking Results

Systematic evaluation across multiple state-of-the-art LLMs reveals pronounced context-specific strengths. In aggregate scores (mean over several scenarios), Gemini-2.5-Pro is the strongest all-rounder (4.41), excelling in Solution Quality and Personalization. Claude-Opus-4 and Grok-4 lead in affective and metacognitive support, whereas models such as GPT-4o and Qwen-2.5 exhibit trade-offs, sometimes prioritizing structural coherence over deep pedagogical reasoning. The framework exposes fine-grained performance variance both within and across scenario types:

Model	Explain	Guided	LessonPlan	ContextQ	All-Avg
Gemini-2.5-Pro	4.55	4.38	4.25	4.46	4.41
Claude-Opus-4	4.21	4.38	3.94	4.34	4.22
Grok-4	4.04	4.37	3.66	4.34	4.10
GPT-4o	4.03	4.18	4.17	3.32	3.93
...	...	...	...	...	...

Key patterns include diverging priorities between generalist and “educator-specialized” models, supporting the need for scenario-appropriate benchmark selection (Wei et al., 27 Jul 2025).

7. Practical Workflow and Extensibility

Installation and usage follow a standard Python workflow:

git clone https://github.com/sii-research/elmes.git cd elmes pip install -r requirements.txt

Experimentation is conducted through the following steps:

1. Prepare config.yaml.
2. Generate dialogues: elmes generate -c config.yaml.
3. Evaluate and aggregate: elmes eval -c config.yaml.
4. Visualize: elmes visualize -c config.yaml.

Results are stored in structured files (CSV/JSON) with full traceability, and can be visualized for comparative analysis. The fully declarative configuration structure ensures rapid definition of new tasks or teaching strategies. The modular package design supports independent updating or substitution of scoring models, scenario logic, and visualization routines, enabling adaptation to future educational AI research demands (Wei et al., 27 Jul 2025).