- The paper introduces a skills-based approach to automate teachers’ geometric reasoning assessment using a refined Van Hiele model.
- It employs a structured skills dictionary and advanced LLM methods (RAG and MTL) to significantly boost classification performance metrics.
- The framework enables large-scale, adaptive teacher assessment and offers potential for broader application in hierarchical reasoning domains.
Automated Diagnosis of Teachers’ Geometric Reasoning via Skills-Based Van Hiele Classification
Background and Motivation
The assessment of teachers’ geometric content knowledge (CK) is critical in mathematics education, directly impacting student outcomes and instructional quality. Traditional approaches relying on the Van Hiele model—a hierarchical framework delineating geometric reasoning into five levels from visualization to rigor—demand manual expert analysis of open-ended responses, precluding scalability and longitudinal evaluation. Recent literature underscores the limitations of multiple-choice assessment for Van Hiele levels, emphasizing the necessity of capturing the distinct reasoning processes underlying teachers' geometric thinking [vanhiele1959], [crowley1987van], [gutierrez1991alternative].
This paper introduces an automated, theory-grounded methodology for inferring teachers’ Van Hiele levels from open-ended responses. The innovation centers on a decomposition of Van Hiele levels into 33 explicit reasoning skills, operationalized via a structured dictionary. The central hypothesis asserts that incorporating granular skills information significantly improves level classification by LLMs, aligning with pedagogical theory and addressing prior critiques of surface correctness evaluation.
Methodological Framework
Skills Dictionary and Data Collection
The research methodology entailed the construction of 59 open-ended geometry problems, synthesized from Usiskin’s Van Hiele test [usiskin1982van] and refined with mathematics education experts. The skills dictionary encapsulates distinct reasoning skills characterizing each Van Hiele level, grounded in Crowley’s theoretical decomposition [crowley1987van]. Thirty-one pre-service mathematics teachers provided 226 open-ended responses via a custom web platform, each annotated for Van Hiele level and demonstrated skills by expert raters, yielding high inter-rater reliability (Cohen’s κ=0.84).
Model Architectures
Two model families were evaluated: Retrieval-Augmented Generation (RAG) and Multi-Task Learning (MTL).
- RAG: Each input is encoded and used to retrieve top-K similar annotated responses. The skills-aware variant augments the prompt with skills annotations and the dictionary, while the baseline omits skills information.
- MTL: A mid-size open-source LLM is fine-tuned to classify Van Hiele levels. The baseline uses standard classification, whereas the skills-aware variant integrates a skills-based attention mechanism and an auxiliary skills prediction task, jointly optimizing classification and skills loss.
Hyperparameters and architectures are detailed and controlled to ensure that observed performance gains are attributable strictly to explicit skills modeling.
Empirical Results
Evaluation was conducted via five-fold cross-validation, reporting macro F1, weighted F1, Mean Absolute Error (MAE), and Quadratic Weighted Kappa (QWK). Skills-aware variants in both RAG and MTL methods outperformed their baselines across all metrics, with improvements validated by paired t-tests for statistical significance.
For RAG, F1-macro increased from $0.62$ to $0.70$, QWK from $0.63$ to $0.72$, and MAE decreased from $0.47$ to $0.38$. MTL achieved similar gains: F1-macro from $0.65$ to $0.73$, QWK from $0.59$ to $0.62$0, and MAE from $0.62$1 to $0.62$2. The skills-aware models also exhibited greater stability, evidenced by lower standard deviations.
Ablation studies reveal that the performance boost depends on correctly aligned and pedagogically meaningful skills information. Introducing noisy or mismatched skills reduces accuracy below baseline, affirming that observed gains are not simply due to increased prompt complexity.
Component analysis of MTL confirms that attention-guided and skills-supervised variants provide complementary benefits, with the full model synergistically leveraging both mechanisms for maximal classification efficacy.
Interpretations and Implications
The results highlight that explicit modeling of reasoning skills—derived from pedagogical theory—enables LLMs to reliably distinguish between nuanced levels of geometric reasoning, beyond surface correctness. Skills such as logical connectives or inclusion relationships in geometric families are critical diagnostic features differentiating adjacent Van Hiele levels.
This automated, scalable framework holds substantial practical implications for mathematics education. It facilitates large-scale evaluation of teacher CK, enables longitudinal tracking, and supports adaptive, personalized teacher training systems. The potential to output fine-grained skill profiles can drive targeted instructional interventions—remediating specific reasoning deficiencies.
On the theoretical front, this methodology bridges AI and educational theory, reinforcing the view that hierarchical learning frameworks should be decomposed into explicit skills for robust automated assessment. The approach is generalizable to other domains structured by hierarchical reasoning (e.g., Bloom’s Taxonomy).
Limitations and Future Directions
The dataset is constrained in size, with limited representation at higher Van Hiele levels, which may affect generalizability. Although skill annotations were corroborated by expert consensus, independent annotation protocols with reliability measurements are warranted. Extension of model applicability to novel, unseen geometry problems constitutes an open challenge.
Further research can explore deployment at scale in teacher education, validation across broader populations, and adaptation to other hierarchically-structured reasoning domains. Techniques for automated skill extraction and annotation, as well as dynamic content adaptation in professional development platforms, are promising directions.
Conclusion
This paper establishes the efficacy of skills-based LLM approaches for automated classification of geometric reasoning in teachers, as defined by the Van Hiele model. Structuring explicit reasoning skills within model architecture substantively enhances diagnostic reliability versus baseline models. The implications for scalable, adaptive teacher assessment and professional development are significant, and the paradigm is extensible to other structured educational frameworks. The methodology closes the gap between scalable AI assessment and rigorous pedagogical theory, offering robust foundations for future research and application.
Reference: "Automatically Inferring Teachers' Geometric Content Knowledge: A Skills Based Approach" (2604.13666)
