Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

Abstract: As Competency-Based Education (CBE) is gaining traction around the world, the shift from marks-based assessment to qualitative competency mapping is a manual challenge for educators. This paper tackles the bottleneck issue by suggesting a "Human-in-the-Loop" benchmarking framework to assess the effectiveness of multiple LLMs in automating secondary-level mathematics assessment. Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation. The multi-provider ensemble, consisted of open-weight models -- Eagle (Llama 3.1-8B) and Orion (Llama 3.3-70B) -- and proprietary frontier models Nova (Gemini 2.5 Flash) and Lyra (Gemini 3 Pro), was benchmarked against a ground truth defined by two senior mathematics faculty members (kappa_w = 0.8652). The findings show a marked "Architecture-compatibility gap". Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w ~ 0.38), the larger Orion (70B) model exhibited "No Agreement" (kappa_w = -0.0261), suggesting that architectural compliance with instruction constraints outweighs the scale of raw parameters in rubric-constrained tasks. We conclude that while LLMs are not yet suitable for autonomous certification, they provide high-value assistive support for preliminary evidence extraction within a "Human-in-the-Loop" framework.

• The paper introduces a multidimensional rubric and human-in-the-loop framework for competency assessment in secondary mathematics.
• The study demonstrates that Mixture-of-Experts LLMs outperform dense models, with Gemini models showing higher rubric adherence and reduced instruction drift.
• Results highlight that model architecture and prompt design are crucial for aligning LLM outputs with expert human judgment in educational assessments.

Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

Introduction

The paper "Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics" (2604.26607) presents an empirical evaluation of multiple LLMs for automated, rubric-based assessment in secondary mathematics. The authors focus on Competency-Based Education (CBE), constructing a framework that rigorously assesses students' cross-cutting competencies rather than relying solely on traditional marks-based criteria. The evaluation targets Grade 10 mathematics in the Nepalese context, leveraging representative domain tasks and expert-curated rubrics.

Experimental Methodology

Competency Framework

A multidimensional rubric was constructed to assess four global topics (Matrix, Coordinate Geometry, Trigonometry, Functions) through four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation. Each competency is evaluated at four ordinal proficiency levels: Awareness, Application, Mastery, Influence. The framework emphasizes observable reasoning, partial credit, and separation of procedural fluency from deeper conceptual understanding.

Dataset and Human Ground Truth

The dataset comprises 33 Grade 10 students from Nepal, whose handwritten, open-ended responses were digitized using a multimodal LLM (Gemini 2.5 Flash) serving as an OCR engine. Two senior mathematics educators independently annotated the dataset following a double-blind protocol; conflicts were resolved by a third adjudicator. The inter-rater reliability for human grading achieved almost perfect consensus (quadratic weighted Cohen’s Kappa $\kappa_w = 0.8652$), establishing a robust ground truth.

LLM Ensemble and Prompting Protocol

The benchmarking framework employs a heterogeneous ensemble of LLMs:

• Orion (Llama 3.3-70B, dense model)
• Eagle (Llama 3.1-8B, latency-optimized)
• Nova (Gemini 2.5 Flash, sparse Mixture-of-Experts)
• Lyra (Gemini 3 Pro, high-end Mixture-of-Experts used as final arbiter)

All models received identical, role-based prompts strictly constrained to evidence mapping, structured outputs, and forbidding assignment of numerical grades, ensuring rubric fidelity. Evaluation outputs included multidimensional competency predictions, confidence, and evidence mapping to mitigate black-box opacity.

Evaluation Metrics

Model performance was quantified via quadratic weighted Cohen’s Kappa ($\kappa_w$) versus human ground truth, capturing ordinal distance penalties for misalignment. Inter-model consistency was measured using pairwise unweighted Cohen’s Kappa.

Results

Alignment with Human Expert Judgment

• Nova (Gemini 2.5 Flash) was the best-performing LLM, achieving "Fair Agreement" with human experts ($\kappa_w \approx 0.38$).
• Lyra (Gemini 3 Pro) showed moderate alignment ($\kappa_w \approx 0.27$).
• Eagle (Llama 3.1-8B) performed poorly ($\kappa_w \approx 0.10$).
• Orion (Llama 3.3-70B, 70B parameters) exhibited negative Kappa ($\kappa_w = -0.0261$), signifying systematic contradiction, not just random error, against expert annotation.
• Human-human reliability is near unity ($\kappa_w = 0.8652$), setting a high upper bound.

Inter-Model Agreement

• The highest inter-model agreement was between Lyra and Nova ($\kappa = 0.56$), suggesting Mixture-of-Experts Gemini models share a common decision logic under rubric constraints.
• Eagle and Orion were consistently least aligned to both human annotation and other models.
• LLMs from different architectural families (Gemini vs. Llama) showed minimal agreement, indicating that architectural and training choices drive divergent rubric interpretations.

Failure and Error Analysis

A central, contradictory finding emerged: the largest model (Orion, Llama 70B) completely failed to comply with the rubric, registering "No Agreement." This is interpreted as evidence of "instruction drift" in high-parameter dense models, as opposed to superior instruction-following compliance exhibited by smaller or sparse MoE models in the Gemini family. Models showed signs of hallucination, often assigning competencies based on surface patterns (formula matching) while misaligning with the procedural or conceptual evidence required by the rubric.

Implications

Practical

Current LLMs do not achieve the reliability required for autonomous student certification in complex, multidimensional assessment settings. However, Mixture-of-Experts sparse models (Gemini family) provide consistent, partial evidence extraction, suitable for use as assistive first-pass graders within a Human-in-the-Loop (HITL) framework. This allows substantial reduction in evaluator workload and promotes reproducibility, conditional on mandatory human oversight to ensure validity.

Theoretical

The results advance the argument that in educational and competency-based assessment, model alignment with rubric and instructional constraints outweighs raw parameter count. Instruction drift and divergence in interpretation logic are pronounced in dense LLM architectures under complex assessment protocols. The findings challenge simplistic scaling hypotheses and highlight the centrality of prompt architecture, model training objective, and mixture-of-experts routing in eliciting rubric-compliant responses.

Future Directions

Several directions are articulated for advancing AI-based educational assessment:

• Qualitative reasoning audits of LLM outputs to distinguish superficial pattern matching from legitimate pedagogical reasoning.
• Expansion to larger, more diverse datasets and longer-term stability studies.
• Investigation into mechanisms behind scaling-driven instruction drift in dense models.
• Implementation of dual-critic (evaluator/critic) model pipelines to enhance error detection and mitigate grading bias.
• Topic-specific calibration to address domain variance in assessment difficulty.

Conclusion

The study provides a rigorous empirical benchmark for automated, competency-based assessment of mathematics with LLMs, demonstrating that instructional compliance and rubric adherence, not parameter count, predict alignment with expert human judgment in this domain. Mixture-of-Experts sparse LLMs (Gemini family) constitute the current best-in-class for assistive roles in HITL frameworks. Autonomous deployment for high-stakes assessment remains unsupported; model outputs should augment, not substitute, professional educators. As LLMs and pedagogical prompting strategies mature, further research must address reasoning transparency, architectural idiosyncrasies, and long-term reliability before AI models can play a certifying role in educational assessment (2604.26607).