Whether an open System-One model can match Jev

Determine whether an open System-One model can match the performance of the closed System-One model Jev for knowledge tracing when few or no learners are logged.

Background

The paper evaluates Jev, a closed, off-the-shelf System-One LLM, for knowledge tracing in cold-start settings with few or no logged learners. Jev achieves higher mean AUC than the evaluated deep knowledge-tracing models trained on small learner pools and than the System-Two LLM-based baselines in several of the reported settings.

Generality beyond Jev remains unresolved because the experiments include only adapter-mediated tests of other LLMs, a Jev-distilled student, and the open-source Laya model. The authors therefore leave open whether an open System-One model can attain performance comparable to Jev.