Verify generalizability across educational settings and programming languages

Verify whether the predictive relationships between keystroke-level editing features and Breakthrough/Fully Stuck outcomes generalize to other academic semesters, institutions, programming languages, and exercise settings, including evaluations on unseen exercises or models that account for exercise difficulty.

Background

The evaluation uses CodeBench data from a single semester at one institution and does not control for exercise difficulty. Because more difficult exercises may generate more Fully Stuck instances, exercise difficulty could confound the relationship between editing behavior and eventual outcomes. The authors explicitly identify generalization to other semesters, institutions, programming languages, and unseen exercises as unresolved, and suggest exercise-level data splits or models incorporating exercise difficulty as possible ways to investigate it.

References

Generalizability to other semesters, institutions, or programming languages remains to be verified. The evaluation also does not test generalization to unseen exercises; future work should examine exercise-level splits or models that account for exercise difficulty.

— Predicting Struggling Students in CS1 Programming Using Keystroke-Level Editing Features  (2608.25769 - Kofune et al., 26 Aug 2026) in Section 5, Discussion, subsection “Limitations,” paragraph “Generalizability”