Identify factors contributing to incomplete reflection surveys

Determine whether assignment difficulty or student attitudes toward LLM use contributed to participants’ failure to complete all seven reflection surveys and thereby to the study’s approximately 36 percent attrition among consenting participants.

Background

The analysis included only students who completed all seven assignments and corresponding reflection surveys, excluding approximately 36 percent of consenting participants. Although the course’s standard policy permitting students to drop assignments explains some attrition, the study did not directly measure other possible contributors.

In particular, the paper leaves unresolved whether assignment difficulty or attitudes toward LLM use affected students’ decisions not to complete assignments or reflection surveys. Establishing these factors would help assess potential attrition bias in the reported cross-subfield comparisons.

References

While the course's standard drop policy (up to two classwork and one homework assignment) accounts for some of this attrition, we cannot rule out that assignment difficulty or attitudes toward LLM use also contributed to non-completion, a factor we did not measure directly.

— Understanding Student Use of Large Language Models Across Computer Science Subfields  (2610.01158 - Nizamani et al., 1 Oct 2026) in Limitations section, paragraph 3