Quantify and validate changes in students’ acquisition-reporting criteria

Determine how much the criterion used by University of Tokyo students to report acquisition of informatics topics changed after the introduction of the Common Test for University Admissions examination in Informatics I, and establish how consistently that criterion aligns with the criteria of teachers, test designers, and curriculum designers.

Background

The study finds that students’ self-reported acquisition rates for computer-science topics increased sharply after Informatics I was introduced into Japan’s Common Test for University Admissions. The magnitude of this increase is judged too large to represent a comparable improvement in actual proficiency and is instead interpreted as a change in students’ standards for deciding whether they had acquired a topic.

Because self-reported acquisition is used as a longitudinal measure, changes in respondents’ reporting criteria undermine direct comparisons across curriculum and examination reforms. The paper partially addresses this issue by comparing direct-entry and gap-year students, but it does not quantify the magnitude of the criterion shift or determine whether students’ standards correspond to those used by teachers and by the institutions responsible for designing examinations and curricula.