A Multi-Faceted Approach to Scrutinizing the Reliability of a Measure of STEM Teacher Strategic Knowledge

Abstract: Score reliability is necessary for establishing a validity argument for an instrument, and is therefore highly important to investigate. Depending on the proposed instrument use and score interpretations, differing degrees of precision in measurement or reliability are required. Researchers sometimes fail to take a critical stance when investigating this important measurement property, and default to accepted values of commonly known measures. This study takes a multi-faceted approach to scrutinizing score reliability from a measure of STEM teacher strategic knowledge using rater agreement, classical test theory conceptions of reliability, and Generalizability Theory. This detailed examination provides insight about where the greatest gains in score reliability can be realized, given the design of the instrument and the context of measurement.