Don't Let Your Likert Scales Grow Up To Be Visual Analog Scales: Understanding the Relationship Between Number of Response Categories and Measurement Error (2502.02846v1)

Abstract: The use of Visual Analog Scales (VAS), which can be broadly conceptualized as items where the response scale is 0-100, has surged recently due to the convenience of digital assessments. However, there is no consensus as to whether the use of VAS scales is optimal in a measurement sense. Put differently, in the 90+ years since Likert introduced his eponymous scale, the field does not know how to determine the optimal number of response options for a given item. In the current work, we investigate the optimal number of response categories using a series of simulations. We find that when the measurement error of an item is not dependent on the number of response categories, there is no true optimum; rather, reliability increases with number of response options and then plateaus. However, under the more realistic assumption that the measurement error of an item increases with the number of response categories, we find a clear optimum that depends on the rate of that increase. If measurement error increases with the number of response categories, then conversion of any Likert scale item to VAS will result in a drastic decrease in reliability. Finally, if researchers do want to change the response scale of a validated measure, they must re-validate the new measure as the measurement error of the scale is likely to change.