Evaluate information used in actual golden-ticket decisions

Evaluate whether reviewers use information absent from their numerical scores when exercising a real golden ticket, using agency data on actual decisive-vote use rather than simulated score-based selection.

Background

The study compares score-based allocation rules within a rejected pool, including panel-mean ranking, minority-support ranking, score-variance ranking, and a lottery. These analyses do not observe reviewers making actual golden-ticket decisions and therefore cannot determine whether reviewers draw on qualitative or otherwise unrecorded information when exercising such discretion.

The paper identifies agency-level data on an operating decisive-vote program as the appropriate setting for resolving this question. A valid evaluation would need to compare proposals actually advanced by one reviewer with comparable proposals not advanced, account for the noncomparability created by reviewer discretion, and specify meaningful outcome differences before examining results.