When is statistical evidence strong enough? Using hypothesis tests to value data collection

Abstract: We recast statistical significance as a choice between making an immediate policy recommendation and deferring it until further evidence is collected. We show that the welfare-optimal decision corresponds, under minimax regret, to a statistical test whose level depends on the cost and precision of additional evidence. Inverting this rule, we introduce and recommend reporting the abstention-value (A-value) alongside traditional p-values to determine where additional data collection is most needed. The A-value defines the break-even welfare cost of abstaining and recommending further experimentation given the initial evidence. When experimentation capacity is limited, prioritizing additional data collection where A-values are the largest yields finite-sample welfare guarantees. We illustrate its implications for economic program evaluation.