Treatment of experimental capabilities in scoring

Determine whether capabilities with Krippendorff’s alpha below 0.667 should be excluded from the AgentStripes score once sufficient repeated-assessment reliability data become available.

Background

AgentStripes uses language-model judges to rate capabilities, and it proposes reporting Krippendorff’s alpha for ordinal agreement across repeated runs and judge models. Capabilities with alpha below 0.667 are classified as experimental rather than necessarily being treated as invalid.

The paper does not resolve whether such low-agreement capabilities should continue contributing to the aggregate score. It leaves the decision dependent on future reliability data, making the treatment of experimentally unreliable capabilities an explicit methodological open problem.