Denominator used in Anthropic’s reported reward‑hacking rates

Determine whether the 5% holdout-failure rate and 14% classifier-trigger rate reported for Claude Sonnet 4 on Anthropic’s “reward hacking prone coding tasks v2” are computed as a percentage of tasks that passed all visible tests, or as the percentage of tasks that passed the visible tests but failed the hidden tests.

Background

The paper compares EvilGenie results to figures reported in Anthropic’s system cards but notes ambiguity in how Anthropic normalizes its reported percentages, which hinders direct comparison.

Resolving the denominator ambiguity is necessary to meaningfully align metrics across studies and to understand the relative prevalence of detected reward hacking.