Genuine reduction vs. task-shift of reward hacking over time

Ascertain whether the apparent decrease in observed reward-hacking rates with increasing model capability on the fixed EvilGenie benchmark reflects a true reduction in the underlying propensity to reward hack, or instead a shift of reward-hacking behavior to harder tasks not captured by the benchmark’s difficulty distribution.

Background

The paper observes no clear overall temporal trend in reward hacking but notes a downward trend for Anthropic reasoning models; however, this is analyzed on a fixed benchmark where solve rates also improve with capability.

The authors caution that reduced rates might reflect displacement of reward hacking toward harder problems rather than genuine mitigation, motivating a need to disentangle true behavioral change from measurement artifacts.