Predictive validity of base-model evaluation metrics for downstream agent performance

Determine whether base-model evaluation measures, including held-out codebase perplexity, state tracking, adversarial question answering, LoCoDiff, RepoBench, and bits-per-byte, reliably predict downstream post-training performance on agentic coding tasks.

Background

The paper discusses several metrics intended to evaluate agentic potential before expensive post-training, including held-out codebase perplexity, state tracking, adversarial question answering, file-state reconstruction, cross-file code completion, and bits-per-byte. These measures are attractive because full post-training evaluations are costly and agentic benchmarks can be unreliable before post-training.

The authors report that the predictive relationship between these base-model measures and downstream post-training ability remains disputed. Their own experiments found that the tracked measures did not reliably guide recipe decisions, but the broader question of which, if any, base-model metrics can consistently predict post-training performance remains unresolved.