Information beyond transcript length in retrospective runtime reports

Determine whether agents’ retrospective runtime reports contain information about actual elapsed time beyond the information provided by the length of their transcripts.

Background

The preliminary study compares agents’ runtime estimates with simple baselines based on visible transcript event counts. The correlations between these quantities and true runtime are substantial, especially for Claude Code, but a high correlation alone does not show that the agents’ reports contribute information beyond transcript length.

The paper therefore leaves unresolved whether retrospective reports reflect genuine temporal estimation or primarily encode a transcript-length proxy. Answering this requires evaluating the reports against transcript-based baselines while controlling for the information available in the transcript.