Human Effort and Resource Expenditure in Kaggle Competitions Underlying MLE-Bench Comparisons

Determine the amount of time and computational resources expended by human participants on the Kaggle competitions from which MLE-Bench tasks are sourced, to enable cost- and resource-matched comparisons between AI systems and human performance.

Background

The authors note that MLE-Bench reports metrics tied to percentiles of human performance on Kaggle competitions, but direct cost-competitive comparisons with AI agents are hindered by unknown human effort and resource usage. Teams often work over months with significant compute resources, complicating fair comparisons.

Quantifying human time and resources would allow equitable benchmarking and clearer interpretation of agent results relative to human effort invested in these competitions.