Time-Budget Performance Scaling of LLMs on Simple Long-Horizon Tasks

Determine whether large language models exhibit disproportionately low performance gains from increased time budgets relative to humans on simple, long-horizon tasks, as opposed to the complex AI R&D tasks where this effect has been reported by METR.

Background

The paper references METR’s finding that LLMs gain far less in performance from increased time budgets compared to humans in complex domains such as AI R&D. However, the generality of this finding beyond highly complex tasks is explicitly identified as unclear.

To paper long-term coherence in a simpler setting, the paper introduces Vending-Bench, a simulated environment where agents operate a vending machine over extended horizons. The open question concerns whether the time-budget scaling trend observed by METR for complex tasks also persists for simpler long-horizon tasks.