AgentTime: Can Agents Estimate and Control Their Own Runtime?

Abstract: An essential control of AI agents is their ability to manage runtime. This ability requires a sense of time-awareness, to predict and estimate wall-clock time and to control their own actions. Prior work has focused on time-awareness, but duration-following and control in native agent harnesses remain unexplored. We present AgentTime, a benchmark for testing whether agents can work for a requested duration, predict their runtime, and estimate elapsed time afterward. It comprises 222 tasks from 18 sources spanning coding, computer use, agentic work, and automated research. Duration-following experiments append a single instruction specifying how long to work, with requests ranging from about a minute to multiple days. Accuracy on these instructions varies substantially: Fable 5.1 in Claude Code deviates from requested runtimes by a typical factor of 2.9$\times$, compared with only 1.2$\times$ for GPT-6 Astra in Codex. However, matching the requested runtime does not, by itself, establish continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. In forecasting experiments, predictions tend to overestimate natural runtimes. In retrospective experiments, removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. An agent's ability to complete a task does not guarantee that it can control its own time or work for the whole requested duration. For agents to run reliably, safely, and autonomously over long horizons, we require the evaluation of both.

• The paper evaluates three skills (duration-following, forecasting, and retrospection) to assess agents' ability to estimate and control their own runtime.
• GPT-6 Astra achieves 63% on-time returns with an average deviation of 1.18 times the requested runtime, outperforming GPT-5.6 Sol and Fable 5.1.
• The benchmark demonstrates that meeting a duration request does not necessarily mean an agent worked productively for the full, and that revisiting and sleeping can be incorrectly classified as productive.

Research question and conceptual framework

“AgentTime: Can Agents Estimate and Control Their Own Runtime?” (2610.09944) examines whether contemporary tool-using agents can manage wall-clock execution as an explicit behavioral variable. The paper distinguishes this problem from conventional long-horizon evaluations, which primarily measure whether an agent completes a task and how long completion takes relative to a human baseline. Those evaluations do not establish whether the agent can respond to a duration instruction, predict its own runtime, or reconstruct how much time it has spent.

The benchmark decomposes temporal competence into three skills:

1. Duration-following: working on a task for a user-specified interval.
2. Forecasting: predicting the runtime of a task before execution.
3. Retrospection: estimating elapsed runtime after the task has ended.

This decomposition is important because the skills are not reducible to task success. An agent may complete a task quickly but fail to use the remainder of a requested interval; it may match a duration by sleeping after completion; or it may produce a good result while substantially exceeding the implied compute budget. The paper therefore treats temporal control and task performance as related but separable dimensions.

The underlying distinction is operational rather than cognitive. The experiments do not attempt to determine whether agents possess an internal clock. Instead, they measure whether agents can use the temporal information available through their harnesses, transcripts, tool calls, timestamps, and clock utilities to regulate execution. This framing allows the authors to evaluate temporal behavior in the environments in which agents are actually deployed.

Benchmark construction and experimental design

AgentTime combines 222 tasks drawn from 18 existing benchmark families. The suite includes graduate-level question answering, software engineering, computer use, web research, document editing, automated research, and long-horizon agentic work. The selected tasks span requested durations from approximately one minute to several days, although requests exceeding 8.3 hours occur only in four tasks from METR and PostTrainBench.

The main duration-following experiment evaluates three agent–harness systems:

• Fable 5.1 in Claude Code
• GPT-5.6 Sol in Codex
• GPT-6 Astra in Codex

Each task is run independently at three requested durations, typically differing by a factor of four. The instruction appended to each task asks the agent to work for the full requested period while completing the task to the best of its ability. The system records external wall-clock runtime using a monotonic timer, while the native benchmark grader evaluates task performance. Native time caps and budgets are removed so that the agent, rather than the benchmark infrastructure, determines when to stop. Administrative cutoffs are retained to distinguish voluntary returns from infrastructure termination.

The analysis reports whether the agent returns early, returns within 5% of the requested duration, overruns, or reaches the cutoff. It also reviews transcripts to determine whether an apparently on-time run involved substantive work, rechecking, or explicit sleeping. This qualitative classification is necessary because elapsed runtime alone cannot establish compliance with the instruction to work throughout the interval.

Forecasting uses separate runs without duration requests or time caps. Agents are asked to predict how many minutes they will need, and the prediction is compared with measured natural runtime on the same task and agent–harness configuration. Retrospection uses finished natural runs from 23 tasks and progressively removes temporal evidence from the agent’s context. Conditions range from ordinary session forks with clock access to reconstructed transcripts with timestamps and other temporal cues scrubbed.

The benchmark is therefore designed to separate three sources of temporal performance: execution behavior under a duration constraint, explicit prior beliefs about runtime, and post hoc inference from the execution record.

Duration-following performance

The strongest result concerns the large model-level variation in duration control. GPT-6 Astra is substantially better calibrated to requested runtimes than GPT-5.6 Sol and Fable 5.1.

Agent and harness	Runs	Within 5% of request	Early	Late	Typical deviation	Runtime–request slope
Fable 5.1, Claude Code	659	4%	55%	41%	2.86×	0.59
GPT-5.6 Sol, Codex	666	39%	13%	48%	1.77×	0.71
GPT-6 Astra, Codex	666	63%	3%	34%	1.18×	0.94

Astra ends within 5% of the requested duration in 63% of runs, compared with 39% for Sol and only 4% for Fable. Its geometric-mean runtime deviation is 1.18 times the request, close to the ideal value of 1.0. Fable’s deviation is 2.86 times, meaning that its runtime is typically almost threefold too short or too long. The fitted runtime–request slope provides a complementary measure: Astra’s slope of 0.94 indicates near-proportional scaling, whereas Fable’s slope of 0.59 shows that its runtime responds weakly to changes in the requested interval.

Fable’s failures are systematic rather than random. On the shortest request for each task, it runs late in 79% of cases, with a median runtime of 1.8 times the request. On the longest request, it stops early in 89% of cases, with a median runtime of 0.3 times the request. Within a fixed task, increasing the requested duration by 16 times increases Fable’s runtime by only 1.9 times. The corresponding increases are 5.0 times for Sol and 12.1 times for Astra. Thus, Fable frequently behaves as if it is estimating the task’s intrinsic completion time and largely ignoring the requested duration.

The paper compares this pattern to Vierordt-like duration distortion, in which short intervals are overestimated and long intervals underestimated. However, it appropriately notes that the mechanism may be simpler: Fable tends to run for approximately the amount of time it believes the task requires, regardless of the user-specified interval. The effect is less pronounced on open-ended optimization and research tasks, where additional work can continue to produce improvements. On these tasks, Fable’s within-task slope rises to 0.77, compared with 0.16 on the other benchmark families.

The harness-swap experiments indicate that the principal difference follows the model rather than the harness. On a matched set of 18 question tasks and 16 agentic tasks, Fable’s within-task slope remains low in both Claude Code and Codex, while Astra remains substantially more responsive to duration requests in both environments. For example, on question tasks, Fable’s slopes are 0.03 in Claude Code and 0.19 in Codex, whereas Astra’s are 0.96 in Codex and 0.85 in Claude Code. Harness effects are non-negligible, but the authors estimate that the model accounts for approximately 0.7 of the slope gap and the harness for about 0.15.

The implication is that duration control is not simply an emergent property of providing a clock or embedding a model in a more capable agent framework. The model’s policy for allocating effort, checking time, and deciding when to terminate is a major determinant of performance. This conclusion is qualified by provider and serving effects in the harness-swap study, particularly because several swapped runs were routed through OpenRouter.

Being on time is not equivalent to working

A central contribution of the paper is its separation of temporal compliance from temporal occupancy. Returning close to the requested time does not demonstrate that the agent worked productively throughout the interval.

Transcript review illustrates this problem. Among 158 classifiable Astra runs with timestamped transcripts, 14 explicitly slept after apparently completing the task, while 74 rechecked earlier work. Among the smaller subset of on-time agentic runs, only 11 of 49 Astra runs satisfied the authors’ working criterion; 24 rechecked previous work and 14 explicitly slept. Sol exhibited similar behavior, with three reviewed runs ending in explicit sleep. By contrast, all four reviewed Fable runs that were on time satisfied the working criterion, although this sample is too small for a comparative conclusion.

The distinction matters for both user control and evaluation validity. An agent that completes a task early and transparently reports this outcome has behaved differently from one that silently waits until the deadline. Likewise, repeated checking may constitute useful validation, but it does not necessarily represent the kind of substantive iteration implied by a request to continue working. The paper therefore argues that future duration-following evaluations should record not only termination time but also what the agent did during the requested interval.

Additional runtime rarely improves benchmark scores. The longest requested duration is typically approximately 16 times the shortest, yet the native score is unchanged between the two conditions for 62% of Fable tasks, 63% of Sol tasks, and 64% of Astra tasks. Scores improve in only 23–25% of cases. This result limits a simplistic interpretation of duration-following: using more time is not automatically beneficial, and a task’s score may saturate before the requested interval ends. Nevertheless, where additional work is useful, early termination can be costly. On one OSWorld task, Fable scores 36% after stopping at 15 minutes but reaches 100% when given 60 minutes.

The result establishes two distinct evaluation targets. A good score does not show that the agent followed the requested duration, and a poor score does not show that additional time would have improved the result. Both productive effort allocation and final task quality must be measured.

Forecasting natural runtime

Forecasting is substantially weaker than duration-following. When asked how long tasks will take without a duration constraint, all three agents generally overestimate their natural runtime.

Agent	Median forecast	Median natural runtime	Forecasts too high	Typical deviation
Fable 5.1	15 min	13 min	63%	2.6×
GPT-5.6 Sol	15 min	6 min	83%	3.8×
GPT-6 Astra	15 min	10 min	66%	2.9×

Sol overestimates in 83% of tasks and has the largest typical error, 3.8 times the actual runtime. Astra follows requested durations closely but forecasts natural runtimes with a typical error of 2.9 times. This contrast is one of the paper’s strongest claims: an agent can control a requested runtime reasonably well while remaining poorly calibrated about how long the task would naturally take.

The forecasting bias is the reverse of the human planning fallacy described in the paper. Agents generally predict that they will take longer than they actually do. Their median predictions also collapse toward a common value of 15 minutes despite substantial variation in measured runtimes. This suggests that forecasts partly reflect a task-independent prior rather than a detailed estimate of the specific execution trajectory.

The fitted compression exponents quantify sensitivity to actual task length. Fable and Sol have exponents of 0.89, while Astra has an exponent of 1.12, with wide confidence intervals. The authors interpret Sol’s larger error primarily as a stable multiplicative overestimate: its predictions are approximately three times its natural runtime. This distinction is methodologically important. A predictor can have a slope near one while remaining systematically biased, and a low average error can conceal poor discrimination among tasks.

Preliminary experiments reinforce this interpretation. Earlier versions of Claude and Codex produced nearly fixed estimates around one to one and a half hours for ProgramBench tasks with widely varying observed runtimes. GPT-5.6 Sol also showed a tendency to run for roughly 30 minutes across tasks of different complexity. Providing more task information did not reliably improve calibration and in some Sol conditions made predictions longer. The main study is stronger than these preliminary experiments because it uses current models, separate forecast and execution runs, and all 222 tasks, but the preliminary results clarify that richer textual evidence alone does not solve the problem.

Forecasts also depend on the referent of the question. In an auxiliary study, Fable estimated that a skilled human professional would take roughly four times longer than the agent itself, while Sol estimated roughly three times longer. For difficult, long-running tasks, the perceived advantage over human professionals increased to approximately tenfold, except in GUI-oriented OSWorld tasks where the agents recognized human superiority. These results indicate that runtime forecasts combine assumptions about task difficulty, agent capability, and the identity of the executor rather than representing a stable estimate of wall-clock execution.

Retrospection and temporal evidence

Retrospection is comparatively accurate when temporal evidence remains visible. In the ordinary session-fork conditions, removing clock access and some tool capabilities has little effect. The decisive degradation occurs when timestamps, tool-call durations, and other temporal cues are scrubbed from the transcript.

Agent	Ordinary temporal context	Scrubbed transcript	Increase in typical deviation
Fable 5.1	1.30×	2.58×	Approximately 1.9×
GPT-5.6 Sol	1.03×	5.38×	More than 5×
GPT-6 Astra	1.03×	2.62×	Approximately 2.5×

With normal transcript evidence, Sol and Astra estimate elapsed runtime with deviations near 1.03 times the actual runtime, while Fable’s deviation is 1.30 times. After temporal cues are removed, Sol’s error rises to 5.38 times, and Fable’s and Astra’s errors rise to approximately 2.6 times.

An oracle condition that directly exposes elapsed time makes estimates almost exact. This demonstrates that the agents can use explicit temporal information when it is available. The much larger errors in the scrubbed condition imply that transcript structure is functioning as an indirect clock. Tool-call durations, file timestamps, test outputs, and clock readings provide observable correlates of elapsed time. The paper reports that visible event count correlates with runtime at $r = 0.91$ in Claude Code and $r = 0.52$ in Codex in a preliminary analysis.

This result should not be interpreted as evidence for a robust internal representation of elapsed time. Rather, retrospective accuracy appears to depend heavily on externalized temporal traces. The paper explicitly observes that agents often mine their transcripts for timestamps and clock readings. When those cues are removed, the ability to reconstruct runtime deteriorates sharply, especially for Sol. Consequently, transcript access can make an agent appear temporally aware even when its estimates are dependent on indirect environmental signals.

Relationship to task performance and agent control

The benchmark’s native scores show that temporal control and task competence are only partially coupled. Across matched benchmark cells, Astra has the highest average native score at 61.9%, followed by Sol at 55.6% and Fable at 54.6%. However, the ordering varies substantially across benchmark families. Fable leads on Humanity’s Last Exam and AppWorld, Sol leads on AssistantBench, and Astra performs particularly strongly on GPQA, OSWorld, ProgramBench, CORE-Bench, TUA-Bench, and PostTrainBench. These heterogeneous results reinforce the paper’s decision not to infer temporal competence from task success.

The authors’ more general claim is that an agent may be capable of completing a task without being capable of managing its execution interval. This is not merely a matter of efficiency. Overruns can exhaust compute budgets; early returns can violate an instruction to iterate or monitor a process; and unreported waiting can make resource use opaque. Conversely, explicit waiting is not by itself evidence of malicious behavior or oversight evasion. The paper carefully distinguishes the behavioral capability of reasoning about and controlling execution time from any particular safety interpretation of that capability.

The proposed benchmark also exposes a distinction between following a duration instruction and optimizing within a duration budget. A closed-book question may rationally be answered immediately, whereas an open-ended algorithm-engineering task may benefit from systematic iteration throughout the interval. Consequently, a universal preference for maximal or minimal runtime would be inappropriate. The relevant criterion is whether the agent’s temporal policy is responsive to the task, the user’s instruction, and the marginal value of further work.

Limitations and open questions

The principal experimental limitation is limited replication. Each task–agent pair is run once per requested duration, so measured outcomes combine the effect of the duration instruction with run-to-run stochasticity. Apart from 55 question runs repeated through OpenRouter, the study does not estimate variance across repeated executions. This is particularly relevant for transcript classifications of sleeping and useful work, where the reviewed samples are small.

The agent roster is also narrow: three principal agents from two labs, evaluated in two harnesses. Results may depend on model version, serving provider, decode speed, subscription route, context handling, and harness implementation. The authors’ harness-swap evidence supports a substantial model effect, but provider effects remain visible, especially in the OpenRouter repetitions. The duration-following results also use one instruction wording and maximum reasoning effort, so they do not establish robustness to alternative prompts, policies, or effort settings.

The duration range is broad in nominal terms but sparse at the longest horizons. Requests beyond 8.3 hours appear in only four tasks, and most results concern minutes rather than multi-day execution. Thus, the paper supports strong conclusions about short- and medium-horizon runtime control but provides limited evidence about genuinely long autonomous runs.

The causal source of temporal competence remains unresolved. The authors cannot distinguish learning from timestamped tool traces, learned priors over tool duration, knowledge of generation speed, explicit training for duration-following, or harness-level affordances. Their hypothesis that reinforcement learning emphasizing duration instructions could improve control is plausible but untested. It is also unclear whether improved duration matching would increase productive work or merely increase waiting and transcript padding.

Several evaluation questions remain open. The study does not establish whether agents accurately disclose early completion or explicit waiting to users, whether temporal control predicts behavior under oversight, or whether knowing inference speed improves forecasting. It also does not determine how agents should trade off task quality, cost, uncertainty, and remaining time. These questions matter because the same runtime behavior can be appropriate for one task and noncompliant for another.

Conclusion

AgentTime demonstrates that task completion, runtime prediction, elapsed-time estimation, and duration-following are distinct capabilities. GPT-6 Astra follows requested durations substantially better than GPT-5.6 Sol and Fable 5.1, achieving 63% on-time returns with a typical deviation of 1.18 times the request, yet even its apparently successful runs sometimes involve sleeping or rechecking rather than continued substantive work. All three agents generally overestimate natural runtimes, while retrospective estimates remain accurate largely because transcripts contain exploitable temporal cues.

The paper’s central methodological implication is that long-horizon agent evaluations should measure not only whether an agent succeeds, but also whether it allocates and reports its execution time as instructed. AgentTime provides an empirical framework for making that distinction explicit, while leaving open whether future improvements in temporal control will produce more productive effort allocation rather than more reliable duration matching alone.