TicToc-v1: Temporal Blindness Benchmark

• TicToc-v1 is a benchmark that quantifies temporal blindness in LLM agents by testing if tool calls align with human perception of elapsed time.
• It comprises 725 multi-turn dialogues across 34 scenarios with varied time gaps to assess whether agents over-rely or under-rely on past tool results.
• Empirical findings show that explicit timestamps modestly improve performance in large models, highlighting the need for post-training interventions.

TicToc-v1 is a benchmark for studying temporal blindness in multi-turn LLM agents: the tendency of an agent to operate with a stationary context and therefore ignore the real-world time elapsed between messages when deciding whether to re-invoke a tool. It was introduced in "Temporal Blindness in Multi-Turn LLM Agents: Misaligned Tool Use vs. Human Time Perception" as a test set of multi-turn user-agent trajectories in which the correct final action depends on elapsed time rather than dialogue content alone (Cheng et al., 27 Oct 2025). In this formulation, TicToc-v1 measures whether tool-calling decisions align with human temporal perception, especially in settings where an agent must choose between relying on a previous observation and making a fresh tool call.

1. Conceptual scope and disambiguation

TicToc-v1 is presented as a benchmark for multi-turn tool use under time-sensitive conditions, not as a training method or a deployment framework. Its central problem setting is a dialogue in which an earlier tool result may or may not still be valid by the time a later user message arrives. The benchmark operationalizes this problem by varying the elapsed interval before the final user question and testing whether an LLM calls a tool accordingly (Cheng et al., 27 Oct 2025).

The benchmark is specifically motivated by a mismatch between static dialogue context and evolving external environments. The paper describes two characteristic failure modes: agents may over-rely on previous context, thereby skipping necessary tool calls, or under-rely on it, thereby unnecessarily repeating tool calls (Cheng et al., 27 Oct 2025). TicToc-v1 is designed to expose both errors.

The name should be distinguished from the earlier systems paper "TicToc: Enabling Bandwidth-Efficient DRAM Caching for both Hits and Misses in Hybrid Memory Systems," which concerns a dual-metadata DRAM cache organization for hybrid DRAM + 3D-XPoint memories (Young et al., 2019). The two uses of the term are unrelated in technical scope. In current LLM-agent literature, TicToc-v1 refers to the temporal-awareness benchmark introduced in 2025 (Cheng et al., 27 Oct 2025).

2. Benchmark construction and scenario design

TicToc-v1 contains 725 high-quality multi-turn user–assistant dialogues, each ending in a follow-up user question whose correct resolution either reuses a cached tool result or demands a fresh tool call, depending on elapsed time (Cheng et al., 27 Oct 2025). Every trajectory begins with a user request that triggers a tool call, includes the assistant’s tool-invocation turn, and then presents a final user message after an implicit time gap. This structure makes the final decision temporally contingent rather than purely text-conditioned.

The benchmark spans 34 scenarios categorized by how rapidly their state can change. These are divided into low sensitivity with 7 scenarios, medium sensitivity with 12 scenarios, and high sensitivity with 15 scenarios. Examples given in the paper include regulatory information and published specifications for low sensitivity, 1–10-day weather forecasts and package tracking for medium sensitivity, and live sports odds and stock order books for high sensitivity. Each scenario is also labeled R or R+W, denoting read-only or read/write settings (Cheng et al., 27 Oct 2025).

Trajectory generation used scenario-specific variant strategies. For each scenario, GPT-4o generated 50 candidate trajectories guided by four read-only variants—Repeated Ask, Comparison, Retrieve-Many Ask-One, Simple Reasoning—and four read+write variants—Repeat After Failure, User Confirmation, Repetition of Same Request, In-Context Availability (Cheng et al., 27 Oct 2025). The resulting candidates then passed through a two-stage filtering pipeline. First, GPT-4.1 judged whether the final user turn contained no explicit tool hints, whether no premises were missing, and whether the sample exhibited genuine time dependence. Human reviewers then checked formatting, role ordering, and hallucinations, yielding the retained 725 trajectories.

Scenario sensitivity	Count	Examples
Low sensitivity	7	regulatory info, published specs
Medium sensitivity	12	1–10-day weather forecast, package tracker
High sensitivity	15	live sports odds, stock order book

This construction makes time sensitivity a controlled variable rather than an incidental feature of dialogue data. A plausible implication is that the benchmark isolates temporal decision boundaries more cleanly than generic agent-evaluation suites, because the final user query is explicitly designed so that the need for tool use depends on elapsed time.

3. Human preference annotation and timestamp integration

The paper defines a sample as a trajectory paired with a sampled elapsed time. For each sample, six annotators judged whether it was clearly acceptable not to call a tool (prefer-noTool), clearly better to call a tool (prefer-Tool), or uncertain (Cheng et al., 27 Oct 2025). Only samples with agreement by at least 5 of 6 annotators on either prefer-noTool or prefer-Tool were retained. This produced 232 prefer-noTool samples and 1147 prefer-Tool samples, for a total of 1379 annotated samples.

To simulate real-world delays, each trajectory was assigned three alternative timestamps for the final user message—small, medium, and large elapsed intervals—sampled from a truncated Gaussian,

$\Delta t \sim T\mathcal{N}(\mu,\sigma,[a,b]).$

The parameters depend on scenario sensitivity (Cheng et al., 27 Oct 2025). The summary provides examples such as high-sensitivity "small" with $\mu = 3\,\mathrm{s}$, $\sigma = 1\,\mathrm{s}$, and support $[1\,\mathrm{s}, 6\,\mathrm{s}]$, and medium-sensitivity "large" with $\mu = 3\,\mathrm{days}$, $\sigma = 1\,\mathrm{day}$, and support $[1\,\mathrm{day}, 6\,\mathrm{days}]$. Each sampled $\Delta t$ is added to the penultimate message’s ISO 8601 timestamp, while earlier message times remain fixed.

Timestamp integration is not treated as a change to the environment itself; rather, it augments the dialogue with explicit temporal context. The paper characterizes this as bridging the gap between static dialogue and evolving environments (Cheng et al., 27 Oct 2025). This suggests that TicToc-v1 is partly a benchmark of whether LLMs can use explicit temporal metadata, not merely whether they can infer time sensitivity from semantics.

4. Evaluation protocol and metrics

The evaluation frames the final assistant turn as a binary prediction: whether to call a tool or not (Cheng et al., 27 Oct 2025). For each sample, the full dialogue—either with or without explicit timestamps—is fed to an LLM through its standard function-calling API. The model’s last reply is then inspected to determine whether it attempts a tool call.

A model decision is counted as aligned when it matches the human-preference label: in a prefer-Tool sample, the model calls the tool; in a prefer-noTool sample, the model does not (Cheng et al., 27 Oct 2025). Because the two classes are imbalanced, the paper reports the normalized Preference Alignment Rate (PAR),

$$\mathrm{PAR} = \frac{1}{2}\Bigl(\frac{TP}{TP+FN} + \frac{TN}{TN+FP}\Bigr).$$

Here, $TP$ is the number of prefer-Tool samples where the model called the tool, $\mu = 3\,\mathrm{s}$0 is the number of prefer-Tool samples without a call, $\mu = 3\,\mathrm{s}$1 is the number of prefer-noTool samples without a call, and $\mu = 3\,\mathrm{s}$2 is the number of prefer-noTool samples with a call (Cheng et al., 27 Oct 2025).

The benchmark also reports per-class Attempt Rate:

$\mu = 3\,\mathrm{s}$3

$\mu = 3\,\mathrm{s}$4

The paper explicitly states that no other precision/recall/$\mu = 3\,\mathrm{s}$5 metrics are reported (Cheng et al., 27 Oct 2025). This metric choice emphasizes balanced alignment across the two human-preference subsets rather than raw tool-call frequency or aggregate accuracy.

5. Empirical findings on temporal blindness

The principal empirical result is that, without timestamps, most evaluated models perform only slightly better than random, with the top alignment rate being just over 60% (Cheng et al., 27 Oct 2025). The paper notes that chance would be 50% if guessed uniformly. This result is presented as evidence that ordinary multi-turn LLM agents do not reliably recover human temporal judgments from stationary conversational context alone.

When timestamps are added, the improvement is described as slight and concentrated in larger models. The summary reports that large commercial models, including OpenAI GPT-4 variants and DeepSeek-Chat, improve modestly, peaking at around 65% PAR (Cheng et al., 27 Oct 2025). By contrast, smaller open-source models—Mistral-8B, Llama-3.1-8B, and Qwen-2.5-7B—degrade with timestamps, which the paper interprets as suggesting spurious biases rather than genuine time sensitivity.

The attempt-rate analysis is central to that interpretation. Upon adding timestamps, attempt rates rise uniformly for both prefer-Tool and prefer-noTool classes (Cheng et al., 27 Oct 2025). The paper treats this as evidence that models do not learn to differentiate when a tool should be called; instead, they merely call tools more often. In other words, timestamp augmentation alone does not reliably induce calibrated temporal reasoning.

These results support the paper’s claim that LLM agents exhibit temporal blindness in multi-turn tool use (Cheng et al., 27 Oct 2025). A plausible implication is that the failure is not exclusively one of missing information, since performance remains limited even when explicit timestamps are supplied.

6. Prompt-based mitigations, limitations, and research directions

The paper analyzes several prompt-based interventions and reports limited effectiveness overall (Cheng et al., 27 Oct 2025). Enabling Qwen-3-8B’s long CoT mode changes PAR negligibly. A general reminder prompt of the form "Be aware of elapsed time" has no effect. A few-shot example rules prompt yields a substantial PAR boost for advanced reasoning models such as o3 and o4-mini, but only marginal gains for most other models. The authors therefore conclude that naive prompt engineering fails for most architectures.

This leads to the paper’s central prescriptive claim: aligning tool use with human temporal judgment likely requires post-training interventions, specifically SFT, RLHF or DPO (Cheng et al., 27 Oct 2025). The benchmark itself does not implement these alignment methods; it is framed as a diagnostic resource for measuring failure before such interventions are applied.

Several limitations are explicitly noted. First, the benchmark has class imbalance, since prefer-Tool samples substantially outnumber prefer-noTool samples. Second, the annotator pool is limited to six annotators, and future work aims for at least 100 per sample for more robust consensus. Third, although there are 725 trajectories and three time levels, giving 2175 possible samples, only 1379 were retained after annotation filtering. The paper also identifies potential extensions in more scenarios, more languages, and richer time patterns, including non-Gaussian delays (Cheng et al., 27 Oct 2025).

In summary, TicToc-v1 is characterized as the first large-scale benchmark exposing the temporal blindness of LLM agents in multi-turn tool use (Cheng et al., 27 Oct 2025). Its significance lies in making elapsed time a first-class evaluation variable for tool-calling agents. Within the scope of the reported results, timestamp augmentation and prompt engineering are insufficient to produce strong alignment with human temporal perception, and the benchmark therefore motivates dedicated post-training work on temporally grounded tool-use policies.