TIME Benchmark Overview

• TIME Benchmark is a dual-concept framework that denotes both a repository-aware software engineering benchmark and a time-series forecasting benchmark.
• The repository-level benchmark uses strict pre/post snapshot splits to isolate the effect of repository-derived knowledge while preventing prompt leakage and temporal contamination.
• The time-series benchmark combines Time-IMM and IMM-TSF to evaluate irregular, multimodal forecasting under asynchronous sampling and pervasive missing data.

Searching arXiv for the cited benchmark and related benchmark context. TIME Benchmark is not a single universally fixed designation in the recent arXiv literature. The label is used explicitly for at least two distinct benchmark constructions: a time-consistent benchmark for repository-level software engineering evaluation, centered on strict pre/post repository splits and matched A/B testing of repository-derived knowledge, and a benchmark for irregular multimodal multivariate time series forecasting formed by the combination of Time-IMM and IMM-TSF (Xianpeng et al., 27 Mar 2026, Chang et al., 12 Jun 2025). A related source of confusion is that several other papers present time-based or time-series benchmarks without formally naming them “TIME Benchmark,” including work on Time Series Management Systems, automotive CAN intrusion detection, automated forecasting, and high-frequency databases (Arora, 2021, Blevins et al., 2021, Sreedhara et al., 2024, Barez et al., 2023).

1. Nomenclature and principal meanings

In current usage, “TIME Benchmark” denotes two principal benchmark families rather than a single canonical artifact. One is a repository-aware software engineering benchmark whose defining property is temporal consistency between repository knowledge and future engineering tasks. The other is a time-series benchmark that combines a dataset suite and a benchmarking library for irregular, multimodal, multivariate forecasting.

Usage of “TIME Benchmark”	Domain	Core construction
TIME as a time-consistent benchmark	Repository-level software engineering	Snapshot at $T_0$, pre-$T_0$ knowledge construction, tasks from pull requests merged in $(T_0, T_1]$, matched A/B evaluation
TIME Benchmark as Time-IMM + IMM-TSF	Irregular multimodal multivariate time series	Cause-driven irregularity dataset suite plus a plug-and-play forecasting benchmark library with asynchronous multimodal fusion

The software-engineering usage is defined around validity threats specific to repository-aware agents: synthetic task design, prompt leakage, and temporal contamination. The benchmark asks how much an explicit repository-knowledge layer helps when an agent is constrained to what was historically available at repository snapshot time $T_0$, while tasks are derived from pull requests merged later in $(T_0, T_1]$ (Xianpeng et al., 27 Mar 2026).

The time-series usage defines the TIME Benchmark as the combination of Time-IMM, a dataset suite for irregular multimodal multivariate time series, and IMM-TSF, a benchmarking library for forecasting under asynchronous multimodality, variable sampling rates, and pervasive missingness. Its organizing principle is that cause-driven irregularity should be treated as a first-class property rather than as noise to be removed (Chang et al., 12 Jun 2025).

Several neighboring benchmarks are often conflated with these usages because of the shared prominence of time or time-series data, but they are nomenclaturally distinct. The TSMS benchmarking demo is referred to as AIoT-Bench or simply “our benchmark,” not as TIME Benchmark; the forecasting study proposes a benchmarking methodology but explicitly does not coin the name; the CAN work is a time-based intrusion detection benchmark; and the high-frequency database study describes a benchmarking suite that aligns with the concept of a TIME benchmark without introducing that formal title (Arora, 2021, Sreedhara et al., 2024, Blevins et al., 2021, Barez et al., 2023).

2. Time-consistent repository-level software engineering benchmark

The repository-level TIME benchmark formalizes evaluation for repository-aware coding agents under a strict temporal protocol. Let $S(T_0)$ denote the repository snapshot at time $T_0$, and let repository-derived knowledge be constructed by a knowledge extraction function

$\kappa: S(T_0) \to K(T_0),$

where $K(T_0)$ may use only artifacts available at or before $T_0$, including source files, pre-$T_0$0 commit history, docs, issues, and existing tests. Pull requests merged in the future interval $T_0$1 form the task source:

$T_0$2

with a task generation function

$T_0$3

where prompt granularity $T_0$4 controls how much problem framing is provided (Xianpeng et al., 27 Mar 2026).

Its central methodological claim is that repository-aware evaluation is otherwise easily confounded. The benchmark distinguishes two failure modes. Prompt leakage occurs when the task prompt encodes solution-critical details such as exact function names, control flow, or fix instructions that developers would not normally receive up front. Temporal contamination occurs when agent inputs or constructed repository knowledge include post-$T_0$5 artifacts such as PR diffs, post-$T_0$6 files, review comments, or downstream test outputs (Xianpeng et al., 27 Mar 2026).

Evaluation is formalized as a matched A/B comparison. The same agent $T_0$7 is run twice on the identical task set $T_0$8 and the same repository snapshot $T_0$9: once as $(T_0, T_1]$0, augmented with repository-derived knowledge $(T_0, T_1]$1, and once as $(T_0, T_1]$2, with no repository-derived knowledge. Aggregate performance is defined as

$(T_0, T_1]$3

for $(T_0, T_1]$4, where $(T_0, T_1]$5 denotes PR-ground truth such as the modified file set. The paired contrast is

$(T_0, T_1]$6

The intended hypothesis for future knowledge-augmented runs is $(T_0, T_1]$7 versus $(T_0, T_1]$8 under identical model, prompt, repository snapshot, environment, and metric (Xianpeng et al., 27 Mar 2026).

This design makes causal isolation the benchmark’s distinctive feature. Unlike benchmarks that emphasize realism or execution-based validation alone, TIME treats the marginal effect of repository-derived knowledge as the primary estimand, and it does so under a fixed snapshot and tasks derived from real future pull requests (Xianpeng et al., 27 Mar 2026).

3. Prompt control, scoring, and baseline characterization

Prompt generation is itself a benchmark variable in the repository-level TIME framework. Each historical pull request is converted into a natural-language task through an LLM-assisted pipeline, written as $(T_0, T_1]$9 or equivalently $T_0$0, with guardrails that exclude post-$T_0$1 code diffs, exact function names introduced after $T_0$2, explicit repair instructions derived from the PR, review comments, and test outputs created after $T_0$3. The four prompt granularities are defined as Minimal, Concise, Contextual, and Guided. Minimal is a compressed restatement of the high-level change; Concise adds affected component or module; Contextual adds functionality, usage, or failure-mode context and the affected area while avoiding explicit function names or fix steps; Guided adds richer framing and partial hints about the nature of the fix while still avoiding direct copying of the PR implementation (Xianpeng et al., 27 Mar 2026).

The baseline characterization focuses on file-level localization. For task $T_0$4, let $T_0$5 be the set of files modified by the ground-truth PR and $T_0$6 the set of files predicted as relevant by the agent. Then

$T_0$7

with

$T_0$8

and

$T_0$9

Macro-averaged Precision, Recall, and F1 are reported across tasks. No top-$(T_0, T_1]$0 accuracy or execution-based patch correctness is reported in the draft; this is identified as a limitation and intended extension (Xianpeng et al., 27 Mar 2026).

The reported study uses two open-source repositories, DragonFly and React; three Claude-family models, Claude-Sonnet-4, Claude-Sonnet-4.5, and Claude-Ops-4.6; and the four prompt granularities. React analyses include 90 historical PR tasks per model–prompt condition, whereas the DragonFly task count is not explicitly reported. Across both repositories, file-level F1 increases monotonically from Minimal to Guided for every model. On DragonFly, Sonnet-4 rises from 0.2026 to 0.6828, Sonnet-4.5 from 0.2487 to 0.7163, and Ops-4.6 from 0.2952 to 0.8081. On React, Sonnet-4 rises from 0.1898 to 0.6283, Sonnet-4.5 from 0.1946 to 0.7229, and Ops-4.6 from 0.2335 to 0.8078 (Xianpeng et al., 27 Mar 2026).

Distributional analyses reinforce the same pattern. For React, aggregated across models and 90 tasks per condition, mean F1 is 0.242 for Minimal, 0.479 for Concise, 0.574 for Contextual, and 0.770 for Guided; the Precision$(T_0, T_1]$1 rate falls from 60.2% to 3.5%, while the Recall$(T_0, T_1]$2 rate increases from 14.4% to 59.8%. The paper interprets prompt construction as a first-order variable and proposes Contextual prompts as the most defensible default because they reduce under-specification while preserving the need for repository exploration, thereby leaving room to measure the marginal benefit of $(T_0, T_1]$3 (Xianpeng et al., 27 Mar 2026).

A common misconception is that the benchmark already establishes knowledge-augmentation gains. It does not. The present paper reports baseline arms and formalizes the paired A/B protocol, but the with-versus-without $(T_0, T_1]$4 deltas, confidence intervals, significance tests, and effect sizes are explicitly left for future work (Xianpeng et al., 27 Mar 2026).

4. Time-IMM and IMM-TSF as a time-series TIME Benchmark

In the time-series literature, the TIME Benchmark is defined as the combination of Time-IMM and IMM-TSF. Time-IMM is a cause-driven, irregular, multimodal, multivariate time series dataset suite; IMM-TSF is a plug-and-play benchmarking library for forecasting on irregular multimodal time series. The benchmark targets irregular multimodal multivariate time series in domains such as healthcare, environmental sensing, finance, software operations, and networks, where variable sampling rates, asynchronous modalities, and pervasive missingness are the norm (Chang et al., 12 Jun 2025).

Its organizing taxonomy partitions irregularity into three mechanisms with nine concrete types. Trigger-based irregularities comprise Event-Based Logging (GDELT), Adaptive/Reactive Sampling (RepoHealth), and Human-Initiated Observations (MIMIC). Constraint-based irregularities comprise Operational Window Sampling (FNSPID), Resource-Aware Collection (ClusterTrace), and Human Scheduling/Availability (StudentLife). Artifact-based irregularities comprise Missing Data/Gaps (ILINet), Scheduling Jitter/Delay (CESNET), and Multi-Source Asynchrony (EPA-Air). Each dataset preserves asynchronous textual context in addition to numerical multivariate signals (Chang et al., 12 Jun 2025).

The benchmark preserves native timestamps and sampling rates. Numerical features can have different clocks within a dataset, while text arrives at its own timestamps, independent of the numerical streams. Missingness is pervasive and structured, and masks and time-gaps are provided or are derivable. The benchmark uses per-dataset entity-specific sliding-window forecasting with a chronological split of 60% train, 20% validation, and 20% test. Representative dataset-level statistics include, for example, GDELT with 193,205 observations and Mean IOI of approximately 7.24h, RepoHealth with 67,830 observations and Mean IOI of approximately 1.82d, MIMIC with 219,949 observations and Mean IOI of approximately 14.6min, CESNET with 512,760 observations and Mean IOI of approximately 1.17h, and EPA-Air with 49,552 observations and Mean IOI of approximately 1.02h (Chang et al., 12 Jun 2025).

The mathematical task formulation distinguishes unimodal irregular forecasting from multimodal irregular forecasting. For irregular numerical observations,

$(T_0, T_1]$5

the goal is to learn $(T_0, T_1]$6. With asynchronous text

$(T_0, T_1]$7

the multimodal setting becomes $(T_0, T_1]$8 (Chang et al., 12 Jun 2025).

The benchmark also introduces descriptive statistics for irregularity. Feature Observability Entropy summarizes normalized Shannon entropy of per-feature observation shares, Temporal Observability Entropy summarizes counts across time bins, and Mean Inter-Observation Interval summarizes average gaps between observations. These are descriptors of dataset properties rather than forecast losses (Chang et al., 12 Jun 2025).

5. Modeling stack, evaluation protocol, and empirical findings in Time-IMM

IMM-TSF provides a standardized modeling interface for irregular forecasting with asynchronous multimodality. Its components include numerical encoders for irregular models and regular models adapted via canonical pre-alignment; frozen text encoders, including GPT-2, BERT, LLaMA-3.1-8B, and DeepSeek-7B; Timestamp-to-Text Fusion (TTF) for timestamp-aligned text context; and Multimodality Fusion (MMF) for integrating text into forecasts. Canonical pre-alignment constructs a unified timestamp grid over past and future, populates values and binary masks, adds normalized timestamps as features, and appends query timestamps to the input so that models such as Informer, DLinear, and PatchTST can condition on future target times while preserving timing and missingness features (Chang et al., 12 Jun 2025).

The two TTF variants are Recency-Weighted Averaging (RecAvg) with Gaussian proximity and Time2Vec-Augmented Cross-Attention (T2V-XAttn). The two MMF variants are GRU-Gated Residual Addition (GR-Add) and Cross-Attention Addition (XAttn-Add). The library also supports generic time-gap encoding, recency-aware averaging, time-aware attention, and GRU-D-style missingness decays. Training and evaluation use Adam with learning rate $(T_0, T_1]$9, batch size 8, early stopping on validation MSE, and dataset-specific context/query windows, such as 14d/14d for GDELT, 31d/31d for RepoHealth, 24h/24h for MIMIC, 4w/4w for ILINet, and 7d/7d for CESNET and EPA-Air (Chang et al., 12 Jun 2025).

The benchmark compares regular time-series forecasting models, large time-series models, and irregular time-series forecasting models. Evaluated baselines include Informer, DLinear, PatchTST, TimesNet, TimeMixer, TimeLLM, TTM, CRU, Latent-ODE, Neural Flow, and t-PatchGNN. The empirical findings are explicit. Adding textual context reduces MSE on average by 6.71%, with gains up to 38.38% when text is highly informative and semantically aligned with the forecasting target. t-PatchGNN achieves the best unimodal irregular performance, while multimodal variants improve further (Chang et al., 12 Jun 2025).

Ablation results clarify which mechanisms matter. RecAvg and T2V-XAttn perform similarly, suggesting that timestamp awareness via simple recency weighting can be as effective as attention with sinusoidal time encodings on these datasets. GR-Add consistently outperforms XAttn-Add, with the paper attributing the advantage to learned gating that suppresses noisy text and modulates contribution dynamically. Frozen GPT-2, BERT, LLaMA-3.1-8B, and DeepSeek-7B yield similar performance, whereas replacing pretrained encoders with Doc2Vec degrades performance (Chang et al., 12 Jun 2025).

The benchmark also characterizes relative task difficulty. Missing Data (ILINet), Resource-Aware Collection (ClusterTrace), and Event-Based Logging (GDELT) are described as harder because of sparsity, erratic triggers, or latent semantics, whereas Operational Window (FNSPID) and Multi-Source Asynchrony (EPA-Air) are more recoverable. This suggests that the benchmark is intended not merely to test interpolation under missing values, but to stress forecasting systems under heterogeneous causal mechanisms of irregularity (Chang et al., 12 Jun 2025).

6. Related benchmark families and recurrent misconceptions

Several adjacent benchmark traditions help delimit what TIME Benchmark is not. The cloud-based TSMS demo introduces AIoT-Bench, an interactive framework for comparing TimescaleDB, MonetDB, ExtremeDB, and Kairos-H2 on 13 advanced analytical operators such as Centroid Decomposition, SAXRepresentation, ZNormalization, KMeans, KNN, Screen, Recovery, HotSAX, Select, Sum, Distance, and DSTree. Its primary metric is end-to-end runtime