TimeSeriesAgentEvals: Evaluating Time-Series AI Agents

• TimeSeriesAgentEvals is a suite of benchmark frameworks that evaluate AI agents on diverse time-series tasks such as forecasting, anomaly detection, data handling, and code migration.
• The evaluations combine black-box testing with artifact-aware assessments using metrics like MSE, MAE, RMSE, and CRPS while analyzing multi-turn, stateful workflows.
• The paradigm emphasizes non-stationary, aspect-based, and temporal moving-score methodologies to foster extensibility and robust real-world workflow validation.

TimeSeriesAgentEvals denotes a class of benchmark suites and evaluation blueprints for AI agents that operate on time-series problems, including forecasting, anomaly detection, data handling, code migration, repository navigation, data quality assessment, and conversational analysis. Recent work evaluates not only prediction outputs but also code, trained models, structured dialogue traces, and decision rationales, and it increasingly tests multi-turn, stateful, and incident-specific workflows rather than isolated single-step tasks (Cai et al., 19 May 2025, Kong et al., 31 May 2026, Maddi et al., 12 Mar 2026, Wu et al., 2 Jun 2026). Related model-centric work on non-stationary scoring, aspect-based forecast evaluation, and broad zero-shot forecasting benchmarks supplies important methodological foundations for temporally resolved and condition-aware assessment (Erhardt et al., 2017, Cerqueira et al., 31 Mar 2025, Aksu et al., 2024).

1. Benchmark landscape and scope

Current TimeSeriesAgentEvals resources differ in the object of evaluation and in the granularity of what is scored. Some target machine learning engineering agents, some target conversational data-analysis agents, some target multi-turn reasoning systems, and some remain forecast-model benchmarks that are directly reusable when an agent exposes a forecast API. Together they define a broad evaluation surface in which the central unit is no longer only a forecast vector but an end-to-end analytical workflow (Cai et al., 19 May 2025, Kong et al., 31 May 2026, Maddi et al., 12 Mar 2026, Aksu et al., 2024).

Framework	Evaluated system	Distinctive scope
TimeSeriesGym	AI agents on time series machine learning engineering challenges	34 challenges; submission files, code, and models
TimeSage-MT	Agentic time series reasoning systems	240 tasks, 2,680 dialogue turns, 8 domains
AgentFuel / RockfishData/TimeSeriesAgentEvals	Conversational data analysis agents	Domain-customized datasets; stateless, stateful, and incident-specific queries
TSQBench / TSQAgent	LLMs for time-series data quality rating	Dimension identification and dimension-specific quality comparison
GIFT-Eval	General forecasting models or agent-wrapped forecasting APIs	23 datasets, 144,000 time series, 177 million data points
ModelRadar	Forecasting models under specific aspects	Stationarity, anomalies, and forecasting horizons

A recurring theme across these resources is that evaluation is explicitly black-box at the agent boundary while being artifact-aware internally. TimeSeriesGym, for example, treats an AI agent as a black box that can read and write files and run commands. AgentFuel similarly performs black-box, one-shot testing of six popular agents given schema and data previews. TimeSage-MT extends this idea to multi-turn traces whose answers remain verifiable from held-out data and deterministic extractors (Cai et al., 19 May 2025, Kong et al., 31 May 2026, Maddi et al., 12 Mar 2026).

2. Task construction and benchmark design

Task construction varies sharply across frameworks. TimeSeriesGym organizes each challenge around three files under the gym directory: resources/, description.md, and grader.py, with the grader exposing a grade(submission_path) function. Its 34 challenges come from three sources—12 Kaggle competitions, 14 original research-style tasks, and 8 derived “skill isolate” variants—and cover 8 problem types across 15+ domains. The framework is explicitly skill-based: challenges are tagged by core skills such as Data Handling, Model Development, and Code Migration, and templates support scalable generation of missing-data variants, multi-source tasks, and code migration scaffolds (Cai et al., 19 May 2025).

AgentFuel takes a different route by synthesizing domain-customized end-to-end evals. Its pipeline has three phases. In Phase 1, data generation begins from entity specifications, static attribute distributions, and a semi-Markov state machine

$StateTrans=(S,\;s_0,\;\Delta,\;\tau),$

together with per-state measurement specifications and exemplars encoding behaviors such as spike, dip, or flash-crowd. In Phase 2, question–answer generation instantiates both stateless queries, such as count or average over a window, and stateful queries involving first-passage times, sojourn durations, or sequence matches. In Phase 3, the resulting datasets and Q/A pairs are exported into JSON or CSV for external harnesses. The benchmarks reported in this framework cover e-commerce, IoT, and telecom settings and were designed specifically to expose failures on stateful and incident-specific queries (Maddi et al., 12 Mar 2026).

TimeSage-MT formalizes task generation as a deterministic five-stage pipeline over 65 real-world time-series sources. The stages are data selection, profiling, task assignment, reasoning-graph construction, and dialogue generation. The reasoning graph is a typed DAG

$G=(V,E,\tau_V,\tau_E,\alpha),$

whose nodes represent analytical steps such as profile, forecast, anomaly detection, decomposition, and decision. Quality control then applies reproducibility audits, cross-family LLM review, human review, and a final release gate before freezing the 240 tasks and 2,680 dialogue turns (Kong et al., 31 May 2026).

TSQBench is narrower but more formalized around data quality reasoning. It defines two progressive capabilities: dimension identification and quantitative comparison. The master set of quality dimensions is

$$\mathcal{D}=\{ \text{Missing Value}, \text{Noise Level}, \text{Rare Pattern}, \text{Trend}, \text{Frequency}, \text{Amplitude}, \text{Pattern Consistency} \},$$

and the benchmark uses 4,320 total pairwise instances, with 3,320 for training the Perceiver and 1,000 held out for evaluation (Wu et al., 2 Jun 2026).

3. Scoring protocols and evaluation metrics

The metric space used in TimeSeriesAgentEvals is unusually heterogeneous because different frameworks score different artifacts. TimeSeriesGym directly computes numeric measures such as MSE, MAE, RMSE, MAPE, WMAPE, AUROC, event-AP, and F1, and it complements them with LLM-based “judge” assessments for code quality, architecture design, and documentation. Its judge mechanism can produce a 0–1 continuous score and commentary; in the PTB-XL code example, the exact grader uses

$$0.25\cdot(\text{TensorBoard usage})+ 0.25\cdot(\text{Code quality})+ 0.25\cdot(\text{Hydra config})+ 0.25\cdot(\text{Model accuracy}).$$

This dual numeric-versus-judge design is intended to balance objective assessment with contextual judgment (Cai et al., 19 May 2025).

AgentFuel emphasizes repeated runs and correctness labeling. Every question is issued three times, and responses are manually labeled as Correct, Incorrect, or Error. It reports

$$\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat y_i = y_i],$$

$$\mathrm{Pass@2} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\text{at least one of two runs is correct}],$$

and

$$\mathrm{Consistency} = \frac{1}{3} \max_{\ell\in\{\text{Corr,Err}\}} \bigl|\{\,\text{run}_j:\text{label}_j=\ell\}\bigr|.$$

It also reports subgroup accuracies for stateless, stateful, and incident-specific queries (Maddi et al., 12 Mar 2026).

TimeSage-MT separates outcome from capability. Its Outcome Track can include Numerical Accuracy, Factual Verification, Analytical Quality, Code Correctness, and Decision Quality. For forecasting tasks it uses MASE,

$$\mathrm{MASE} = \frac{\frac1h\sum_{i=1}^h |x_{T+i} - \hat x_{T+i}|} {\frac1{T-1}\sum_{t=2}^T |x_t - x_{t-1}|},$$

while anomaly tasks use Precision, Recall, and $F_1$. The per-task outcome score is

$$\mathrm{Outcome}(t)= \frac1{|M(t)|}\sum_{m\in M(t)} s_m(t).$$

Its Capability Track scores nine capabilities, C1–C9, over 28 subchecks, combining deterministic checks with GPT-5.4 judgments for decision coherence and domain grounding (Kong et al., 31 May 2026).

TSQBench uses precision and recall for dimension identification and accuracy for dimension-wise comparison. The Perceiver predicts a dimension set $G=(V,E,\tau_V,\tau_E,\alpha),$0, while the Inspector produces a scalar evidence metric $G=(V,E,\tau_V,\tau_E,\alpha),$1 whose sign determines whether A, B, or TIE wins under dimension $G=(V,E,\tau_V,\tau_E,\alpha),$2. The Adjudicator then aggregates dimension-wise results into global quality scores $G=(V,E,\tau_V,\tau_E,\alpha),$3 and $G=(V,E,\tau_V,\tau_E,\alpha),$4 and an overall winner (Wu et al., 2 Jun 2026).

Forecast-model benchmarks add another layer of normalization. GIFT-Eval evaluates 97 dataset/frequency/horizon configurations and reports normalized MAPE medians, normalized CRPS medians, and an average rank across configurations, with all point-forecast metrics normalized by Seasonal Naive so that Seasonal Naive is the unit baseline (Aksu et al., 2024).

4. Temporal, aspect-based, and non-stationary evaluation

A central methodological issue for TimeSeriesAgentEvals is that time-series performance is rarely homogeneous over time. The non-stationary evaluation framework of Ziel and colleagues addresses this by detecting changepoints in the observed series with PELT, assuming stationarity only within shorter windows, forming empirical predictive distributions inside those windows, and then computing “moving scores.” With a window $G=(V,E,\tau_V,\tau_E,\alpha),$5, model sample forecasts $G=(V,E,\tau_V,\tau_E,\alpha),$6, and observation $G=(V,E,\tau_V,\tau_E,\alpha),$7, the moving-window score is

$G=(V,E,\tau_V,\tau_E,\alpha),$8

For continuous outputs, the framework emphasizes proper scoring rules such as SE and CRPS, including sample versions computed from window-based samples. Three window-construction strategies are given: overlapping fixed-width, overlapping varying-width, and disjoint varying-width. This yields a time series of scores rather than a single global average, and the paper explicitly motivates aggregation only after obtaining time-resolved diagnostics (Erhardt et al., 2017).

ModelRadar generalizes the same intuition from temporal locality to aspect locality. It splits evaluation by stationary versus non-stationary segments, anomalous versus normal observations, and short versus long horizons. Stationarity is detected with KPSS windows and a $G=(V,E,\tau_V,\tau_E,\alpha),$9-value threshold of 0.05, anomalies are defined by a 99% prediction interval of a seasonal-naïve model, and aspect-wise loss is

$$\mathcal{D}=\{ \text{Missing Value}, \text{Noise Level}, \text{Rare Pattern}, \text{Trend}, \text{Frequency}, \text{Amplitude}, \text{Pattern Consistency} \},$$0

The framework further supports a weighted composite score, expected shortfall over the worst $$\mathcal{D}=\{ \text{Missing Value}, \text{Noise Level}, \text{Rare Pattern}, \text{Trend}, \text{Frequency}, \text{Amplitude}, \text{Pattern Consistency} \},$$1 of series, and a win/loss ratio between models. In effect, it formalizes the claim that averaging over all samples can dilute behavior under varying conditions (Cerqueira et al., 31 Mar 2025).

GIFT-Eval extends condition-aware evaluation to scale. It spans 23 datasets over 144,000 time series and 177 million data points, with seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts. Its zero-shot protocol reserves the last 10% of each series for test, segments the test region into non-overlapping windows of horizon length, and permits validation-based tuning of context length or patch size but no in-domain weight updates for foundation models. This provides a standardized external scaffold for agents whose interface can be reduced to forecast(history, horizon) (Aksu et al., 2024).

A common misconception is that a single global score is sufficient. The temporal moving-score framework, ModelRadar’s aspect decomposition, and GIFT-Eval’s configuration-wise aggregation all indicate that robust evaluation often requires explicit conditioning on changepoints, aspect classes, frequency, horizon, or domain rather than only one pooled statistic (Erhardt et al., 2017, Cerqueira et al., 31 Mar 2025, Aksu et al., 2024).

5. Empirical findings and recurrent failure modes

Empirical results across recent benchmarks show that apparent competence on simple or well-structured tasks often does not transfer to stateful, research-grade, or decision-oriented settings. In TimeSeriesGym’s six-task Lite comparison, AIDE+GPT-4.1 produced valid submissions on 57.3% of full tasks and 66.7% on Lite, OpenHands+GPT-4.1 reached 44.4%, and the reasoning model o3 achieved 94.4% valid on Lite. At the same time, original research-code tasks such as CSDI imputation and code migration drove success rates to 40% or below, and increasing time or step budgets did not uniformly improve performance; some tasks degraded when time increased from 4 h to 12 h and from 50 to 150 steps (Cai et al., 19 May 2025).

AgentFuel reports an equally sharp gap by query type. Average accuracy was 66% on e-commerce, 60% on IoT, and 21% on telecom. Broken out by query type, stateless queries were 73% correct, stateful queries 34% correct, and incident-specific queries 10% correct. The best-performing agent on non-incident queries, PandasAI/O4Mini, still collapsed to near-zero on telecom incident queries. Prompt optimization via GEPA with 30 additional AgentFuel evals increased incident-query accuracy by 17% on average (Maddi et al., 12 Mar 2026).

TimeSage-MT shows that multi-turn depth penalizes even strong code-enabled systems. Across six backbones, overall outcome scores cluster around 35–40%, degrading from approximately 50% at L1 to approximately 33% at L4. Code Correctness is relatively high at 70–80%, but Numerical Accuracy is approximately 18% and Analytical Quality approximately 30%. Memory (C4) and Chaining (C5) remain above 80%, whereas Read (C1) and Configure (C3) stay near 20%; Decision-making (C7) and Uncertainty (C8) fall below 50%. This directly contradicts the assumption that successful code execution implies reliable time-series reasoning (Kong et al., 31 May 2026).

TSQBench exposes a related gap in data-quality reasoning. Baseline LLMs achieve 33–43% precision on dimension identification, improving to approximately 60% with the GRPO-trained Perceiver. For quality comparison, baseline text-only systems score 54–61% accuracy, while the tool-augmented Inspector reaches 78–84%. On eleven real-world datasets, these improvements translate into better quality-aware data selection; under a 50% high-quality-data budget, TSQAgent-selected subsets attained RMSE reductions of up to 10% versus random and showed gains over DataShapley, TimeInf, and TSRating (Wu et al., 2 Jun 2026).

Model-centric studies reinforce the same caution against single-score interpretation. ModelRadar found that NHITS attains the best overall average SMAPE, yet this superiority varies with condition: ETS and Theta dominate at 1-step ahead, NHITS dominates at the last horizon, Theta is best on stationary series, NHITS is best on non-stationary series, and ETS and Theta are more robust to anomalies. In financial structured workflows, TS-Agent extends the evaluation target beyond forecast error to trading-risk fidelity and auditability: on Crypto, TS-Agent with GPT-4o improves RMSE, MAE, MAPE, and sMAPE over AutoGluon and agentic baselines, while maintaining a 100% success rate across LLMs and logging every action and rationale in sequential code-edit logs (Cerqueira et al., 31 Mar 2025, Ang et al., 19 Aug 2025).

6. Extensibility, customization, and open directions

A notable property of TimeSeriesAgentEvals is that many frameworks are explicitly designed to be extended rather than merely reused. TimeSeriesGym provides tsgym.templates for missing-data variants, multi-source tasks, and code migration scaffolds, and its authors state that the template library is modality-agnostic: the same pattern can be transferred to image classification or text summarization by supplying a data loader and a metric in grader.py. The framework is open source under the MIT License, and its documentation includes instructions for adding new challenges (Cai et al., 19 May 2025).

AgentFuel likewise treats evaluation as “evals as code.” It supports extensible modules for pattern injection, SQL-based data checkers, and LLM-assisted schema inference, and exports datasets and Q/A pairs for harnesses such as Opik, GenieBench, and Snowflake CI. Its future directions include multi-turn conversational refinements, extension to data-cleaning and ETL workflows such as missing-data imputation, automated grading for scalable CI pipelines, and integration with backend traces and tool-call logs for fine-grained failure diagnosis (Maddi et al., 12 Mar 2026).

TimeSage-MT identifies several next steps that are directly relevant to future TimeSeriesAgentEvals: collaborative multi-agent workflows, evolving user goals, streaming data scenarios in which assumptions must be revised mid-stream, better uncertainty and risk quantification for rare-event or heavy-tailed series, expert-in-the-loop evaluation for healthcare and finance, and domain-specific safety wrappers with user-defined cost functions (Kong et al., 31 May 2026).

TSQAgent’s extension path is more formal and benchmark-centric. The paper suggests adding new quality dimensions such as multivariate cross-correlation drift, spectral entropy, or missing-pattern periodicity; using alternative injection modes such as adversarial corruptions or real-world noise profiles; and expanding beyond pairwise comparison to absolute scoring tasks or multi-way ranking formats. The moving-score literature points to another extension axis: the same sliding-window principle can be applied spatially through moving neighborhoods or through spatio-temporal blocks, not only along a single temporal axis (Wu et al., 2 Jun 2026, Erhardt et al., 2017).

Taken together, these works suggest that TimeSeriesAgentEvals is best understood not as one benchmark but as a layered evaluation paradigm. At one layer are large-scale standardized forecast benchmarks; at another are aspect-aware and non-stationary scoring methods; and at another are agent benchmarks that test repository engineering, multi-turn dialogue, data quality adjudication, stateful analytics, and high-stakes decision support. The unifying principle is that time-series agents are evaluated on evolving data, heterogeneous artifacts, and workflows whose correctness depends on temporal context rather than on a single aggregated score (Aksu et al., 2024, Cerqueira et al., 31 Mar 2025, Kong et al., 31 May 2026).