ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response

Abstract: Time series question-answering (TSQA), in which we ask natural language questions to infer and reason about properties of time series, is a promising yet underexplored capability of foundation models. In this work, we present ARFBench, a TSQA benchmark that evaluates the understanding of multimodal foundation models (FMs) on time series anomalies prevalent in software incident data. ARFBench consists of 750 questions across 142 time series and 5.38M data points from 63 production incidents sourced exclusively from internal telemetry at Datadog. We evaluate leading proprietary and open-source LLMs, VLMs, and time series FMs and observe that frontier VLMs perform markedly better than existing baselines; the leading model (GPT-5) achieves a 62.7% accuracy and 51.9% F1. We next demonstrate the promise of specialized multimodal approaches. We develop a novel TSFM + VLM hybrid prototype which we post-train on a small set of synthetic and real data that yields comparable overall F1 and accuracy with frontier models. Lastly, we find models and human domain experts exhibit complementary strengths. We define a model-expert oracle, a best-of-2 oracle selector over model and expert answers, yielding 82.8% F1 and 87.2% accuracy and establishing a new superhuman frontier for future TSQA models. The benchmark is available at https://huggingface.co/datasets/Datadog/ARFBench.

• The paper introduces ARFBench, a novel benchmark for time series QA in incident response, built on real-world data and human annotations.
• It systematically evaluates multivariate QA tasks across three tiers using multimodal data from production incidents.
• Results highlight superior performance of vision-language models over LLM-only methods and underscore the potential of hybrid frameworks.

Benchmarking Multimodal Time Series Question Answering for Incident Response: A Summary of ARFBench

Motivation and Context

Failures in large-scale production software systems can result in enormous economic impact, necessitating rapid and accurate root cause analysis and mitigation. Central to modern incident diagnosis is the use of observability metrics—multivariate time series representing signals such as CPU, latency, or error counts. The human workflow in incident response is heavily driven by natural language question-answering (QA) about these time series, e.g. to verify whether a pattern is anomalous, to locate anomaly boundaries, or to analyze inter-series relationships.

Despite the prevalence of such question-driven reasoning, it is unclear to what extent contemporary foundation models (FMs)—especially vision-LLMs (VLMs) and LLMs—can perform complex, multimodal TSQA tasks that involve structured, multivariate time series analysis in realistic observability settings. The "ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response" (2604.21199) paper addresses this by introducing ARFBench, a TSQA benchmark that directly reflects the challenges faced in real-world incident response.

Benchmark Design and Data

ARFBench is constructed from 63 production incidents and 142 complex, highly-multivariate time series, sampled from Datadog's internal monitoring telemetry. The curated dataset comprises 750 multiple-choice QA pairs spanning 5.38 million data points, all anonymized and annotated with context extracted from production incident timelines. Context includes human-authored incident logs, metric descriptions, and time series data provided in both image and tabular forms.

Figure 1: ARFBench consists of 750 QA pairs derived from 63 real-world incidents and 142 observability time series. Nonstationarity and high multivariate complexity are central to the benchmark.

Questions are rigorously structured across three escalating tiers:

• Tier I: Binary anomaly detection (presence)
• Tier II: Property analysis and localization (identification, magnitude, start/end boundaries, and categorization)
• Tier III: Multi-metric compositional reasoning (correlation and leading/lagging anomaly relationships across paired series)

Each question is enriched with operational metadata, providing full context for correct anomaly interpretation and requiring substantive multivariate and temporal reasoning.

Figure 2: Example ARFBench questions spanning all tiers. Models receive structured context and data, including channel tags for variate-level reasoning.

Figure 3: Multivariate context is indispensable. Many anomalies are only detectable within the context of grouped variates, as illustrated by the need to reason about channel-level deviations.

ARFBench contrasts with prior TSQA datasets by its real-world provenance, context-rich construction, multivariate focus, and grounding in human annotations, thereby filling critical gaps in the existing TSQA benchmark landscape.

Modeling Approach and Evaluation Protocol

The benchmark is designed to evaluate a spectrum of modeling approaches:

• Few-shot foundation models: State-of-the-art proprietary and open-source LLMs (e.g., GPT-5, Qwen3 32B) and VLMs (e.g., Qwen3-VL, GPT-4o, Gemini 3 Pro) are evaluated using few-shot prompting with carefully standardized input contexts.
• Post-trained and hybrid models: The paper introduces a family of post-trained models, notably a TSFM+VLM hybrid (Toto-1.0-QA-Experimental), trained with both synthetic and real data, leveraging RLVR (reinforcement learning with verifiable rewards) and SFT (supervised fine-tuning) over multimodal embeddings.

All models are assessed on overall, per-tier, and per-category accuracy and macro-F1. The multiple-choice format allows for robust, interpretable evaluation, circumventing many ambiguities inherent in traditional anomaly detection metrics.

Experimental Results

Baseline Model Performance

Frontier VLMs substantially lead TSQA benchmarks, with GPT-5 achieving 62.7% accuracy and 51.9% macro-F1, exceeding random and majority-class baselines by >17 percentage points. VLMs consistently outperform LLM-only architectures, particularly on higher-difficulty questions involving temporal and multivariate reasoning.

Tier-wise breakdown shows that while models are comparatively adept at anomaly presence detection (Tier I), performance degrades on property localization (Tier II) and especially on compositional cross-series reasoning (Tier III), where even top models' F1 drops below 50%.

Hybrid TSFM-VLM Modeling

Specialized post-trained architectures yield competitive or superior results. The Toto-1.0-QA-Experimental model (TSFM-VLM) achieves 63.9% accuracy and 48.9% F1, surpassing leading proprietary models in several per-tier metrics while trailing GPT-5 slightly in F1. This validates the importance of jointly training time series encoders with multimodal decoders for complex TSQA.

Notably, pure TS-LLMs and TSFM-LLMs remain substantially behind VLMs, underlining the critical role of visual and context integration in high-dimensional observability time series analysis.

Figure 4: Category and domain distributions in ARFBench: coverage spans the main incident QA types and critical observability domains (application, infrastructure, networking, database, security).

Model-Human Complementarity

A user study with domain and non-domain experts (Datadog researchers) establishes that human experts achieve higher performance (up to 72.7% accuracy, 64.6% F1) than models overall, but error patterns are substantially non-overlapping:

• Domain experts: More robust to domain-specific signals, but susceptible to fine-grained perceptual and instruction-following errors (e.g., visually subtle anomalies, prompt interpretation issues).
• Models: Strong perceptual abilities (especially VLMs), but frequent hallucination and domain knowledge errors, e.g., failure to use context fully or misclassifying subtle system behaviors.

A theoretical "model-expert oracle" (choosing the correct answer between the two when possible) yields 87.2% accuracy and 82.8% F1, a new superhuman performance ceiling, indicating strong potential value from collaborative decision systems.

Figure 5: Models and domain experts make complementary errors; a model-expert ensemble reaches superhuman accuracy, highlighting different skill domains.

Qualitative Error Analysis

In-depth qualitative analysis reveals primary model failure modes:

• Limited context usage (e.g., neglecting crucial signal descriptions or failing to leverage multivariate structure)
• Incorrect perception (hallucinating time series properties or missing small anomalies)
• Domain knowledge gaps (failure to apply incident-specific semantics)
• Instruction following/inference errors (often more common in experts than models, especially in long multi-turn QA)

Case studies illustrate these trends, including errors due to missing visual cues, overfitting to explicit prompt structure, or misreading incident context.

Figure 6: Example where ChatTS does not utilize provided textual context, leading to an incorrect anomaly start classification.

Figure 7: GPT-4.1 (VLM) fails to detect a sustained anomaly (level shift) due to limited domain perceptual abilities.

Figure 8: Model (GPT-5) incorrectly reasons about inter-series causal relationships, likely due to insufficient integration of context and series descriptors.

Ablation Studies

Ablation experiments on context components (time series image, caption, or both) show that performance is significantly impaired when either is omitted, verifying the benchmark's reliance on genuinely multimodal and context-dependent reasoning. Model performance is not uniformly sensitive across architectures, further emphasizing the need for specialized integration methods.

Theoretical and Practical Implications

ARFBench fundamentally shifts the evaluation of time series reasoning models towards benchmarks grounded in realistic, multivariate, and context-rich incident data. The clear results establish that while modern VLMs and hybrid approaches are progressing rapidly, there remains a significant gap to human-level QA on incident data, particularly as question complexity increases.

On the practical side, the strong complementarity between human experts and automated models indicates that future incident response systems will benefit most from tightly integrated human-AI collaboration, where models provide perceptual and recall capabilities and humans contribute domain and process knowledge.

On the modeling front, the findings strongly motivate:

• Further development of hybrid TSFM-VLM frameworks that can efficiently scale to high-variate, long-horizon settings
• Multimodal learning techniques that make full use of rich context (e.g., natural language descriptions, channel tags, incident timelines)
• Benchmark expansion toward more open-ended, generative, and interactive agentic tasks in incident investigation

Conclusion

ARFBench establishes a rigorous, context-rich standard for TSQA in the software observability setting. It provides a challenging, carefully annotated dataset for benchmarking multimodal foundation models. The study reveals the present strengths and weaknesses of both generic and specialized models, demonstrates the upper bound achievable via model-expert composition, and charts clear directions for domain- and task-specific advances in automated incident analysis. ARFBench and potential successors represent necessary infrastructure for the development of safer, more reliable, and more efficient AI-augmented operational workflows in complex software systems.