CloudAnoBench: Multimodal Cloud Anomaly Benchmark
- CloudAnoBench is a multimodal benchmark combining multivariate time-series metrics with temporally aligned terminal logs to enable precise cloud anomaly detection.
- It features 49 curated cases with detailed anomaly, type, and difficulty labels, facilitating evaluation of LLM-based and neuro-symbolic diagnostic methods.
- The benchmark addresses gaps in metrics-only datasets by incorporating fine-grained anomaly semantics and realistic synthetic log generation for robust evaluation.
CloudAnoBench is a multimodal benchmark for cloud anomaly detection introduced together with CloudAnoAgent. It is defined at the case level and, for each case, provides multivariate time-series metrics, temporally aligned terminal-style log text, and fine-grained anomaly behavior annotations, specifically a binary anomaly label, an anomaly type label, and a difficulty label. Its stated purpose is to support systematic evaluation of systems that jointly reason over metrics and logs, especially LLM-based and neuro-symbolic methods, while addressing the absence of paired, temporally aligned metrics-and-logs datasets with explicit anomaly semantics in prior cloud anomaly corpora (Zou et al., 3 Aug 2025).
1. Origins and conceptual positioning
CloudAnoBench was introduced to fill two gaps identified in prior cloud anomaly datasets: many widely used datasets are metrics-only and therefore unsuitable for evaluating log-aware methods, and existing datasets often provide coarse or missing annotations, such as the absence of precise anomaly intervals, anomaly types, or difficult normal cases whose metrics look anomalous but whose logs reveal benign causes (Zou et al., 3 Aug 2025). In that sense, the benchmark is designed not merely as a repository of traces but as an evaluation substrate for multimodal reasoning.
The benchmark is positioned against several adjacent traditions in cloud benchmarking. MalStone focuses on analytics on large data clouds through Subsequent Proportion of Marks over 10 billion to 1 trillion 100-byte records, emphasizing wall-clock performance of middleware stacks rather than anomaly semantics (Bennett et al., 2010). Cloud WorkBench emphasizes Infrastructure-as-Code, scheduling, and reproducible execution of cloud benchmarks on IaaS platforms (Scheuner et al., 2014). CLAMBS targets cross-layer, multi-cloud QoS monitoring and benchmarking of distributed application components (Alhamazani et al., 2015). ExaBench targets continuous application-level performance evaluation for scientific workloads such as HPL, VASP, and GROMACS (Mohammadi et al., 2018). FaaSdom specializes in serverless benchmarking across AWS, Azure, Google, and IBM, with latency, scaling, and cost analyses (Maissen et al., 2020). Relative to these efforts, CloudAnoBench is narrower in scale but more explicit in multimodal anomaly semantics and type-level diagnostic structure (Zou et al., 3 Aug 2025).
A recurrent source of ambiguity is that some cloud benchmarking papers use “CloudAnoBench” as a hypothetical or framework-like label for anomaly-aware or reliable cloud benchmarking, rather than as this specific benchmark artifact. For example, application-level cloud benchmarking work around Black Friday discusses implications “for a framework like ‘CloudAnoBench’,” while a heterogeneous-cloud benchmarking report explicitly states that it does not define a benchmark with that specific name (Henning et al., 14 Oct 2025, Duggi et al., 10 Jan 2025). This suggests two coexisting usages: a concrete multimodal anomaly benchmark introduced in 2025, and a broader editorial shorthand for anomaly-aware cloud benchmarking design.
2. Dataset structure, modalities, and case semantics
Each CloudAnoBench case combines a metrics window and a log window covering the same interval. The metric side includes multivariate time series over CPU usage, GPU usage, memory usage, disk I/O, and network throughput. The benchmark consolidates five canonical anomaly patterns for time-series behavior: Spike, Dip, Gradual Increase, Gradual Decrease, and Fluctuation (Zou et al., 3 Aug 2025).
The log side consists of terminal-style log outputs representing events such as job submissions, CRON tasks, package downloads, network connection attempts, system errors including OOM kills, security events including port scans, ARP spoofing, and DoS behavior, and storage-related behavior such as backups or log growth. The logs are intentionally constrained so that they do not restate metric values; lines such as “CPU over 70%” are excluded to preserve cross-modal independence. They also include benign, unrelated entries such as SSH logins and standard scheduled task messages, which create realistic diagnostic clutter (Zou et al., 3 Aug 2025).
CloudAnoBench is case-based rather than timestamp-labeled. It contains 49 total cases, of which 19 are anomaly cases and 30 are normal cases. Labels are assigned at the case level: anomaly versus normal, anomaly type, and difficulty level. Difficulty has two levels, easy and difficult. Easy cases use shorter windows, single-metric anomalies, and concise log traces; difficult cases use longer windows, multiple metrics, and heavier log noise (Zou et al., 3 Aug 2025).
The benchmark covers 10 representative anomaly types:
| Anomaly type | Typical metrics behavior | Typical log semantics |
|---|---|---|
mine |
CPU spike | xmrig download and execution via CRON |
oom |
memory gradual rise | GC failure and OOM kill logs |
gpu_hijack |
GPU usage spike | deep learning job from unknown container |
port_scan |
network fluctuation | repeated port connection attempts |
icmp_flood_dos |
spike in incoming network traffic | ICMP echo request bursts |
dns_amplification |
spike in outgoing network traffic | excessive DNS queries to open resolvers |
data_exfiltration |
gradual increase in outbound throughput | outbound scp or curl-like transfer indicators |
arp_spoofing |
intermittent network instability | ARP reply storms |
log_storm |
disk I/O spike | burst of crawler logs from unknown IPs |
log_growth_anomaly |
gradual disk space or I/O growth | large scheduled backup writes |
A notable design choice is that normal cases are deliberately nontrivial. Many exhibit metrics that appear anomalous in isolation, while logs indicate benign explanations such as legitimate deep learning jobs or scheduled backups. This directly targets false-positive-prone behavior in metric-only anomaly detectors (Zou et al., 3 Aug 2025).
3. Construction process and annotation model
The metric sequences are generated using automated top-based scripts and then perturbed through constrained randomization within realistic anomaly-specific ranges. The paper does not specify exact sampling distributions or a normalization pipeline, but the stated intent is to avoid trivial synthetic signals and to approximate realistic operational noise (Zou et al., 3 Aug 2025).
For each metric sequence, the corresponding logs are LLM-generated. They are conditioned on scenario semantics such as anomaly type, benign-versus-anomalous interpretation, and difficulty level, and are constructed to be temporally aligned with the metric window. The benchmark is therefore synthetic in two distinct senses: the metrics are simulated with controlled noise, and the logs are generated by LLMs to emulate realistic terminal output (Zou et al., 3 Aug 2025).
Annotation is deterministic by construction rather than post hoc human labeling over naturally collected traces. Designers select an intended incident scenario, generate metrics and logs consistent with that scenario, and assign the case-level labels anomaly, anomaly type, and difficulty. The anomaly type label functions as a coarse root-cause category, but the benchmark does not provide a separate causal graph, per-event labels, or per-timestamp onset/offset annotations (Zou et al., 3 Aug 2025).
This design makes the benchmark especially suitable for evaluation of reasoning pipelines that must disambiguate similar metric signatures using textual evidence. It also means that CloudAnoBench is not a large, naturally occurring telemetry corpus in the sense of production AIOps datasets. That distinction is essential for correct interpretation.
4. Tasks, metrics, and evaluation protocol
CloudAnoBench supports at least two explicit supervised evaluation tasks. The first is binary anomaly detection, where a method receives the metrics sequence and aligned log text for a case and outputs a binary anomaly label. The corresponding metric is Anomaly Classification Accuracy, or ACA, defined in the standard accuracy form over all cases (Zou et al., 3 Aug 2025):
The second is anomaly type classification, where the output is one of the 10 anomaly types, or a normal/no-anomaly output for normal cases. The metric is Anomaly Type Classification Accuracy, or ATCA (Zou et al., 3 Aug 2025):
The paper also discusses false positive rate through performance on normal cases, though it does not formalize the FPR equation in the main benchmark definition. Precision, recall, F1, and AUROC are not reported; the benchmark uses accuracy-oriented evaluation because it is small, case-based, and centered on anomaly-versus-normal and type-level correctness rather than threshold sweeps (Zou et al., 3 Aug 2025).
The evaluation protocol is case-wise over the full set of 49 cases. The paper does not specify a train/validation/test split. This is consistent with the fact that many compared baselines are rule-based, unsupervised, or prompt-based LLM systems rather than models trained on CloudAnoBench itself. For LLM-backed evaluations, every evaluation step is repeated three times per LLM and average results are reported to mitigate stochasticity (Zou et al., 3 Aug 2025).
Baseline families effectively induce benchmark tracks. Traditional baselines include RuleEnsembleAD, Isolation Forest, Decision Tree, Logistic Regression, KMeans, RarityModel, and OOV Detector. A separate LLM baseline uses a single prompt over metrics and logs. CloudAnoAgent is then evaluated both with and without symbolic verification, allowing direct measurement of the contribution of neuro-symbolic decomposition (Zou et al., 3 Aug 2025).
5. CloudAnoBench as the evaluation substrate for CloudAnoAgent
CloudAnoBench was introduced together with CloudAnoAgent, a neuro-symbolic LLM-based anomaly detection agent for cloud environments. In the reported pipeline, Fast Detection analyzes metrics-only sliding windows and assigns a potential anomaly status and one of the five metric-pattern categories. Slow Detection then processes temporally aligned logs for candidate windows, assigns an anomaly possibility level, proposes an anomaly type from the 10 benchmark categories, and produces a natural-language explanation. A Decision-maker integrates the metric and log signals, and a Symbolic Verifier checks metric-pattern consistency and log-pattern consistency for the predicted type before either accepting the decision or triggering reflective re-evaluation (Zou et al., 3 Aug 2025).
This architecture is tightly coupled to CloudAnoBench’s annotation scheme. The verifier depends on the benchmark’s explicit anomaly taxonomy and on the characteristic metric/log signatures associated with each anomaly type. CloudAnoBench is therefore not just an input corpus but an evaluative environment structured around multimodal disambiguation, type inference, and interpretability (Zou et al., 3 Aug 2025).
On this benchmark, the paper reports that CloudAnoAgent improves anomaly classification accuracy by 46.36% and 36.67% on average over traditional baselines and the LLM-only baseline, reduces the false positive rate by 36.67% and 33.89% on average over those two baseline families, and improves anomaly type detection accuracy by 12.8% compared to vanilla LLM prompting (Zou et al., 3 Aug 2025). With symbolic verification enabled, overall ACA ranges from 95.24% to 98.64%, and overall ATCA ranges from 91.84% to 97.96% across the evaluated LLM backends. The symbolic verifier alone reduces FPR by 17.04%, improves ATCA by 4.97%, and increases overall ACA by 10.65% relative to CloudAnoAgent without verification (Zou et al., 3 Aug 2025).
The benchmark therefore operationalizes a specific claim: in cloud anomaly diagnosis, log-aware multimodal reasoning and explicit symbolic consistency checks can materially outperform both metric-only heuristics and monolithic prompting. CloudAnoBench makes that claim testable because it includes normal cases that look anomalous in the metrics but are rendered benign by the logs.
6. Interpretive significance, limitations, and likely extensions
CloudAnoBench occupies a distinctive niche in cloud benchmarking. It is much smaller than traditional performance suites and lacks the large-scale systems focus of benchmarks such as MalStone or ExaBench, but it provides an explicit multimodal structure that many cloud-performance and QoS benchmarks do not attempt (Bennett et al., 2010, Mohammadi et al., 2018). This makes it particularly relevant to LLM-era AIOps research, where the central problem is often not merely detecting a performance deviation but interpreting whether an apparent deviation is anomalous, benign, malicious, or operationally expected.
A common misconception would be to treat CloudAnoBench as a production-trace benchmark comparable to large industrial telemetry corpora. The benchmark itself does not support that interpretation. Its metrics are generated via scripts and constrained randomization, its logs are LLM-generated, its total size is 49 cases, and it lacks per-timestamp localization, explicit causal graphs, and quantitative explanation metrics (Zou et al., 3 Aug 2025). These are not incidental omissions; they define the current scope of the artifact.
Another misconception would be to equate CloudAnoBench with general cloud benchmarking. Benchmarks such as CLAMBS, Cloud WorkBench, FaaSdom, and the SPEC-oriented cloud-metrics framework address cross-layer QoS monitoring, Infrastructure-as-Code reproducibility, serverless benchmarking, or elasticity/isolation/availability/risk measurement, respectively (Alhamazani et al., 2015, Scheuner et al., 2014, Maissen et al., 2020, Herbst et al., 2016). CloudAnoBench does not replace those traditions. Rather, it introduces a benchmark layer centered on multimodal anomaly semantics. A plausible implication is that future cloud benchmark suites may combine these perspectives: reproducible orchestration, longitudinal measurement, SLA-oriented metrics, and case-level anomaly interpretation.
The most direct extension path is already suggested by adjacent work on temporal cloud benchmarking. Application-level benchmarking on AWS EKS over months found moderate but non-negligible variability, a coefficient of variation of 3.69%, daily effects up to 2.15%, weekly effects up to 2.52%, and small Black Friday deviations that were not catastrophic (Henning et al., 14 Oct 2025). This suggests that future versions of CloudAnoBench could move beyond isolated case windows toward event-aware and time-aware anomaly benchmarks, where normality is conditioned not only on logs but also on hour-of-day, day-of-week, and special-event context. That extension is not part of the current benchmark, but it is consistent with the broader anomaly-aware benchmarking agenda.
In its present form, CloudAnoBench is best understood as a first-of-its-kind multimodal, scenario-based benchmark for cloud anomaly detection: small, curated, strongly structured, and explicitly optimized for evaluating whether a system can combine metrics and logs to reduce false positives and infer anomaly types with interpretable reasoning (Zou et al., 3 Aug 2025).