Thunderbird Dataset: Log Analysis Benchmark
- Thunderbird dataset is a public system log corpus from a supercomputer, containing over 211 million entries and nearly 27,000 unique patterns.
- It serves as a critical benchmark for testing scalable log parsing, contextual anomaly detection, and models addressing severe class imbalance.
- Researchers leverage Thunderbird to evaluate methods including minhash-based LSH, LCS verification, PEFT with LoRA, and federated privacy techniques.
Searching arXiv for recent and canonical papers on the Thunderbird dataset in log analysis. arXiv search query: "Thunderbird dataset log analysis" The Thunderbird dataset is a publicly available supercomputer system log corpus that has become a recurrent benchmark in log parsing and log-based anomaly detection. Across studies, it is described as a log collection from the Thunderbird supercomputer at Sandia National Laboratories in Albuquerque, and it is also identified as a public supercomputer log dataset in the tools collection of Zhu et al., the Usenix CFDR repository, and GitHub/LogHub (Agrawal et al., 2019, Ocansey et al., 15 Jul 2025, Wang et al., 2024, Thompson et al., 21 Apr 2026). Its defining property is scale: multiple papers report about $211$ million log entries or lines and a raw size of $29.60$ GB, while parsing-oriented work emphasizes close to unique patterns and anomaly-detection work emphasizes severe class imbalance, temporal evolution, and heterogeneous deployment conditions (Agrawal et al., 2019, Wang et al., 2024).
1. Provenance and corpus identity
Thunderbird is consistently treated as a system-log dataset from a large supercomputing environment rather than as an application-specific trace. One study describes it as a “publicly available supercomputer system log dataset” from the tools collection of Zhu et al.; another describes it as an open-source log collection from the Thunderbird supercomputer at Sandia National Laboratories; a third places it among four distributed supercomputing system log datasets drawn from the Usenix CFDR repository (Agrawal et al., 2019, Ocansey et al., 15 Jul 2025, Wang et al., 2024).
The reported dataset scale is stable across these papers, although not numerically identical. Delog and LogTinyLLM report 211,212,192 lines or log entries and a total size of 29.60 GB (Agrawal et al., 2019, Ocansey et al., 15 Jul 2025). CroSysLog reports 211,212,174 total log events, spanning 2005–2006 over 244 days, with 34 unique anomaly labels (Wang et al., 2024). This small discrepancy in total count suggests that different preprocessing conventions or corpus snapshots may have been used, but the papers agree on the practical point that Thunderbird is a very large, real-world supercomputer log benchmark.
Two further infrastructure descriptors appear in later anomaly-detection studies. CroSysLog lists Thunderbird as a supercomputer system with 9,024 processors and 27,072 GB memory (Wang et al., 2024). DP-FLogTinyLLM describes it as coming from a 4,096-node Dell high-performance computer cluster at Sandia National Laboratories (Thompson et al., 21 Apr 2026). These descriptions are not presented as contradictory resolutions of the same metadata; they are simply the reported characterizations in those respective studies.
2. Scale, labels, and statistical profile
Thunderbird’s importance in AIOps and log mining derives primarily from its combination of volume, pattern diversity, and label structure. In Delog, the dataset is highlighted as a stress test for parsing because it contains 211,212,192 lines and close to 27,000 unique patterns, with the parser ultimately identifying 27,784 patterns on Thunderbird (Agrawal et al., 2019). The same study states that earlier distributed log parsing methods degrade as the number of unique patterns grows and that existing approaches “fail to parse in reasonable time” on Thunderbird.
For anomaly detection, the dataset is commonly used with a binary normal/abnormal convention derived from the raw log format. LogTinyLLM adopts the convention that entries that begin with “-” are normal, while entries that do not begin with “-” in the first column are abnormal (Ocansey et al., 15 Jul 2025). DP-FLogTinyLLM describes the first column similarly: a hyphen denotes non-alert messages, while other values denote alert messages, and the corpus contains alert and non-alert messages tagged by alert category (Thompson et al., 21 Apr 2026).
CroSysLog provides the most explicit event-level class statistics. Its Table 1 reports 193,744,715 normal log events (91.7%) and 17,467,459 abnormal log events (8.3%), again over 211,212,174 total events (Wang et al., 2024). The same paper emphasizes that Thunderbird is challenging because it combines low anomaly prevalence, clustered normal/anomalous events, and evolving logs over time. This makes Thunderbird suitable not only for static classification benchmarks but also for evaluating robustness to temporal drift and cross-system transfer.
3. Role in large-scale log parsing
In parsing research, Thunderbird functions as a benchmark that stresses both scalability and template fidelity. Delog uses it as one of the paper’s key large-scale datasets and describes its pipeline in three stages: preprocessing, LSH-based blocking, and block reduction (Agrawal et al., 2019).
The preprocessing stage splits each line on spaces, identifies “easy to identify variables” such as URLs and file paths, further splits remaining tokens on non-alphanumeric characters, and recognizes numbers, hexadecimal values, encoded strings, IP addresses, and dates without relying on manually defined regular expressions. After normalization, consecutive non-string tokens are replaced by wildcards, and duplicate processed lines are removed. The paper reports that preprocessing typically reduces most datasets to below 0.5% of the original line count, although no Thunderbird-specific post-preprocessing count is given (Agrawal et al., 2019).
The central scalability mechanism is minhash-based Locality Sensitive Hashing (LSH) over token-level shingles, not character shingles. Delog uses an empirically chosen Jaccard threshold to form candidate blocks and reports fixed settings of 2-gram shingles, 100 permutations, and LSH Jaccard threshold = 0.75 in the main experiments. The paper explicitly attributes Delog’s ability to handle Thunderbird to the resulting sub-linear behavior in the number of patterns (Agrawal et al., 2019).
Approximate LSH grouping is then corrected by exact Longest Common Subsequence (LCS) verification. For patterns , a sub-block must satisfy
with in the evaluation (Agrawal et al., 2019). This step is particularly important on Thunderbird because false grouping becomes costly when the pattern inventory is large.
Reduction is performed via an iterative Multiple Sequence Alignment variant based on Needleman–Wunsch pairwise alignment. Delog sorts patterns by length, aligns them longest-to-shortest, and then labels each alignment column constant or variable by majority logic: where is the mode frequency in column , is the number of patterns in the matrix, and $29.60$0 in the reported experiments (Agrawal et al., 2019). This design is motivated by the observation that many log variables span more than one token, so fixed token-position assumptions are inadequate.
Evaluation on Thunderbird uses Logan’s quality-loss metric rather than $29.60$1-measure, because manual labeling at Thunderbird’s scale is impractical. Delog reports quality loss = 0.0012 on Thunderbird, and the runtime table reports Thunderbird as the only dataset for which Delog has a runtime result among the compared parsers; the narrative conclusion is that Delog is the only parser in the comparison that can process Thunderbird in a respectable amount of time, whereas SHISO, Spell, Drain, IPLoM, and Logan do not finish in reasonable time (Agrawal et al., 2019). The paper also states that Thunderbird is not used in Delog’s privacy-preserving filtering experiments, which instead use a synthetic dataset.
4. Contextual anomaly detection with tiny LLMs
Later work repurposes Thunderbird from a parsing stress test into a benchmark for contextual log anomaly detection. LogTinyLLM treats the corpus as a supervised sequence-classification problem: raw logs are first parsed with Drain, which extracts log keys using a fixed-depth tree; the resulting keys are then organized into sequences by a sliding window technique so that models learn from context rather than isolated lines (Ocansey et al., 15 Jul 2025).
The downstream task is binary sequence classification. Given a log-key sequence $29.60$2, the model predicts a class probability
$29.60$3
and is trained with weighted cross-entropy loss to address class imbalance (Ocansey et al., 15 Jul 2025). The study evaluates four decoder-only models on Thunderbird: OPT-1.3B, Phi-1.5, TinyLlama-1.1B, and DeepSeek-R1-Distill-Qwen-1.5B.
The paper focuses on parameter-efficient fine-tuning (PEFT), especially LoRA and an adapter-based approach. For LoRA, the pretrained model is frozen and low-rank updates are injected into the attention projections: $29.60$4 with adaptation applied to the query, key, and value projections; mean pooling over hidden states is used before classification (Ocansey et al., 15 Jul 2025). The study reports that single-module key-projection adaptation is often the best trade-off between efficiency and performance.
On Thunderbird, the LoRA results are substantially stronger than the LogBERT baseline. Reported LoRA performance ranges from 97.76% to 98.83% accuracy and 97.98% to 98.57% F1. The best accuracy is 98.83% with DeepSeek-R1-Distill-Qwen-1.5B using key-projection-only LoRA; the best F1-score is 98.57% with the same backbone in the key+value configuration; and the lowest LoRA loss is 0.0328 with key+query+value adaptation (Ocansey et al., 15 Jul 2025). By contrast, LogBERT full fine-tuning is reported at 79.37% accuracy, 92.03% precision, 51.47% recall, and 66.02% F1.
The adapter-based results are weaker but still competitive. The strongest adapter configuration is DeepSeek-R1-Distill-Qwen-1.5B with 87.95% accuracy, 88.79% precision, 87.95% recall, and 0.8669 F1 (Ocansey et al., 15 Jul 2025). The paper interprets this as a trade-off between computational lightness and reduced predictive quality relative to LoRA.
5. Cross-system transfer and privacy-preserving federation
Thunderbird has also been used to evaluate whether anomaly detectors can generalize across systems or operate under privacy constraints. In CroSysLog, Thunderbird is used only as a target system in the main experiments, with Liberty + BGL as source systems (Wang et al., 2024). Logs are loaded with LogLead, sorted chronologically, stripped of null entries, and converted into “log events” by concatenating reporting component, severity level, and log message. Instead of template parsing, CroSysLog uses a NeuralLog-style neural representation: lowercase conversion, sensitive-variable replacement, removal of non-alphabetic characters, WordPiece tokenization, and BERT base embeddings with 12 encoder layers and 768 hidden dimensions, followed by mean pooling (Wang et al., 2024).
Thunderbird is split into 20 meta-testing tasks $29.60$5. Each task uses a 2,000-event support set and a 10,000-event query set, with unique non-overlapping chronological splits distributed across the 244-day collection period. The support sets contain 0.2–0.5% anomalous events and the query sets 0.01–0.5% anomalous events (Wang et al., 2024). Adaptation is based on first-order MAML: $29.60$6 In the main Thunderbird experiment, CroSysLog reports 96.99 precision, 98.17 recall, and 97.55 F1. Its reported efficiency is 6.5419 s training time and 0.6328 s testing time, which the paper identifies as the most efficient overall among the compared methods in that setting (Wang et al., 2024).
In DP-FLogTinyLLM, Thunderbird becomes a benchmark for federated, differentially private anomaly detection on non-IID data (Thompson et al., 21 Apr 2026). Raw logs are parsed with Drain, grouped into 5-minute windows with 1-minute step size and minimum 5 logs per window, and labeled anomalous if any entry in the window is anomalous. To simulate distributed deployment, Thunderbird is partitioned into 14 federated clients using Round Robin assignment by compute node. The experiment configuration specifies maximum samples = 4,500,000, minimum anomaly rate per node = 0.001, 10 communication rounds, 50% participation rate, and FedProx coefficient $29.60$7 (Thompson et al., 21 Apr 2026).
The model combines Tiny LLM backbones with LoRA, FedProx, and central differential privacy. The paper gives the LoRA form
$29.60$8
with Thunderbird settings rank $29.60$9, scaling factor 0, and dropout = 0.1 (Thompson et al., 21 Apr 2026). Privacy uses 1-DP with 2, 3, clipping bound 4, and noise multiplier 5. Training uses 10 local epochs, batch size 8, learning rate 6, weight decay 0.01, gradient accumulation steps 2, and max gradient norm 1.0 (Thompson et al., 21 Apr 2026).
The reported Thunderbird results are unusually strong. Final metrics are:
| Study setting | Thunderbird role | Representative outcome |
|---|---|---|
| Delog | distributed parsing stress test | 27,784 patterns; quality loss 0.0012 |
| LogTinyLLM | contextual sequence anomaly detection | LoRA accuracy 97.76%–98.83%; F1 97.98%–98.57% |
| CroSysLog | cross-system target for meta-learning | F1 97.55% |
| DP-FLogTinyLLM | federated differentially private anomaly detection | OPT-1.3B F1 0.9935; precision 0.9996 |
Within DP-FLogTinyLLM, the main table reports 0.9992 F1 for both DeepSeek-R1 and OPT-1.3B in one summary, while the federated-comparison table reports average F1 = 0.9935 and precision = 0.9996 for OPT-1.3B, which the paper identifies as the strongest overall Thunderbird model under FlogTinyLLM (Thompson et al., 21 Apr 2026). The same study reports much higher precision than earlier federated baselines such as Federated LogBERT, LogDeep(FL), and Federated LogGPT, while noting substantial training-time overhead relative to centralized LogTinyLLM: 5.99× for Phi-1.5, 9.81× for DeepSeek-R1, 11.42× for OPT-1.3B, and 15.91× for TinyLlama (Thompson et al., 21 Apr 2026).
6. Naming ambiguities and scope boundaries
“Thunderbird dataset” should not be conflated with THUNDER or with unrelated bird datasets. The paper “THUNDER: Tile-level Histopathology image UNDERstanding benchmark” explicitly states that THUNDER is not a dataset named Thunderbird; it is an open-source benchmark/evaluation suite for tile-level histopathology image understanding built from 16 public datasets and 23 foundation models (Marza et al., 10 Jul 2025). Its domain, data modality, and tasks—classification, segmentation, calibration, adversarial robustness, feature-space alignment, retrieval, and transformation invariance—are unrelated to system-log analytics.
Likewise, “Real bird dataset with imprecise and uncertain values” is a crowdsourced bird-photo annotation corpus for belief-function research, not a system-log dataset (Thierry et al., 2022). It contains campaigns such as multi_birds_precise, multi_birds_imprecise, and 10_birds_iterative, with uncertainty and imprecision explicitly modeled through contributor-selected label sets and certainty ratings. Its relevance is bibliographic rather than substantive: the lexical similarity between “Thunderbird,” “THUNDER,” and “bird” datasets can produce search ambiguity, but only the log corpus described above is the Thunderbird dataset used in log parsing and anomaly-detection research.
Taken together, the literature positions Thunderbird as a high-scale, high-diversity, operationally realistic log benchmark whose main value lies in exposing failure modes of parsers that assume modest pattern counts and of anomaly detectors that assume centralized, IID, or temporally stationary data. Delog uses it to demonstrate scalable parsing under close to 7 patterns; LogTinyLLM uses it to show strong contextual anomaly detection under PEFT; CroSysLog uses it as a chronologically evolving cross-system target; and DP-FLogTinyLLM uses it to test non-IID federated anomaly detection under differential privacy (Agrawal et al., 2019, Ocansey et al., 15 Jul 2025, Wang et al., 2024, Thompson et al., 21 Apr 2026).