---
title: 'Thunderbird Dataset: Log Analysis Benchmark'
url: https://www.emergentmind.com/topics/thunderbird-dataset
type: topic
---

# Thunderbird Dataset: Log Analysis Benchmark

Searching arXiv for recent and canonical papers on the Thunderbird dataset in log analysis.
arXiv search query: "Thunderbird dataset log analysis"
The **Thunderbird dataset** is a publicly available supercomputer system log corpus that has become a recurrent benchmark in log parsing and log-based anomaly detection. Across studies, it is described as a log collection from the Thunderbird supercomputer at Sandia National Laboratories in Albuquerque, and it is also identified as a public supercomputer log dataset in the tools collection of Zhu et al., the Usenix CFDR repository, and GitHub/LogHub [1902.04843, 2507.11071, 2412.15445, 2604.19118]. Its defining property is scale: multiple papers report about \(211\) million log entries or lines and a raw size of \(29.60\) GB, while parsing-oriented work emphasizes close to \(27{,}000\) unique patterns and anomaly-detection work emphasizes severe class imbalance, temporal evolution, and heterogeneous deployment conditions [1902.04843, 2412.15445].

## 1. Provenance and corpus identity

Thunderbird is consistently treated as a **system-log dataset** from a large supercomputing environment rather than as an application-specific trace. One study describes it as a “publicly available supercomputer system log dataset” from the tools collection of Zhu et al.; another describes it as an open-source log collection from the Thunderbird supercomputer at Sandia National Laboratories; a third places it among four distributed supercomputing system log datasets drawn from the Usenix CFDR repository [1902.04843, 2507.11071, 2412.15445].

The reported dataset scale is stable across these papers, although not numerically identical. Delog and LogTinyLLM report **211,212,192** lines or log entries and a total size of **29.60 GB** [1902.04843, 2507.11071]. CroSysLog reports **211,212,174** total log events, spanning **2005–2006** over **244 days**, with **34 unique anomaly labels** [2412.15445]. This small discrepancy in total count suggests that different preprocessing conventions or corpus snapshots may have been used, but the papers agree on the practical point that Thunderbird is a very large, real-world supercomputer log benchmark.

Two further infrastructure descriptors appear in later anomaly-detection studies. CroSysLog lists Thunderbird as a supercomputer system with **9,024 processors** and **27,072 GB memory** [2412.15445]. DP-FLogTinyLLM describes it as coming from a **4,096-node Dell high-performance computer cluster** at Sandia National Laboratories [2604.19118]. These descriptions are not presented as contradictory resolutions of the same metadata; they are simply the reported characterizations in those respective studies.

## 2. Scale, labels, and statistical profile

Thunderbird’s importance in AIOps and log mining derives primarily from its combination of **volume**, **pattern diversity**, and **label structure**. In Delog, the dataset is highlighted as a stress test for parsing because it contains **211,212,192 lines** and close to **27,000 unique patterns**, with the parser ultimately identifying **27,784 patterns** on Thunderbird [1902.04843]. The same study states that earlier distributed log parsing methods degrade as the number of unique patterns grows and that existing approaches “fail to parse in reasonable time” on Thunderbird.

For anomaly detection, the dataset is commonly used with a binary normal/abnormal convention derived from the raw log format. LogTinyLLM adopts the convention that entries that **begin with “-” are normal**, while entries that **do not begin with “-” in the first column are abnormal** [2507.11071]. DP-FLogTinyLLM describes the first column similarly: a hyphen denotes **non-alert messages**, while other values denote **alert messages**, and the corpus contains alert and non-alert messages tagged by alert category [2604.19118].

CroSysLog provides the most explicit event-level class statistics. Its Table 1 reports **193,744,715 normal log events (91.7%)** and **17,467,459 abnormal log events (8.3%)**, again over **211,212,174** total events [2412.15445]. The same paper emphasizes that Thunderbird is challenging because it combines **low anomaly prevalence**, **clustered normal/anomalous events**, and **evolving logs over time**. This makes Thunderbird suitable not only for static classification benchmarks but also for evaluating robustness to temporal drift and cross-system transfer.

## 3. Role in large-scale log parsing

In parsing research, Thunderbird functions as a benchmark that stresses both **scalability** and **template fidelity**. Delog uses it as one of the paper’s key large-scale datasets and describes its pipeline in three stages: **preprocessing**, **LSH-based blocking**, and **block reduction** [1902.04843].

The preprocessing stage splits each line on spaces, identifies “easy to identify variables” such as URLs and file paths, further splits remaining tokens on non-alphanumeric characters, and recognizes numbers, hexadecimal values, encoded strings, IP addresses, and dates without relying on manually defined regular expressions. After normalization, consecutive non-string tokens are replaced by wildcards, and duplicate processed lines are removed. The paper reports that preprocessing typically reduces most datasets to below **0.5%** of the original line count, although no Thunderbird-specific post-preprocessing count is given [1902.04843].

The central scalability mechanism is **minhash-based Locality Sensitive Hashing (LSH)** over **token-level shingles**, not character shingles. Delog uses an empirically chosen Jaccard threshold to form candidate blocks and reports fixed settings of **2-gram shingles**, **100 permutations**, and **LSH Jaccard threshold = 0.75** in the main experiments. The paper explicitly attributes Delog’s ability to handle Thunderbird to the resulting **sub-linear behavior in the number of patterns** [1902.04843].

Approximate LSH grouping is then corrected by exact **Longest Common Subsequence (LCS)** verification. For patterns \(P_1, P_2, \ldots, P_n\), a sub-block must satisfy
\[
\text{LCS}(P, Q).\text{length} - \alpha * \max(P.\text{length}, Q.\text{length}) \ge 0
\]
with \(\alpha = 0.65\) in the evaluation [1902.04843]. This step is particularly important on Thunderbird because false grouping becomes costly when the pattern inventory is large.

Reduction is performed via an iterative **Multiple Sequence Alignment** variant based on **Needleman–Wunsch** pairwise alignment. Delog sorts patterns by length, aligns them longest-to-shortest, and then labels each alignment column constant or variable by majority logic:
\[
f_j - \beta * n \ge 0
\]
where \(f_j\) is the mode frequency in column \(C_j\), \(n\) is the number of patterns in the matrix, and \(\beta = 0.7\) in the reported experiments [1902.04843]. This design is motivated by the observation that many log variables span more than one token, so fixed token-position assumptions are inadequate.

Evaluation on Thunderbird uses **Logan’s quality-loss metric** rather than \(F\)-measure, because manual labeling at Thunderbird’s scale is impractical. Delog reports **quality loss = 0.0012** on Thunderbird, and the runtime table reports Thunderbird as the only dataset for which Delog has a runtime result among the compared parsers; the narrative conclusion is that Delog is the only parser in the comparison that can process Thunderbird in a respectable amount of time, whereas SHISO, Spell, Drain, IPLoM, and Logan do not finish in reasonable time [1902.04843]. The paper also states that Thunderbird is not used in Delog’s privacy-preserving filtering experiments, which instead use a synthetic dataset.

## 4. Contextual anomaly detection with tiny LLMs

Later work repurposes Thunderbird from a parsing stress test into a benchmark for **contextual log anomaly detection**. LogTinyLLM treats the corpus as a supervised sequence-classification problem: raw logs are first parsed with **Drain**, which extracts **log keys** using a fixed-depth tree; the resulting keys are then organized into sequences by a **sliding window technique** so that models learn from context rather than isolated lines [2507.11071].

The downstream task is binary sequence classification. Given a log-key sequence \(k_i\), the model predicts a class probability
\[
P(y_i = c \mid k_i) = \frac{\exp(\text{logits}_{i,c})}{\sum_{j=0}^{1} \exp(\text{logits}_{i,j})}
\]
and is trained with **weighted cross-entropy loss** to address class imbalance [2507.11071]. The study evaluates four decoder-only models on Thunderbird: **OPT-1.3B**, **Phi-1.5**, **TinyLlama-1.1B**, and **DeepSeek-R1-Distill-Qwen-1.5B**.

The paper focuses on **parameter-efficient fine-tuning (PEFT)**, especially **LoRA** and an adapter-based approach. For LoRA, the pretrained model is frozen and low-rank updates are injected into the attention projections:
\[
W^l_{\text{adapted}} = W^l + \Delta W^l = W^l + \alpha \cdot B^l A^l
\]
with adaptation applied to the **query, key, and value projections**; mean pooling over hidden states is used before classification [2507.11071]. The study reports that **single-module key-projection adaptation** is often the best trade-off between efficiency and performance.

On Thunderbird, the LoRA results are substantially stronger than the LogBERT baseline. Reported LoRA performance ranges from **97.76% to 98.83% accuracy** and **97.98% to 98.57% F1**. The **best accuracy** is **98.83%** with **DeepSeek-R1-Distill-Qwen-1.5B** using **key-projection-only LoRA**; the **best F1-score** is **98.57%** with the same backbone in the **key+value** configuration; and the **lowest LoRA loss** is **0.0328** with **key+query+value** adaptation [2507.11071]. By contrast, **LogBERT full fine-tuning** is reported at **79.37% accuracy**, **92.03% precision**, **51.47% recall**, and **66.02% F1**.

The adapter-based results are weaker but still competitive. The strongest adapter configuration is **DeepSeek-R1-Distill-Qwen-1.5B** with **87.95% accuracy**, **88.79% precision**, **87.95% recall**, and **0.8669 F1** [2507.11071]. The paper interprets this as a trade-off between computational lightness and reduced predictive quality relative to LoRA.

## 5. Cross-system transfer and privacy-preserving federation

Thunderbird has also been used to evaluate whether anomaly detectors can generalize across systems or operate under privacy constraints. In **CroSysLog**, Thunderbird is used **only as a target system** in the main experiments, with **Liberty + BGL** as source systems [2412.15445]. Logs are loaded with **LogLead**, sorted chronologically, stripped of null entries, and converted into “log events” by concatenating reporting component, severity level, and log message. Instead of template parsing, CroSysLog uses a **NeuralLog-style neural representation**: lowercase conversion, sensitive-variable replacement, removal of non-alphabetic characters, **WordPiece** tokenization, and **BERT base** embeddings with **12 encoder layers** and **768 hidden dimensions**, followed by mean pooling [2412.15445].

Thunderbird is split into **20 meta-testing tasks** \(T_{ts_j}=(S_{ts_j}, Q_{ts_j})\). Each task uses a **2,000-event support set** and a **10,000-event query set**, with unique non-overlapping chronological splits distributed across the **244-day** collection period. The support sets contain **0.2–0.5% anomalous** events and the query sets **0.01–0.5% anomalous** events [2412.15445]. Adaptation is based on first-order MAML:
\[
\theta'_i = \theta - \alpha \nabla_{\theta} \mathcal{L}_{S_i}(\theta).
\]
In the main Thunderbird experiment, CroSysLog reports **96.99 precision**, **98.17 recall**, and **97.55 F1**. Its reported efficiency is **6.5419 s** training time and **0.6328 s** testing time, which the paper identifies as the most efficient overall among the compared methods in that setting [2412.15445].

In **DP-FLogTinyLLM**, Thunderbird becomes a benchmark for **federated**, **differentially private** anomaly detection on non-IID data [2604.19118]. Raw logs are parsed with **Drain**, grouped into **5-minute windows** with **1-minute step size** and **minimum 5 logs per window**, and labeled anomalous if any entry in the window is anomalous. To simulate distributed deployment, Thunderbird is partitioned into **14 federated clients** using **Round Robin assignment by compute node**. The experiment configuration specifies **maximum samples = 4,500,000**, **minimum anomaly rate per node = 0.001**, **10 communication rounds**, **50% participation rate**, and **FedProx coefficient \(\mu = 0.01\)** [2604.19118].

The model combines Tiny LLM backbones with LoRA, FedProx, and central differential privacy. The paper gives the LoRA form
\[
W_{\text{adapted}} = W + \frac{\alpha}{r}BA
\]
with Thunderbird settings **rank \(r=8\)**, **scaling factor \(\alpha=32\)**, and **dropout = 0.1** [2604.19118]. Privacy uses \((\varepsilon,\delta)\)-DP with **\(\varepsilon = 10.0\)**, **\(\delta = 10^{-5}\)**, **clipping bound \(C=1.0\)**, and **noise multiplier \(\sigma = 0.01\)**. Training uses **10 local epochs**, **batch size 8**, **learning rate \(2\times 10^{-5}\)**, **weight decay 0.01**, **gradient accumulation steps 2**, and **max gradient norm 1.0** [2604.19118].

The reported Thunderbird results are unusually strong. Final metrics are:

| Study setting | Thunderbird role | Representative outcome |
|---|---|---|
| Delog | distributed parsing stress test | 27,784 patterns; quality loss 0.0012 |
| LogTinyLLM | contextual sequence anomaly detection | LoRA accuracy 97.76%–98.83%; F1 97.98%–98.57% |
| CroSysLog | cross-system target for meta-learning | F1 97.55% |
| DP-FLogTinyLLM | federated differentially private anomaly detection | OPT-1.3B F1 0.9935; precision 0.9996 |

Within DP-FLogTinyLLM, the main table reports **0.9992 F1** for both **DeepSeek-R1** and **OPT-1.3B** in one summary, while the federated-comparison table reports average **F1 = 0.9935** and **precision = 0.9996** for **OPT-1.3B**, which the paper identifies as the strongest overall Thunderbird model under FlogTinyLLM [2604.19118]. The same study reports much higher precision than earlier federated baselines such as Federated LogBERT, LogDeep(FL), and Federated LogGPT, while noting substantial training-time overhead relative to centralized LogTinyLLM: **5.99×** for Phi-1.5, **9.81×** for DeepSeek-R1, **11.42×** for OPT-1.3B, and **15.91×** for TinyLlama [2604.19118].

## 6. Naming ambiguities and scope boundaries

“Thunderbird dataset” should not be conflated with **THUNDER** or with unrelated **bird** datasets. The paper titled “THUNDER: Tile-level Histopathology image UNDERstanding benchmark” explicitly states that **THUNDER is not a dataset named Thunderbird**; it is an open-source **benchmark/evaluation suite** for tile-level histopathology image understanding built from **16 public datasets** and **23 foundation models** [2507.07860]. Its domain, data modality, and tasks—classification, segmentation, calibration, adversarial robustness, feature-space alignment, retrieval, and transformation invariance—are unrelated to system-log analytics.

Likewise, “Real bird dataset with imprecise and uncertain values” is a crowdsourced **bird-photo annotation** corpus for belief-function research, not a system-log dataset [2211.11809]. It contains campaigns such as `multi_birds_precise`, `multi_birds_imprecise`, and `10_birds_iterative`, with uncertainty and imprecision explicitly modeled through contributor-selected label sets and certainty ratings. Its relevance is bibliographic rather than substantive: the lexical similarity between “Thunderbird,” “THUNDER,” and “bird” datasets can produce search ambiguity, but only the log corpus described above is the Thunderbird dataset used in log parsing and anomaly-detection research.

Taken together, the literature positions Thunderbird as a high-scale, high-diversity, operationally realistic log benchmark whose main value lies in exposing failure modes of parsers that assume modest pattern counts and of anomaly detectors that assume centralized, IID, or temporally stationary data. Delog uses it to demonstrate scalable parsing under close to \(27{,}000\) patterns; LogTinyLLM uses it to show strong contextual anomaly detection under PEFT; CroSysLog uses it as a chronologically evolving cross-system target; and DP-FLogTinyLLM uses it to test non-IID federated anomaly detection under differential privacy [1902.04843, 2507.11071, 2412.15445, 2604.19118].

Source: https://www.emergentmind.com/topics/thunderbird-dataset