---
title: CASTLE Microbenchmark Suite
url: https://www.emergentmind.com/topics/castle-microbenchmark-suite
type: topic
---

# CASTLE Microbenchmark Suite

Searching arXiv for the specified CASTLE papers to ground the article and verify bibliographic details.
{"query":"arXiv 2503.09433 CASTLE Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection", "max_results": 5}
arxiv_search({"query":"2503.09433", "max_results": 5})
The name **CASTLE** has been used for two distinct security-oriented microbenchmark suites in arXiv literature. In "CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection," CASTLE expands to **CWE Automated Security Testing and Low-Level Evaluation** and denotes a benchmarking framework for vulnerability detection in source code, built around 250 hand-crafted C micro-benchmarks and a unified **CASTLE Score** [2503.09433]. In "A Benchmark Suite for Evaluating Caches’ Vulnerability to Timing Attacks," CASTLE denotes a benchmark suite for empirically testing timing side/covert channels in processor caches, built around 1,094 automatically generated C test programs and the **Cache Timing Vulnerability Score (CTVS)** [1911.08619]. Both suites are microbenchmark-based and score-driven, but they target different layers of the security stack: one evaluates program-analysis methods against CWE-labeled code, while the other evaluates cache implementations against timing-attack classes.

## 1. Terminological scope and shared benchmarking logic

The two CASTLE suites share a methodological pattern: both reduce complex security phenomena to small, repeatable tests, attach explicit decision criteria to each test, and aggregate outcomes into a unified score. The 2025 CASTLE targets source-code vulnerability detection across **traditional static analyzers (SAST)**, **formal verification (FV)** tools, and **Large Language Models (LLMs)**. The 2019 CASTLE targets **timing-based cache vulnerabilities** in processor caches under a three-step attack model. This suggests a common design principle: microbenchmarks are used to isolate detection capability while minimizing ambiguity in ground truth or attack interpretation [2503.09433] [1911.08619].

| Aspect | CASTLE for CWE detection | CASTLE for cache timing attacks |
|---|---|---|
| Primary target | Source-code vulnerability detection | Cache timing-based vulnerabilities |
| Artifact size | 250 micro-benchmark programs | 1,094 test programs |
| Implementation language | C | Auto-generated C |
| Core score | CASTLE Score | Cache Timing Vulnerability Score |
| Unit of analysis | Single-file programs with one vulnerability or none | Three-step timing-vulnerability variants |

## 2. CASTLE for CWE detection: dataset design and benchmark construction

The 2025 CASTLE suite was created to address shortcomings in prior datasets, including lack of compilable programs, imbalanced CWE coverage, and coarse labeling. It provides **250 micro-benchmark programs written in C**, covering **25 CWEs**, with **exactly 10 cases per CWE (6 vulnerable, 4 non-vulnerable)**. All programs are **single-file C sources**, **compilable without errors (warnings allowed)**, and each case contains **a single, clearly identified CWE or is explicitly marked as non-vulnerable**. Code size ranges from **7–164 lines per file (average 42)**, with **1–8 functions (average 2.2)**, **cyclomatic complexity 1–29 (mean 6.3)**, and **Halstead volume ~89.9–5,246.7 (average 1,104.8)**. Including the LLM system prompt, the total input across the dataset is **~115,620 tokens (cl100k_base)** [2503.09433].

The covered CWEs are: **CWE-22**, **CWE-78**, **CWE-89**, **CWE-125**, **CWE-134**, **CWE-190**, **CWE-253**, **CWE-327**, **CWE-362**, **CWE-369**, **CWE-401**, **CWE-415**, **CWE-416**, **CWE-476**, **CWE-522**, **CWE-617**, **CWE-628**, **CWE-674**, **CWE-761**, **CWE-770**, **CWE-787**, **CWE-798**, **CWE-822**, **CWE-835**, and **CWE-843**. The suite is therefore balanced at the per-CWE level and is explicitly structured to support robust measurement of **true positives**, **false positives**, **false negatives**, and **true negatives**.

Each file embeds a **YAML-like metadata header** containing dataset name, filename, version, compile command, vulnerability presence flag, short description, and CWE number. Vulnerable lines are annotated with **`// {!LINE}`**, enabling **line-level ground truth**. During preprocessing, metadata is stripped and files are converted to a **JSON record** with code, metadata, and computed metrics. Human experts iteratively validated every case to ensure **compilability**, **presence of exactly one vulnerability or none**, **clear CWE mapping**, and **removal of side-channel hints for LLM evaluation**.

A representative example is the paper’s **CWE-787: Out-of-Bounds Write** case, where `scanf("%s", reg_name); // {!LINE}` copies an unbounded string into a `12`-byte buffer. The paper identifies correct fixes as `scanf("%11s", reg_name);` or `fgets(reg_name, sizeof(reg_name), stdin)` followed by newline stripping. The benchmark repository contains the micro-benchmarks, wrappers, and result-processing scripts at **https://github.com/CASTLE-Benchmark** [2503.09433].

## 3. Evaluation protocol, scoring, and tool normalization in the CWE-detection suite

The 2025 CASTLE suite evaluates **13 static analysis tools**, **2 formal verification tools**, and **10 LLMs**. The static analyzers are **CodeQL 2.20.1, GCC Fanalyzer 13.3.0, Snyk 1.1295.4, Semgrep Code 1.110.0, SonarQube 25.3.0, Aikido (API, Feb 2025), Jit (API, Feb 2025), Coverity 2024.12.1, Cppcheck 2.13.0, Clang Analyzer 18.1.3, GitLab SAST 15.2.1, Splint 3.1.2, CodeThreat (API, Feb 2025)**. The formal verification tools are **ESBMC 7.8.1** and **CBMC 5.95.1**. The LLMs are **GPT-o3 Mini, Chat GPT-o1, DeepSeek R1, GPT-4o, GPT-4o Mini, QWEN 2.5CI (32B), Falcon 3 (7B), Mistral Instruct (7B), Gemma 2 (9B), LLAMA 3.1 (8B)**. Open-source tools are executed through **Dockerized wrappers** and standardized **Python** scripts; closed-source SAST tools are evaluated through vendor APIs, with results parsed and manually cross-checked for consistency with dashboards.

The suite computes the standard metrics **Precision**, **Recall (Sensitivity)**, **F1-score**, **Accuracy**, **False Positive Rate**, and **False Negative Rate**. It also introduces the **CASTLE Score**, designed to normalize comparison across heterogeneous tools that differ in reporting styles, rule granularity, and false-positive behavior. The paper defines the dataset as $d^n = \{ d_1, d_2, \dots, d_n \}$, with ground-truth label $v_i$ and tool output $t(d_i)$. The scoring rule is:

$$
\mathrm{CASTLE}\bigl(t^n\bigr) \;=\; \sum_{i=1}^{n}\; \begin{cases}
5 \;-\;\bigl(|t(d_i)| - 1\bigr)\;+\;B\bigl(t_{cwe}\bigr), & \text{if } v_i \neq \emptyset \;\land\; v_i \in t(d_i) \\
2, & \text{if } v_i = \emptyset \;\land\; t(d_i) = \emptyset \\
-\,\bigl|t(d_i)\bigr|, & \text{otherwise}
\end{cases}
$$

The score rewards correct detections on vulnerable cases with a **base 5 points** plus a **Top-25 CWE bonus**, rewards correct “no findings” on non-vulnerable cases with **2 points**, and penalizes extraneous findings by subtracting the number of reported findings. The paper’s stated rationale is that the score **avoids inflating scores for tools that “spray” many low-value alerts and favors tools that detect the exact vulnerability with minimal noise**.

The LLM evaluation enforces a strict **OpenAI-style JSON interface** with elements of the form `{"severity": string, "line": int, "cwe": int, "message": string, "line_content": string}`. The prompt explicitly states: **“You get 5 points for each vulnerability you find, but get -1 for all false positives you report.”** Responses violating the schema are invalid. Formal-verification runs use relaxed settings suitable for short code; for **ESBMC**, the paper reports `--overflow --no-unwinding-assertions --memory-leak-check --timeout 60 --multi-property --show-stacktrace`. CASTLE also supports **tool combinations** by taking the union of findings, de-duplicating true positives while aggregating false-positive penalties [2503.09433].

## 4. Empirical findings of the CWE-detection suite

Across the **250 tests**, the highest overall CASTLE Score was achieved by **GPT-o3 Mini**, with **TP=126, TN=60, FP=73, FN=36, Precision=63%, Recall=78%, Accuracy=63%, CASTLE Score=977**. Other high-performing LLMs include **Chat GPT-o1** with **CASTLE=962**, **DeepSeek R1** with **TP=148, TN=41, FP=166, FN=17, P=47%, R=90%, CASTLE=956**, and **GPT-4o** with **CASTLE=954**. Among classical tools, **ESBMC (7.8.1)** reached **TP=53, TN=91, FP=32, FN=97, Precision=62%, Recall=35%, Accuracy=53%, CASTLE Score=661**, the top result among formal verification and static-analysis systems. **CodeQL (2.20.1)** was the best SAST tool overall, with **TP=45, TN=79, FP=49, FN=112, P=48%, R=29%, A=44%, CASTLE=634**. **CBMC (5.95.1)** had **Precision=100%** and **FP=0**, but **Recall=12%** and **CASTLE Score=536**. At the opposite extreme, **Splint** and **CodeThreat** received **negative CASTLE Scores** because of **FP=1,027; CASTLE −598** and **FP=1,101; CASTLE −692**, respectively [2503.09433].

The reported pattern is structured by method family. **Formal verification** minimizes false positives and performs well on memory-safety classes, but misses many higher-level CWEs, including **weak cryptography**, **SQL injection**, **path traversal**, and **hard-coded credentials**. **Static analyzers** vary widely: tighter rule sets yield high precision but low recall, whereas aggressive coverage increases manual triage through high false-positive rates. **LLMs** perform exceptionally well on **short, self-contained C snippets**, especially with high recall, but their behavior degrades with larger contexts.

The paper states that **their accuracy declines, and hallucinations increase as the code size grows**, and corroborates this with a synthetic **~400+ LoC** test formed by combining multiple non-vulnerable cases while injecting a hidden vulnerability. In that experiment, **most LLMs raised spurious alerts and missed an injected hidden vulnerability**. A plausible implication is that the benchmark’s deliberately small, single-file structure is especially favorable to pattern-based reasoning over localized defects, whereas longer contexts expose token-budget and context-management limitations.

Combination experiments show that **ESBMC + CodeQL achieved 814 CASTLE points—up 153 over ESBMC alone**. The paper presents this as evidence of **complementary coverage without excessive FP accumulation**. It also argues that **LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities**. The repository guidance recommends regular CASTLE runs in **CI**, calibration of tool configurations, strict schema enforcement for LLMs, and use of balanced non-vulnerable cases to tune false-positive rates before deployment [2503.09433].

## 5. CASTLE for cache timing attacks: threat model, formalization, and benchmark generation

The 2019 CASTLE suite addresses a different problem: **timing-based side or covert channels in processor caches**. Its goal is to cover all theoretically possible timing-based cache vulnerabilities captured by an improved **three-step model** and to provide automatically generated microbenchmarks that determine, with statistical rigor, whether a cache implementation is vulnerable to each type. The suite focuses on **L1 data cache vulnerabilities** in a **multi-core system with possible hyper-threading (SMT)** and distinguishes **local** and **remote** cores. It models an adversary who exploits timing differences in **loads**, **stores**, and **flush/invalidation** events to infer a victim’s unknown address $u$ from a set of sensitive addresses $x$ [1911.08619].

The three steps are: **Step 1 (Setup/Prime, S1)**, **Step 2 (Trigger/Victim activity, S2)**, and **Step 3 (Measure/Probe, S3)**. Compared to prior work, the model incorporates **local vs. remote cores**, **time-slicing vs. hyper-threading**, **read and write operations**, **explicit flush and coherence-induced invalidation**, and **many distinct “fast/slow” sub-classes**. It tracks one cache block through **17 symbolic states**, producing **$17^3 = 4913$ combinations**. After removing repeats and redundancies, and accounting for **66 timing classes for S3**, the simulator identifies **88 Strong vulnerabilities** and **80 Weak vulnerabilities**. The paper states that the broadened modeling assumptions produce **32 Strong vulnerabilities not seen in prior work**.

The taxonomy groups the 88 Strong vulnerabilities by familiar strategy families, including **Cache Collision**, **Flush+Reload**, **Reload+Time**, **Flush+Probe**, **Flush+Time**, **Prime+Probe**, **Evict+Time**, **Evict+Probe**, **Prime+Time**, **Bernstein’s Attack**, and their invalidation counterparts. It also introduces new coherence-based families: **“Cache Coherence Flush+Reload”**, **“Cache Coherence Prime+Probe”**, and **“Cache Coherence Evict+Time.”** Vulnerabilities are categorized by **interference source**—**I (internal)** or **E (external)**—and by **information granularity**—**S (set-based)**, **A (address-based)**, or **SA**.

CASTLE automatically generates **C-based microbenchmarks** for each vulnerability type and operational variant. Each vulnerability can produce up to **8 cases** from the cross-product of access and invalidation modes, and same-core attacker/victim cases can further double under **time-slicing** versus **hyper-threading**. The result is **1,094 test programs**. Implementation uses **fork()**, **shared memory for synchronization**, **thread pinning**, **rdtsc for cycle-level measurement**, **fences between memory-related instructions**, **clflush** for flush-based invalidation, and **remote write** for coherence-induced invalidation. Noise reduction uses **8 cache lines from different sets per trial**, immediate repetition of **S3** to obtain a per-set baseline, and **RUN_NUM=600** iterations. Decision making uses **Welch’s t-test**, with significance threshold **$p < 0.0005$**, and a Strong vulnerability is declared when **exactly one candidate’s distribution is significantly different from the other two** [1911.08619].

## 6. Cache Timing Vulnerability Score, machine comparisons, and cross-suite significance

The 2019 CASTLE suite aggregates outcomes into the **Cache Timing Vulnerability Score (CTVS)**, a compact metric indicating how many of the **88 Strong vulnerability types** are observed on a system. The paper states that **smaller CTVS means a design is more robust to cache timing attacks** and reports the score in the form **“N/88.”** It also provides category breakdowns by **I-A, I-S, I-SA, E-A, E-S, E-SA**, making it possible to separate internal from external interference and address-based from set-based leakage [1911.08619].

The reported machine-level comparisons are: **Intel Xeon E5-1620: 73/88**; **Intel Xeon E5-2667 on-chip: 66/88; inter-chip: 64/88**; **Intel Xeon E5-2690: 62/88**; **Intel Core i5-4570: 61/88**; **Intel Xeon E5-2686 (EC2): 66/88**; **Intel Xeon P-8175 (EC2): 73/88**; **AMD FX-8150: 50/88**; **AMD EPYC 7571: 62/88**; **across all CPUs (AND): 47/88**; **across any CPU (OR): 79/88**. The paper concludes that vulnerability profiles are **highly implementation-specific**. It also reports that **read-based observation generally yields more effective cases**, while **write-based observation also succeeds widely**, and that the relative effectiveness of **flush-based invalidation** versus **remote-write-based invalidation** varies by microarchitecture.

The suite is intended for both empirical testing on commodity processors and evaluation of hardware prototypes or simulators. Designers are advised to run the benchmarks, pin processes to cores or threads, and compare resulting CTVS values across designs. The paper frames lower CTVS as the direct indicator of improvement and discusses potential design directions such as reducing **address-based leakage**, reducing **set-based leakage**, and masking timing differences associated with **remote-induced invalidations**.

The two CASTLE suites therefore occupy complementary positions in security benchmarking. One measures how well analysis systems identify **CWE-labeled vulnerabilities in code**; the other measures how often a machine exhibits **Strong cache timing vulnerabilities**. Both emphasize small, controlled tests, explicit statistical or scoring rules, and reproducibility. At the same time, both papers state important scope limits. The 2025 CASTLE is currently limited to **250 C files**, **single vulnerabilities**, and no **C++**, **Java**, or **Python**; the 2019 CASTLE focuses on **L1 data caches**, excludes **directory attacks**, **replacement-policy channels**, **TLB channels**, and **non-cache microarchitectural channels**. This suggests that CASTLE-style microbenchmarking is most powerful when used as a calibrated component of a broader evaluation program rather than as a complete proxy for deployment conditions [2503.09433] [1911.08619].

Source: https://www.emergentmind.com/topics/castle-microbenchmark-suite