CASTLE Microbenchmark Suite

• CASTLE Microbenchmark Suite is a framework of two benchmarking systems that reduce complex security issues into repeatable microbenchmarks with unified scoring.
• One suite evaluates CWE detection via 250 hand-crafted C tests for static analyzers, formal verification tools, and LLMs, while the other measures cache timing vulnerabilities using 1,094 auto-generated programs.
• Both suites employ explicit decision criteria and statistical scoring rules to provide actionable insights across different layers of the security stack.

Searching arXiv for the specified CASTLE papers to ground the article and verify bibliographic details. {"query":"arXiv (Dubniczky et al., 12 Mar 2025) CASTLE Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection", "max_results": 5} arxiv_search({"query":"(Dubniczky et al., 12 Mar 2025)", "max_results": 5}) The name CASTLE has been used for two distinct security-oriented microbenchmark suites in arXiv literature. In "CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection," CASTLE expands to CWE Automated Security Testing and Low-Level Evaluation and denotes a benchmarking framework for vulnerability detection in source code, built around 250 hand-crafted C micro-benchmarks and a unified CASTLE Score (Dubniczky et al., 12 Mar 2025). In "A Benchmark Suite for Evaluating Caches’ Vulnerability to Timing Attacks," CASTLE denotes a benchmark suite for empirically testing timing side/covert channels in processor caches, built around 1,094 automatically generated C test programs and the Cache Timing Vulnerability Score (CTVS) (Deng et al., 2019). Both suites are microbenchmark-based and score-driven, but they target different layers of the security stack: one evaluates program-analysis methods against CWE-labeled code, while the other evaluates cache implementations against timing-attack classes.

1. Terminological scope and shared benchmarking logic

The two CASTLE suites share a methodological pattern: both reduce complex security phenomena to small, repeatable tests, attach explicit decision criteria to each test, and aggregate outcomes into a unified score. The 2025 CASTLE targets source-code vulnerability detection across traditional static analyzers (SAST), formal verification (FV) tools, and LLMs. The 2019 CASTLE targets timing-based cache vulnerabilities in processor caches under a three-step attack model. This suggests a common design principle: microbenchmarks are used to isolate detection capability while minimizing ambiguity in ground truth or attack interpretation (Dubniczky et al., 12 Mar 2025, Deng et al., 2019).

Aspect	CASTLE for CWE detection	CASTLE for cache timing attacks
Primary target	Source-code vulnerability detection	Cache timing-based vulnerabilities
Artifact size	250 micro-benchmark programs	1,094 test programs
Implementation language	C	Auto-generated C
Core score	CASTLE Score	Cache Timing Vulnerability Score
Unit of analysis	Single-file programs with one vulnerability or none	Three-step timing-vulnerability variants

2. CASTLE for CWE detection: dataset design and benchmark construction

The 2025 CASTLE suite was created to address shortcomings in prior datasets, including lack of compilable programs, imbalanced CWE coverage, and coarse labeling. It provides 250 micro-benchmark programs written in C, covering 25 CWEs, with exactly 10 cases per CWE (6 vulnerable, 4 non-vulnerable). All programs are single-file C sources, compilable without errors (warnings allowed), and each case contains a single, clearly identified CWE or is explicitly marked as non-vulnerable. Code size ranges from 7–164 lines per file (average 42), with 1–8 functions (average 2.2), cyclomatic complexity 1–29 (mean 6.3), and Halstead volume ~89.9–5,246.7 (average 1,104.8). Including the LLM system prompt, the total input across the dataset is ~115,620 tokens (cl100k_base) (Dubniczky et al., 12 Mar 2025).

The covered CWEs are: CWE-22, CWE-78, CWE-89, CWE-125, CWE-134, CWE-190, CWE-253, CWE-327, CWE-362, CWE-369, CWE-401, CWE-415, CWE-416, CWE-476, CWE-522, CWE-617, CWE-628, CWE-674, CWE-761, CWE-770, CWE-787, CWE-798, CWE-822, CWE-835, and CWE-843. The suite is therefore balanced at the per-CWE level and is explicitly structured to support robust measurement of true positives, false positives, false negatives, and true negatives.

Each file embeds a YAML-like metadata header containing dataset name, filename, version, compile command, vulnerability presence flag, short description, and CWE number. Vulnerable lines are annotated with // {!LINE}, enabling line-level ground truth. During preprocessing, metadata is stripped and files are converted to a JSON record with code, metadata, and computed metrics. Human experts iteratively validated every case to ensure compilability, presence of exactly one vulnerability or none, clear CWE mapping, and removal of side-channel hints for LLM evaluation.

A representative example is the paper’s CWE-787: Out-of-Bounds Write case, where scanf("%s", reg_name); // {!LINE} copies an unbounded string into a 12-byte buffer. The paper identifies correct fixes as scanf("%11s", reg_name); or fgets(reg_name, sizeof(reg_name), stdin) followed by newline stripping. The benchmark repository contains the micro-benchmarks, wrappers, and result-processing scripts at https://github.com/CASTLE-Benchmark (Dubniczky et al., 12 Mar 2025).

3. Evaluation protocol, scoring, and tool normalization in the CWE-detection suite

The 2025 CASTLE suite evaluates 13 static analysis tools, 2 formal verification tools, and 10 LLMs. The static analyzers are CodeQL 2.20.1, GCC Fanalyzer 13.3.0, Snyk 1.1295.4, Semgrep Code 1.110.0, SonarQube 25.3.0, Aikido (API, Feb 2025), Jit (API, Feb 2025), Coverity 2024.12.1, Cppcheck 2.13.0, Clang Analyzer 18.1.3, GitLab SAST 15.2.1, Splint 3.1.2, CodeThreat (API, Feb 2025). The formal verification tools are ESBMC 7.8.1 and CBMC 5.95.1. The LLMs are GPT-o3 Mini, Chat GPT-o1, DeepSeek R1, GPT-4o, GPT-4o Mini, QWEN 2.5CI (32B), Falcon 3 (7B), Mistral Instruct (7B), Gemma 2 (9B), LLAMA 3.1 (8B). Open-source tools are executed through Dockerized wrappers and standardized Python scripts; closed-source SAST tools are evaluated through vendor APIs, with results parsed and manually cross-checked for consistency with dashboards.

The suite computes the standard metrics Precision, Recall (Sensitivity), F1-score, Accuracy, False Positive Rate, and False Negative Rate. It also introduces the CASTLE Score, designed to normalize comparison across heterogeneous tools that differ in reporting styles, rule granularity, and false-positive behavior. The paper defines the dataset as $d^n = \{ d_1, d_2, \dots, d_n \}$, with ground-truth label $v_i$ and tool output $t(d_i)$. The scoring rule is:

$$\mathrm{CASTLE}\bigl(t^n\bigr) \;=\; \sum_{i=1}^{n}\; \begin{cases} 5 \;-\;\bigl(|t(d_i)| - 1\bigr)\;+\;B\bigl(t_{cwe}\bigr), & \text{if } v_i \neq \emptyset \;\land\; v_i \in t(d_i) \ 2, & \text{if } v_i = \emptyset \;\land\; t(d_i) = \emptyset \ -\,\bigl|t(d_i)\bigr|, & \text{otherwise} \end{cases}$$

The score rewards correct detections on vulnerable cases with a base 5 points plus a Top-25 CWE bonus, rewards correct “no findings” on non-vulnerable cases with 2 points, and penalizes extraneous findings by subtracting the number of reported findings. The paper’s stated rationale is that the score avoids inflating scores for tools that “spray” many low-value alerts and favors tools that detect the exact vulnerability with minimal noise.

The LLM evaluation enforces a strict OpenAI-style JSON interface with elements of the form {"severity": string, "line": int, "cwe": int, "message": string, "line_content": string}. The prompt explicitly states: “You get 5 points for each vulnerability you find, but get -1 for all false positives you report.” Responses violating the schema are invalid. Formal-verification runs use relaxed settings suitable for short code; for ESBMC, the paper reports --overflow --no-unwinding-assertions --memory-leak-check --timeout 60 --multi-property --show-stacktrace. CASTLE also supports tool combinations by taking the union of findings, de-duplicating true positives while aggregating false-positive penalties (Dubniczky et al., 12 Mar 2025).

4. Empirical findings of the CWE-detection suite

Across the 250 tests, the highest overall CASTLE Score was achieved by GPT-o3 Mini, with TP=126, TN=60, FP=73, FN=36, Precision=63%, Recall=78%, Accuracy=63%, CASTLE Score=977. Other high-performing LLMs include Chat GPT-o1 with CASTLE=962, DeepSeek R1 with TP=148, TN=41, FP=166, FN=17, P=47%, R=90%, CASTLE=956, and GPT-4o with CASTLE=954. Among classical tools, ESBMC (7.8.1) reached TP=53, TN=91, FP=32, FN=97, Precision=62%, Recall=35%, Accuracy=53%, CASTLE Score=661, the top result among formal verification and static-analysis systems. CodeQL (2.20.1) was the best SAST tool overall, with TP=45, TN=79, FP=49, FN=112, P=48%, R=29%, A=44%, CASTLE=634. CBMC (5.95.1) had Precision=100% and FP=0, but Recall=12% and CASTLE Score=536. At the opposite extreme, Splint and CodeThreat received negative CASTLE Scores because of FP=1,027; CASTLE −598 and FP=1,101; CASTLE −692, respectively (Dubniczky et al., 12 Mar 2025).

The reported pattern is structured by method family. Formal verification minimizes false positives and performs well on memory-safety classes, but misses many higher-level CWEs, including weak cryptography, SQL injection, path traversal, and hard-coded credentials. Static analyzers vary widely: tighter rule sets yield high precision but low recall, whereas aggressive coverage increases manual triage through high false-positive rates. LLMs perform exceptionally well on short, self-contained C snippets, especially with high recall, but their behavior degrades with larger contexts.

The paper states that their accuracy declines, and hallucinations increase as the code size grows, and corroborates this with a synthetic ~400+ LoC test formed by combining multiple non-vulnerable cases while injecting a hidden vulnerability. In that experiment, most LLMs raised spurious alerts and missed an injected hidden vulnerability. A plausible implication is that the benchmark’s deliberately small, single-file structure is especially favorable to pattern-based reasoning over localized defects, whereas longer contexts expose token-budget and context-management limitations.

Combination experiments show that ESBMC + CodeQL achieved 814 CASTLE points—up 153 over ESBMC alone. The paper presents this as evidence of complementary coverage without excessive FP accumulation. It also argues that LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities. The repository guidance recommends regular CASTLE runs in CI, calibration of tool configurations, strict schema enforcement for LLMs, and use of balanced non-vulnerable cases to tune false-positive rates before deployment (Dubniczky et al., 12 Mar 2025).

5. CASTLE for cache timing attacks: threat model, formalization, and benchmark generation

The 2019 CASTLE suite addresses a different problem: timing-based side or covert channels in processor caches. Its goal is to cover all theoretically possible timing-based cache vulnerabilities captured by an improved three-step model and to provide automatically generated microbenchmarks that determine, with statistical rigor, whether a cache implementation is vulnerable to each type. The suite focuses on L1 data cache vulnerabilities in a multi-core system with possible hyper-threading (SMT) and distinguishes local and remote cores. It models an adversary who exploits timing differences in loads, stores, and flush/invalidation events to infer a victim’s unknown address $u$ from a set of sensitive addresses $x$ (Deng et al., 2019).

The three steps are: Step 1 (Setup/Prime, S1), Step 2 (Trigger/Victim activity, S2), and Step 3 (Measure/Probe, S3). Compared to prior work, the model incorporates local vs. remote cores, time-slicing vs. hyper-threading, read and write operations, explicit flush and coherence-induced invalidation, and many distinct “fast/slow” sub-classes. It tracks one cache block through 17 symbolic states, producing $17^3 = 4913$ combinations. After removing repeats and redundancies, and accounting for 66 timing classes for S3, the simulator identifies 88 Strong vulnerabilities and 80 Weak vulnerabilities. The paper states that the broadened modeling assumptions produce 32 Strong vulnerabilities not seen in prior work.

The taxonomy groups the 88 Strong vulnerabilities by familiar strategy families, including Cache Collision, Flush+Reload, Reload+Time, Flush+Probe, Flush+Time, Prime+Probe, Evict+Time, Evict+Probe, Prime+Time, Bernstein’s Attack, and their invalidation counterparts. It also introduces new coherence-based families: “Cache Coherence Flush+Reload”, “Cache Coherence Prime+Probe”, and “Cache Coherence Evict+Time.” Vulnerabilities are categorized by interference source—I (internal) or E (external)—and by information granularity—S (set-based), A (address-based), or SA.

CASTLE automatically generates C-based microbenchmarks for each vulnerability type and operational variant. Each vulnerability can produce up to 8 cases from the cross-product of access and invalidation modes, and same-core attacker/victim cases can further double under time-slicing versus hyper-threading. The result is 1,094 test programs. Implementation uses fork(), shared memory for synchronization, thread pinning, rdtsc for cycle-level measurement, fences between memory-related instructions, clflush for flush-based invalidation, and remote write for coherence-induced invalidation. Noise reduction uses 8 cache lines from different sets per trial, immediate repetition of S3 to obtain a per-set baseline, and RUN_NUM=600 iterations. Decision making uses Welch’s t-test, with significance threshold $p < 0.0005$, and a Strong vulnerability is declared when exactly one candidate’s distribution is significantly different from the other two (Deng et al., 2019).

6. Cache Timing Vulnerability Score, machine comparisons, and cross-suite significance

The 2019 CASTLE suite aggregates outcomes into the Cache Timing Vulnerability Score (CTVS), a compact metric indicating how many of the 88 Strong vulnerability types are observed on a system. The paper states that smaller CTVS means a design is more robust to cache timing attacks and reports the score in the form “N/88.” It also provides category breakdowns by I-A, I-S, I-SA, E-A, E-S, E-SA, making it possible to separate internal from external interference and address-based from set-based leakage (Deng et al., 2019).

The reported machine-level comparisons are: Intel Xeon E5-1620: 73/88; Intel Xeon E5-2667 on-chip: 66/88; inter-chip: 64/88; Intel Xeon E5-2690: 62/88; Intel Core i5-4570: 61/88; Intel Xeon E5-2686 (EC2): 66/88; Intel Xeon P-8175 (EC2): 73/88; AMD FX-8150: 50/88; AMD EPYC 7571: 62/88; across all CPUs (AND): 47/88; across any CPU (OR): 79/88. The paper concludes that vulnerability profiles are highly implementation-specific. It also reports that read-based observation generally yields more effective cases, while write-based observation also succeeds widely, and that the relative effectiveness of flush-based invalidation versus remote-write-based invalidation varies by microarchitecture.

The suite is intended for both empirical testing on commodity processors and evaluation of hardware prototypes or simulators. Designers are advised to run the benchmarks, pin processes to cores or threads, and compare resulting CTVS values across designs. The paper frames lower CTVS as the direct indicator of improvement and discusses potential design directions such as reducing address-based leakage, reducing set-based leakage, and masking timing differences associated with remote-induced invalidations.

The two CASTLE suites therefore occupy complementary positions in security benchmarking. One measures how well analysis systems identify CWE-labeled vulnerabilities in code; the other measures how often a machine exhibits Strong cache timing vulnerabilities. Both emphasize small, controlled tests, explicit statistical or scoring rules, and reproducibility. At the same time, both papers state important scope limits. The 2025 CASTLE is currently limited to 250 C files, single vulnerabilities, and no C++, Java, or Python; the 2019 CASTLE focuses on L1 data caches, excludes directory attacks, replacement-policy channels, TLB channels, and non-cache microarchitectural channels. This suggests that CASTLE-style microbenchmarking is most powerful when used as a calibrated component of a broader evaluation program rather than as a complete proxy for deployment conditions (Dubniczky et al., 12 Mar 2025, Deng et al., 2019).