ExCyTIn-Bench: Dual Benchmark Suites

• ExCyTIn-Bench is a dual-purpose benchmark suite that rigorously evaluates LLM-based cyber threat investigations and interval arithmetic libraries.
• It employs realistic multi-step attack simulations, graph-based threat analysis, and comprehensive tests for floating-point interval accuracy.
• The benchmarks emphasize reproducibility, verifiable performance metrics, and real-world complexity across diverse technical domains.

ExCyTIn-Bench serves as the designation for two unrelated, independently-published benchmark suites in distinct technical areas: (1) the evaluation of LLMs for cyber threat investigation (Wu et al., 14 Jul 2025), and (2) the certification and comparison of interval arithmetic libraries in floating-point computation (Tang et al., 2021). Both benchmarks aim to rigorously evaluate automated agents or libraries against standards of verifiability, correctness, and real-world complexity, but their domains, methodologies, and artifacts have no commonality beyond the shared name “ExCyTIn-Bench.” Details for each meaning are presented in the sections below.

1. ExCyTIn-Bench for LLM-Based Cyber Threat Investigation

ExCyTIn-Bench (Wu et al., 14 Jul 2025) is a benchmark designed to evaluate agents powered by LLMs on realistic cyber threat investigation workflows. Its primary focus is to emulate and test the capabilities required by security analysts, specifically handling complex incident traces across heterogeneous security logs.

1.1 Dataset Construction and Attack Scenarios

The corpus underlying ExCyTIn-Bench is constructed from security telemetry generated within a controlled Microsoft Azure tenant. It features 8 simulated multi-step attacks, with each incident mapping high-level MITRE kill-chain phases—such as Initial Access, Credential Access, Lateral Movement, and Impact—to real alert sources. Data is sourced from 57 distinct log tables compiled from Microsoft Sentinel and related Azure services, summarized in the following structure:

ID	Title	#Alerts	DeviceProc (%)	SignIn (%)	Email (%)	Net (%)	Other (%)
5	Operation Alpine LockBit: Manatee Tempest	2770	35	15	10	20	20
...	...	...	...	...	...	...	...

Typical attack phases are mapped to specific log tables, e.g.: Initial Access (EmailEvents, UrlClickEvents, SecurityAlert); Credential Access (DeviceProcessEvents, SigninLogs, SecurityAlert); Lateral Movement (DeviceNetworkEvents), among others. Each table includes explicit column-level schemas, such as DeviceProcessEvents (30 fields) or SecurityAlert (18 fields) (Wu et al., 14 Jul 2025).

1.2 Threat Investigation Graphs

Each incident is formalized as a bipartite graph $G = (V, E)$ with $V = U \cup W$, where $U$ are alert nodes and $W$ are entity nodes (IP addresses, user accounts, file names, etc.). An edge $(u, w) \in E$ exists if alert $u$ refers to entity $w$ in its JSON-encoded “Entity” field. SecurityAlert nodes are systematically derived from expert-crafted detection logic, e.g.,

$W$8

This graph formulation provides a concrete, explainable ground for multi-hop investigation tasks.

1.3 Question Generation and Benchmark Task

LLM-driven QA pairs are synthesized by selecting ordered pairs $(u_s, u_e)$ of alerts in the graph trace and anchor entities. The prompt construction algorithm operates as follows:

• The context is drawn from $u_s$ and the $k$ entities farthest from $V = U \cup W$0 on paths toward $V = U \cup W$1.
• A natural-language question, referencing the investigative context and answer entity $V = U \cup W$2, is generated using an LLM.
• Gold-standard solutions are the shortest paths in $V = U \cup W$3 from $V = U \cup W$4 to $V = U \cup W$5, also serving as explanations.

Example:

• Context: “A drive-by download from vectorsandarrows.com delivered via curl.exe triggered ‘Suspicious hands on keyboard’ on host vnevado-win11t.”
• Question: “Which user account executed the BLISTER loader on vnevado-win11t?”
• Answer: “dp5hn”

1.4 Procedural Task Environment and Reward Design

Each question induces a multi-hop task: the LLM agent issues a sequence of SQL queries, receives database observations, and ultimately provides a final answer. The reward $V = U \cup W$6 is

$V = U \cup W$7

where $V = U \cup W$8 enumerates the ground-truth investigative steps, $V = U \cup W$9 is a discount factor, and $U$0 is the agent’s submitted answer. This verifiable, partial-credit reward is well-suited for procedural RL.

1.5 Evaluation Protocol and Results

Evaluation proceeds over 589 generated questions, with core metrics:

• Average Reward: $U$1
• Final-Answer Accuracy: $U$2
• F₁-Score: for evaluating the retrieval of intermediate steps

Empirical results (base prompt, temperature 0, 25 steps per trace):

Model	Average Reward
o4-mini	0.368
GPT-4.1	0.338
Gemini 2.5 Flash	0.305
Llama4-Mav-17B	0.290
Phi-4-14B	0.085
...	...

The best model achieves a mean reward of 0.368, indicating substantial room for progress in this multi-hop setting.

1.6 Extensibility and Open Challenges

ExCyTIn-Bench is engineered for expansion:

• New log sources and attack types can be integrated by extending the graph construction and detection logic modules.
• The pipeline supports the definition of new procedural tasks, enabling RL-based training due to the dense, verifiable reward signal.

Challenges remain: handling real-world noise and undetected events; disambiguating conflicting logs; and integrating broader toolsets (e.g., direct PowerShell or REST API calls). The benchmark is positioned as a foundation for next-generation LLM-based autonomous cyber threat investigation agents (Wu et al., 14 Jul 2025).

2. ExCyTIn-Bench for Interval Arithmetic Library Certification

ExCyTIn-Bench (Tang et al., 2021) (“EXact, Cross-plaTform Interval ComputaTIon BENChmark”) is an independent suite aimed at certifying and comparing C/C++ interval arithmetic libraries. Its main objective is to ensure that interval-based computations truly enclose exact real solutions and to benchmark properties such as correctness, tightness, performance, consistency, and portability.

2.1 Goals and Scope

• Certification: Verify that candidate interval libraries guarantee enclosure of the exact result, for both basic and transcendental operations, across floating-point expressions.
• Comparison: Evaluate correctness, interval tightness, computational efficiency, answer consistency across platforms, and build portability.

Evaluation comprises 28 core expressions (arithmetic, transcendental, composite) and 104 complex FPBench formulas, with 1 million randomly sampled inputs per expression. Reference results are computed with arbitrary-precision rationals via GMP and Mathematica (Tang et al., 2021).

2.2 Methodology

Test class selection includes:

• Basic arithmetic: $U$3, $U$4, $U$5, $U$6
• Single transcendental: $U$7, $U$8, $U$9, $W$0
• Composite: Arithmetic and arithmetic+transcendental blends
• 104 FPBench formulas from varied domains

All inputs and outputs are rational numbers; library-generated intervals are compared to ground truth by verifying that every sampled value lies within the reported interval.

2.3 Core Definitions

Intervals are denoted $W$1, with directed rounding guaranteeing enclosure:

$W$2

and similarly for other operations, with $W$3 denoting round-to-$W$4 and $W$5 respectively. Transcendentals adopt the convention $W$6.

2.4 Libraries and Platforms

Tested libraries:

• filib (C, bit-level rounding control)
• filib++ (C++ wrapper), offering modes: native, pred-succ, multiplicative
• Boost.Interval
• BIAS (Linux-only, unmaintained)

Host systems: Windows 10/x86-64/MSVC, macOS Intel and M1/arm64/Clang, Linux/GCC, ensuring wide cross-platform coverage.

2.5 Metrics and Evaluation

Benchmarks include:

• Correctness/Containment: Fraction of test expressions where the library interval contains the exact value, for all samples.
• Tightness: Interval size $W$7 (as exact rationals), normalized.
• Performance: Wall-clock time, per-operation (exclusive of I/O).
• Consistency: Bitwise equality of intervals across platforms.
• Portability: Success of out-of-the-box build and execution.

2.6 Key Findings

• filib (C): Fully correct; narrow intervals; 2–10× faster than Boost; portable via CMake.
• filib++ (C++):
  • pred-succ/multiplicative: fully correct; multiplicative is fastest for arithmetic-only workloads.
  • native: suffers correctness issues in certain composite tests on Linux.
• Boost.Interval: Fails transcendental correctness in ≈50% of tests; inconsistent across OSes due to platform libm calls; much slower for composites.
• BIAS: Only basic correctness; composite behavior unpredictable; unmaintained.

Best practices: Use filib/filib++ pred-succ or multiplicative for highest safety; avoid native or Boost for certified computation. Rerun benchmarks following any build environment change (Tang et al., 2021).

2.7 Reproducibility

Automated scripts and public repositories enable benchmarking and validation. Example invocation (for Linux/macOS/Windows):

$W$9

This generates correctness, size, and performance reports, exactly reproducing published analyses (Tang et al., 2021).

3. Conclusion

“ExCyTIn-Bench” refers to rigorously engineered benchmarks for (i) LLM-based cyber threat investigation and (ii) interval arithmetic library certification. Both emphasize reproducibility, transparency, and measurable progress on complex, real-world technical challenges. Each benchmark suite has defined metrics, protocols, reference implementations, and public resources intended to drive advances in their respective domains (Wu et al., 14 Jul 2025, Tang et al., 2021).