RedSage-Bench: Cybersecurity LLM Evaluation

• RedSage-Bench is a large-scale evaluation suite for benchmarking cybersecurity LLMs by assessing factual knowledge, offensive skills, and tool proficiency.
• It comprises 30,000 multiple-choice and 240 open-ended QA items generated via agentic augmentation and validated through automated and human review.
• Performance metrics reveal enhanced accuracy in factual recall and tool usage, setting new standards for domain-specific LLM evaluation in cybersecurity.

RedSage-Bench is a large-scale, multi-dimensional evaluation suite specifically developed for benchmarking LLMs on cybersecurity knowledge, practical skills, and tool expertise. Designed to address the limitations of prior benchmarks—which have tended to focus narrowly on either factual recall or interactive agentic tasks but not both—RedSage-Bench offers a unified, rigorously curated resource that systematically covers multiple dimensions of cybersecurity proficiency and establishes high standards for both quantitative and qualitative evaluation (Suryanto et al., 29 Jan 2026).

1. Motivation and Scope

RedSage-Bench was introduced to overcome deficiencies in existing cybersecurity LLM evaluation paradigms. Previous benchmarks such as SecEval and CyberMetric primarily measured knowledge recall, while agentic tasks featured in NYU-CTF and CyBench focused on interactive problem-solving. None have jointly assessed domain knowledge, practical offensive skills, or proficiency with security tools, nor have they systematically evaluated qualitative, free-form answers beyond multiple-choice formats. In addition, most available resources have lacked open access and reliable open-ended response scoring. RedSage-Bench provides the following key innovations:

• A unified evaluation suite assessing knowledge, skills, and tool usage.
• Large-scale coverage with 30,000 multiple-choice questions (MCQs) and 240 open-ended QA items.
• Rigorous verification and balanced subdomain representation through multi-stage LLM and human review.
• Openness and transparency for reproducible research.

2. Dataset Composition and Subdomain Structure

RedSage-Bench comprises 30,240 items, split into:

• 30,000 MCQs, divided evenly across three subdomains: Knowledge, Skills (offensive operations), and Tools (CLI usage and Kali Linux tools), yielding 10,000 MCQs per category.
• 240 open-ended QA items, similarly distributed as 80 per subdomain.

The source corpus, termed RedSage-Seed, contains 28,637 curated documents, encompassing:

• Knowledge: General references (e.g., Wikipedia, roadmap.sh), frameworks such as MITRE ATT&CK, CAPEC, CWE, and OWASP Top 10.
• Skills: Offensive security write-ups and tutorials from sources like 0xdf CTFs, HackTricks, and Null Byte.
• Tools: CLI cheat-sheets (tldr-pages, Linux man pages) and documentation for Kali Linux tools.

MCQs are generated using Llama-3.3-70B/Instruct under a closed-book protocol, producing four-option questions (one correct, three plausible distractors). Open-ended items are created via a two-stage "agentic augmentation" pipeline, simulating realistic evaluation types such as command construction, vulnerability reasoning, and framework mapping, each with self-contained prompts and model reference answers.

3. Design Methodology and Verification Pipeline

The construction of RedSage-Bench involves a layered agentic pipeline:

• Agentic Augmentation: A Planner Agent analyzes each seed document chunk and proposes multiple augmentation plans, targeting specific testing skills (e.g., command-line construction, threat TTP classification) and interaction formats. The Augmenter Agent instantiates these plans as multi-turn dialogues or QA prompts, preserving technical nuance and procedural depth.
• Item Verification: The verification protocol consists of:
  • LLM-based format and validity checking (structural correctness, self-containment, relevance).
  • Automated quality scoring ($s\in[0,10]$); only items with $s>8$ are retained.
  • Quota-aware sampling ensures identical MCQ and QA counts for each subdomain.
  • Human inspection is applied to all 240 open-ended QA items to remove subtle errors.
  • Decontamination removes synthetic training instances with semantic overlap $>0.9$ with benchmark queries, preventing train–test leakage (removing 2.96% of candidates).

4. Evaluation Protocol

RedSage-Bench specifies precise evaluation procedures for both MCQs and open-ended QA:

• MCQ Evaluation:
  • Zero-shot setting: models produce the answer option (A–D).
  • Accuracy is computed as:

  $$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\bigl(\arg\max_{j\in\{A,B,C,D\}}\log p_{i,j} = \text{correct}_i\bigr)$$ - Baselines include both general models (Llama-3.1-8B, Qwen3-8B-Base) and cybersecurity-tuned models (Llama-Primus, Foundation-Sec, Lily-Cybersecurity-7B, DeepHat-V1-7B, and RedSage variants).

• Open-Ended QA Evaluation:

  • Assessed via an LLM-as-Judge rubric, with binary correctness and a quality score (0–10) based on helpfulness, relevance, depth, and detail.
  • Metrics: mean correctness and mean quality per subdomain.

5. Key Results and Comparative Performance

RedSage-Bench enables detailed comparison between generalist LLMs and cybersecurity-adapted models. Notable results (0-shot macro accuracy %, MCQs):

Model	Macro	Gen	Frm	Skill	CLI	Kali
Qwen3-8B	84.24	83.08	81.94	88.23	85.08	78.86
RedSage-8B-CFW	84.86	83.62	83.30	88.81	85.30	79.32
RedSage-8B-Seed	85.21	83.64	84.56	88.82	85.50	79.90
RedSage-8B-Base	85.05	83.12	84.94	88.72	85.44	79.36

On open-ended QA (mean correctness): RedSage-8B-DPO achieved 0.87, outperforming Qwen3-8B by 7 percentage points. Mean quality scores (0–10) were 7.12 for RedSage-8B-DPO versus 7.05 for Qwen3-8B. Subdomain analysis indicated:

• Knowledge: >90% correctness with tight scoring distribution.
• Skills: 75–85% correctness.
• Tools: 60–75% correctness, with significant lower-tail performance drops.

6. Domain Insights and Cross-Benchmark Correlations

RedSage-Bench surfaced the following insights:

• All models demonstrated strong factual recall in knowledge-heavy tasks.
• Skill-based (offensive operations) items revealed reasoning limitations, particularly in multi-step exploit logic.
• Tool proficiency (correct flags and syntax) represented the hardest subdomain, with fine-grained errors persisting across most models.

Seed-only continual pretraining optimized framework and Kali tool performance, while large-scale "CyberFineWeb" filtering improved general knowledge. Agentic augmentation of free-response items achieved greater procedural understanding, corresponding to a +7 pp open-QA correctness improvement for RedSage.

Performance on RedSage-Bench MCQs correlated with gains on established benchmarks (CTI-Bench, SecBench, CyberMetric, SECURE; +2–4 pp over Qwen3-8B), and superior open-response results reflected improved outcomes on generic instruction-following tasks (IFEval, ARC-C, GSM8K).

A plausible implication is that rigorous, domain-aware agentic data generation and validation pipelines such as those in RedSage-Bench may set a new bar for high-fidelity LLM benchmarking in technical domains beyond cybersecurity.

7. Availability and Research Impact

All RedSage-Bench resources, including the benchmark itself, models, datasets, and code, are publicly available (Suryanto et al., 29 Jan 2026). This open-science approach enables reproducibility, fosters further research on domain-specific LLMs, and facilitates comparative evaluation across a spectrum of open and proprietary models. RedSage-Bench thus constitutes a foundational asset for systematically advancing the capabilities and assessment rigor of agentic LLMs within the cybersecurity domain and potentially in other high-stakes, technical fields.