---
title: 'RedSage: Cybersecurity LLM'
url: https://www.emergentmind.com/topics/redsage
type: topic
---

# RedSage: Cybersecurity LLM

RedSage is an open-source, domain-specialized large language model (LLM) designed for cybersecurity operations and workflows. It addresses the challenge of supporting diverse, expert-level cybersecurity interactions without incurring the privacy and data protection risks of proprietary LLM APIs. RedSage utilizes a combination of domain-aware continual pretraining, massive agentic augmentation, and post-training alignment to achieve superior performance on closed-book, conversational, and general reasoning tasks. All models, benchmarks, and datasets are publicly released for on-premise deployment and reproducible research [2601.22159].

## 1. Domain Data Curation and Pretraining

RedSage leverages a multi-stage data curation approach, selectively filtering, deduplicating, and augmenting content from large-scale web corpora and authoritative resources.

### CyberFineWeb Construction

- **Corpus Source**: The FineWeb corpus (Common Crawl 2013–2024; ∼17.2T tokens) serves as the base.  
- **Cybersecurity Filtering**: A ModernBERT binary classifier, trained on 4.6M positive and negative samples, identifies cybersecurity-relevant content. This yields ∼125M candidate documents (89.8B tokens).
- **Data Mixture Strategy**: For continual pretraining, a 30% replay ratio of FineWeb-Edu (broad general knowledge, 1.3T tokens) is included in each chunk to mitigate catastrophic forgetting.
- **Deduplication**: Global MinHash-LSH (64-bit, 8 hashes, 14 buckets) removes 58% of documents and 48% of tokens, resulting in ∼52M documents (46.8B tokens).
- **Final Selection**: The last 5 chronological chunks are kept for cost control, yielding ∼13M documents (11.7B tokens).

### RedSage-Seed Curation

- **Authoritative Seed Documents**: 28,637 Markdown files (0.15B tokens) are manually curated and organized:
  - Knowledge–General: MITRE ATT&CK, CWEs, OWASP Top 10, Wikipedia, roadmap.sh
  - Knowledge–Frameworks: CAPEC, CWE, MITRE ATT&CK, OWASP
  - Skills–Offensive: CTF write-ups, HackTricks, Nullbyte, PayloadsAllTheThings
  - Tools–CLI: TLDR pages, Linux man pages
  - Tools–Kali: Official Kali Linux documentation

### Data Mixing Statistics

| Selection Stage                  | %Docs (orig. FineWeb) | %Tokens (orig. FineWeb) |
|----------------------------------|----------------------:|------------------------:|
| Filtered cybersecurity           | 0.51%                 | 0.52%                   |
| After deduplication + general mix| 0.21%                 | 0.27%                   |
| Final used (for pretraining)     | 0.053%                | 0.068%                  |

*This targeted curation yields a pretraining dataset with high domain relevance and broad coverage, balancing authoritative depth with general contextual knowledge.*

## 2. Agentic Augmentation Pipeline

RedSage adopts a two-stage agentic pipeline—modeled after AgentInstruct—for generating large-scale, realistic expert–assistant multi-turn dialogues across the cybersecurity domain.

### Pipeline Components

1. **Planner Agent**:  
   - Consumes Markdown input from RedSage-Seed (≤32K tokens per chunk).
   - Outputs structured JSON "plans," each covering skill sets (e.g., Network Discovery, Exploitation Workflows) with various “augmentation types” (e.g., ARP scan simulation, command-crafting).
   - Prompting enforces relevance, technical diversity, creativity, and conversational grounding.

2. **Augmenter Agent**:  
   - Takes a single augmentation plan and original content.
   - Generates fixed-format, multi-turn (typically 8–12 turns) chat conversations. Each interaction features technical rigor, step-by-step reasoning, actionable best practices, and integrating references to standards (MITRE, NIST).

### Augmentation Scale

| Category               | #Samples | Avg Tokens | Avg Turns |
|------------------------|---------:|-----------:|----------:|
| Knowledge–General      |   67,635 |      1,327 |      9.96 |
| Knowledge–Frameworks   |   39,908 |      1,285 |      9.46 |
| Skills–Offensive       |   38,870 |      1,345 |     10.10 |
| Tools–CLI              |  109,261 |      1,331 |      9.73 |
| Tools–Kali             |   10,506 |      1,356 |      9.26 |
| **Total**              |  266,180 |      1,326 |      9.70 |

Agentic augmentation expands the supervised fine-tuning dataset by 9.2× in sample count and 2.3× in total tokens relative to seeds.

*This methodology yields high-diversity, deeply contextualized training dialogues aligned with real-world expertise and operational workflows.*

## 3. Model Architecture and Training Regimen

RedSage builds upon the Qwen3-8B-Base architecture, a Transformer-decoder LLM with 8B parameters. No architectural modifications are introduced.

### Training Objectives and Procedure

- **Pretraining Objective**: Standard next-token prediction via cross-entropy:
  $$
  \mathcal{L} = -\sum_{t=1}^T \sum_{v \in V} \mathbf{1}_{\{v = y_t\}} \log p_\theta(v \mid x_{<t})
  $$

#### Training Stages

1. **Continued Pretraining (CPT)**
   - Stage 1: CyberFineWeb (11.7B tokens); 1 epoch; batch size 1024 (on 32×A100); learning rate 2.5×10⁻⁶; AdamW optimizer; linear warmup (1K steps); ZeRO-3.
   - Stage 2: RedSage-Seed (0.15B tokens) plus RedSage-Dump (0.7B tokens); 1 epoch; same hyperparameters.

2. **Supervised Fine-Tuning (SFT)**
   - Data: RedSage-Conv (266K dialogues) + SmolTalk2 non-reasoning subset (~M samples).
   - Mixing Ratio: Implicitly ~15% cybersecurity-specific, 85% generic.
   - 2 epochs; batch size 1024; learning rate 2.5×10⁻⁵; cosine schedule; weight_decay=0.05; warmup=1%.

3. **Direct Preference Optimization (DPO)**
   - Data: Tulu 3 8B Preference Mixture (~250K preference pairs); α=0.2, β=1.0.

*Total training compute: ~134 GPU-hours (4,096 GPU-h on 32×A100), including CPT, SFT, and DPO.*

## 4. Evaluation Benchmarks and Results

RedSage's capabilities are assessed on both cybersecurity-specific and general LLM benchmarks, employing rigorous methodologies and head-to-head comparisons.

### 4.1 RedSage-Bench

A new evaluation suite, RedSage-Bench, comprises 30,000 multiple-choice questions and 240 open-ended QA items, equally distributed over Knowledge, Skills, and Tools. MCQs derive from Llama-3.3-70B Instructor prompts, validated via a two-stage LLM verifier. MCQs use normalized log-likelihood accuracy; open-ended items use LLM-as-Judge with a 0–10 quality rubric.

**RedSage-Bench MCQ 0-shot Accuracy (8B Params)**

| Model                   | Macro | Gen   | Frm   | Off   | CLI   | Kali  |
|-------------------------|-------|-------|-------|-------|-------|-------|
| Qwen3-8B-Base           | 84.24 | 83.08 | 81.94 | 88.23 | 85.08 | 78.86 |
| RedSage-8B-CFW          | 84.86 | 83.62 | 83.30 | 88.81 | 85.30 | 79.32 |
| RedSage-8B-Base         | 85.05 | 83.12 | 84.94 | 88.72 | 85.44 | 79.36 |
| RedSage-8B-Ins (SFT)    | 85.73 | 84.20 | 84.98 | 89.06 | 86.80 | 80.30 |
| RedSage-8B-DPO          | 84.83 | 82.48 | 83.80 | 88.54 | 86.30 | 79.30 |

SFT provides up to +1.49 points macro improvement over Qwen3-8B. On open-ended QA, DPO further boosts correctness (+7% absolute, 0.74→0.81) and quality score (+0.07, 6.43→6.50).

### 4.2 Standard Cybersecurity Benchmarks

| Model                | Mean   | CTI-MCQ | RCM   | CyMtc | MMLU-CSec | SecBench | SECURE-CWET |
|----------------------|--------|---------|-------|-------|-----------|----------|-------------|
| Qwen3-8B-Base        | 80.81  | 68.80   | 63.50 | 93.80 | 86.00     | 83.62    | 93.33       |
| RedSage-8B-Base      | 84.56  | 71.04   | 78.40 | 92.60 | 87.00     | 81.76    | 93.22       |
| RedSage-8B-DPO       | 81.10  | 70.84   | 70.60 | 90.00 | 79.00     | 80.06    | 91.35       |

Improvements of +3.75 (CPT) to +5.59 points (DPO) macro over the Qwen3-8B-Base.

### 4.3 General LLM Benchmarks

Accuracy on Open LLM Leaderboard (MMLU, ARC-C, GSM8K, HellaSwag, TruthfulQA, WinoGrande, IFEval):

| Model               | Mean   | MMLU  | ARC-C | GSM8K | HSwag | TQA   | WinoG | IFEval |
|---------------------|--------|-------|-------|-------|-------|-------|-------|--------|
| Qwen3-8B-Base       | 70.86  | 78.73 | 68.09 | 81.73 | 79.62 | 43.84 | 73.16 | –      |
| RedSage-8B-Ins (SFT)| 73.34  | 77.38 | 69.62 | 86.05 | 79.00 | 47.75 | 73.64 | 79.97  |
| RedSage-8B-DPO      | 74.33  | 77.07 | 71.76 | 82.71 | 79.87 | 52.47 | 73.01 | 83.44  |

RedSage variants outperform base models by 3.48 to 3.77 points mean accuracy.

### Evaluation Methodology

RedSage's evaluation pipeline applies 0-shot or 5-shot prompting as standard for each benchmark. MCQ accuracy reflects direct answer prediction without few-shot context. Open-ended QAs leverage a rubric-based LLM judge for human-like assessment.

## 5. Implications and Discussion

RedSage empirically demonstrates that large-scale, domain-aware continual pretraining—spanning both broad filtered web data and manually curated seeds—confers synergistic gains: broad general coverage from web-scale corpora complements high-precision, framework-grounded expertise from selected documentation. Agentic augmentation on this seed corpus yields an order of magnitude more diverse, operationally grounded multi-turn chats, which improve both factual accuracy and conversational depth.

Supervised fine-tuning that integrates cybersecurity dialogues while maintaining a majority representation of generic instructions ensures RedSage sustains generalized reasoning and instruction-following capabilities, rather than catastrophic overspecialization. Direct Preference Optimization further improves helpfulness and free-form response quality.

A notable finding is that cybersecurity specialization does not degrade general language or reasoning capability; instead, it yields improvements of 3–5 points on standard LLM benchmarks such as MMLU, ARC-C, and GSM8K—a result suggesting that well-structured domain expertise and high-quality, agentic conversational data positively reinforce multi-step reasoning and answer clarity.

RedSage’s open-source release of all artifacts facilitates on-premise, privacy-preserving deployments suitable for security operations centers and research environments where confidentiality requirements preclude third-party cloud APIs. The reproducible data curation, augmentation, and training pipeline may inform future work on other specialized domains.

Source: https://www.emergentmind.com/topics/redsage