Foundation-Sec-8B Cybersecurity LLM

• Foundation-Sec-8B is an open-access cybersecurity LLM built on Llama 3.1–8B, designed through continued pretraining on a rigorously curated corpus of cybersecurity texts.
• The model demonstrates significant improvements on domain-specific benchmarks, achieving a 6% gain on MCQA and a 14% boost on mapping tasks while preserving general language abilities.
• It underpins practical applications such as SOC automation, threat intelligence, and secure engineering workflows, catalyzing AI-driven cybersecurity innovations.

Foundation-Sec-8B is an open-access, cybersecurity-specialized LLM derived from the Llama 3.1–8B architecture, explicitly designed to address the unique representational, benchmarking, and workflow challenges present in cybersecurity applications. It is engineered through continued pretraining on a carefully curated corpus of cybersecurity documents, resulting in strong performance gains on multiple domain-specific benchmarks without catastrophic degradation in general language understanding. The model targets operational tasks such as security operations center (SOC) automation, threat intelligence, and cybersecurity engineering enablement, and it is positioned to catalyze AI-driven advances for both public and private sector cybersecurity applications (Kassianik et al., 28 Apr 2025).

1. Architecture and Model Design

Foundation-Sec-8B employs the Llama 3.1–8B transformer backbone, preserving core architectural features such as multi-head self-attention, layer normalization, and deep feedforward blocks. No architectural changes are made relative to baseline Llama 3.1–8B: the primary innovation lies in continued pretraining on a specialized data corpus.

The pretraining is executed with the AdamW optimizer and a cosine learning rate decay schedule: $$\text{lr}(t) = \text{lr}_{\min} + 0.5\cdot (\text{lr}_{\max}-\text{lr}_{\min}) (1 + \cos(\pi t / T))$$ where $t$ is the current training step and $T$ is the total number of steps. Training sequence length is 4096 tokens to accommodate long-context cybersecurity artifacts.

For efficient large-model deployment and evaluation, Foundation-Sec-8B integrates vLLM-style paged attention for memory management.

2. Specialized Cybersecurity Training Data

A prominent differentiator for Foundation-Sec-8B is the curation and use of a high-fidelity, domain-text corpus. The data collection pipeline consists of:

• Wide-net scraping: Large-scale gathering (>4 TiB) using a relevancy filter initialized by an ~800 term expert-curated cybersecurity keyword/acronym list.
• Targeted scraping: Focused extraction from unique high-quality cybersecurity sources.
• Transformer-based relevancy classification: A lightweight classifier (F1=0.924 on a hand-labeled set of 26k documents) is deployed to reduce false positives.
• Rigorous cleaning: Language ID filtering, code snippet handling, regex-based quality filtering, and deduplication using n-gram Bloom filters.
• Data balancing: High-quality data, especially on Tactics, Techniques, and Procedures (TTPs), is upsampled to ensure domain coverage.

This processing yields 5.1B final tokens (99% train, 1% test). The process is illustrated as a pipeline (see paper Fig. "images/pipeline.png").

3. Performance Benchmarks and Evaluation

Foundation-Sec-8B is evaluated on established and new cybersecurity benchmarks, namely MCQA (e.g., CTIBench-MCQA, referencing NIST, GDPR, MITRE ATT&CK, CAPEC standards) and RCM (mapping CVE descriptions to CWE IDs).

Key benchmark results:

Benchmark	Metric	Llama 3.1–8B	Foundation-Sec-8B	Relative Gain
CTIBench-MCQA	MCQA accuracy	Base	+6%	Significant improvement
CTIBench-RCM	Mapping accuracy	Base	+14%	Significant improvement
MMLU (general)	General accuracy drop	≈0	–2.4 points	Modest reduction

Comparisons with larger models (Llama 3.1–70B, WhiteRabbitNeo-V2–70B, GPT‑4o-mini) show that Foundation-Sec-8B matches or surpasses performance on certain cybersecurity-specific tasks. The performance drop on generic knowledge benchmarks is limited, indicating that the model does not suffer catastrophic forgetting due to specialization.

4. Operational Applications and Implications

Foundation-Sec-8B’s design is driven by practical application in operational cybersecurity. The paper reports:

• SOC Acceleration: The model is trialed for alert triage, threat summary, incident timeline generation, and analyst report drafting, showing improvements in time-to-triage and incident accuracy.
• Proactive Threat Defense: Automatic extraction of TTPs from CTI and hypothesis generation for attack graphs from asset inventories.
• Engineering Enablement: Mapping security policies, configuration validation, and checking for secure defaults in development and deployment workflows.

Fine-tuning for domain-specific tasks such as MITRE ATT&CK Technique extraction yields >10% improvement over non-specialized LLMs.

The public release of Foundation-Sec-8B democratizes access, enabling progress in both academic and industrial settings.

5. Methodologies and Data Processing Innovations

Several methodological choices in pretraining distinguish Foundation-Sec-8B:

• Relevancy filtering: Use of a transformer classifier for precision filtering of domain text.
• Deduplication: Use of n-gram Bloom filters for aggressive removal of near-duplicate web documents—a common source of bias in uncurated corpora.
• Upsampling of high-fidelity data: Preferential weighting for documents with known TTP labels ensures representation of rare but operationally critical concepts.

These strategies collectively yield a clean, high-coverage dataset that efficiently induces domain-specific knowledge.

6. Limitations, Impact, and Future Directions

While Foundation-Sec-8B achieves strong results in cybersecurity domains, there are caveats:

• A ~2.4 point drop in generic tasks (e.g., MMLU) suggests a trade-off, though catastrophic generalization loss is not observed.
• The model’s capacity ceiling (8B parameters) may limit advanced reasoning or multi-modal capability compared to very large models.
• The dependency on corpus quality means domain drift or rapid evolution in cybersecurity may require regular retraining.

Planned future directions include:

• Scaling model size/capacity and data corpus breadth.
• Expanding to code-centric tasks such as automated secure code generation.
• Integration with tool-calling frameworks and agentic systems for interactive threat analysis.
• Further refinement of continued pretraining using synthetic augmentation or advanced prompt-tuning techniques for optimal domain retention.

7. Context within LLMs for Security

Foundation-Sec-8B is representative of a new class of domain-specialized LLMs that combine state-of-the-art model architectures (such as Llama 3.1) with highly filtered and upsampled domain corpora. Its methodological rigor—especially in data curation and evaluation—bridges a key gap between general-purpose LLMs and the requirements of enterprise-grade cybersecurity tools. Its public release is expected to accelerate downstream research in AI-driven cybersecurity and provides an empirical benchmark for future domain-adapted LLMs.

---

In summary, Foundation-Sec-8B establishes a high-performance, cybersecurity-centric LLM through targeted continued pretraining on a rigorously filtered corpus, which yields strong domain performance with minimal general capability loss. Its operational applicability, robust evaluation, and public release serve as a foundation for ongoing innovation and adoption of LLMs in the cybersecurity domain (Kassianik et al., 28 Apr 2025).