Foundation-Sec-8B-Instruct: Cybersecurity LLM

• The model introduces advanced domain adaptation by combining continual pretraining on 25 GiB of cybersecurity data with rigorous instruction tuning and human preference alignment.
• Foundation-Sec-8B-Instruct is a large language model designed for interactive, instruction-following dialogue in cybersecurity, enabling accurate threat analysis and vulnerability mapping.
• The model outperforms general LLMs on key cybersecurity benchmarks, reducing analyst workload with structured outputs and enhancing detection, analysis, and reporting.

Foundation-Sec-8B-Instruct is a LLM specifically designed for interactive, instruction-following dialogue in the cybersecurity domain. Building upon Foundation-Sec-8B—which itself adapts Llama 3.1-8B via continued pretraining on cybersecurity corpora—this model integrates domain-specialized knowledge, advanced instruction-tuning, and alignment with human preferences for high-quality, relevant responses. Released publicly, Foundation-Sec-8B-Instruct is positioned for use as an assistant in Security Operations Centers (SOCs), cyber threat intelligence, vulnerability management, and cybersecurity education.

1. Model Architecture and Domain Adaptation

Foundation-Sec-8B-Instruct maintains the architectural core of Llama 3.1-8B: a transformer with 8 billion parameters and 4096-token contexts. The model architecture is unmodified from its base, consisting of input embeddings, rotary positional encoding, stacked multi-head attention, and feed-forward layers. The critical distinction arises in its training data and post-training pipeline:

• Continual Pretraining: The Foundation-Sec-8B base was subjected to prolonged pretraining on a highly curated multi-stage cybersecurity dataset. This dataset, filtered from 4 TiB of initial data sources to 25 GiB of high-quality content (yielding approximately 5.1 billion tokens), covers authoritative publications (CVE, MITRE ATT&CK, NIST, GDPR, STIX, TAXII, etc.) and technical reports across key subdomains.
• Representation: The model is domain-adapted to encode patterns, vocabulary, and conceptual relationships specific to cybersecurity, such as exploit taxonomies, vulnerability attributions, and regulatory frameworks.

2. Instruction-Tuning and Conversational Alignment

The transition from the domain-specialized, non-interactive Foundation-Sec-8B to the instruction- and dialogue-capable Foundation-Sec-8B-Instruct was achieved in two main steps:

• Supervised Fine-Tuning (SFT): SFT was conducted with both synthetic and manually curated cybersecurity instruction data. Instruction prompts were diverse, mirroring real-world SOC queries, investigation workflows, and report generation. The focus was on response format, clarity, and contextual relevance.
• Alignment with Human Preferences: Direct Preference Optimization (DPO), in contrast to more complex RLHF variants, was used to improve conversational quality and ensure outputs align with human expectations. Multiple rounds of human preference ranking and testing were incorporated.

Decontamination procedures explicitly targeted benchmark leakage, ensuring evaluation tasks such as CTIBench-MCQA, CTIBench-RCM, and CyberMetric were not present in the fine-tuning dataset.

3. Cybersecurity Datasets and Evaluation Benchmarks

The model was tuned and evaluated with benchmark sets constructed for the unique requirements of cybersecurity:

Benchmark	Task Type	Description
CTIBench-MCQA	MCQ Answering	CTI/MITRE ATT&CK, security taxonomy questions
CTIBench-RCM	Root Cause Mapping	CVE-to-CWE root cause resolution
CTIBench-VSP	Severity Prediction	CVSS vector estimation from vulnerability text
CyberMetric-500	Diverse QA	Threat reports, attack paths, compliance, etc.
SecBench/SecEval	Mixed (QA, generation)	Broad security task coverage

A representative evaluation formula for CTIBench-VSP (severity prediction) normalizes CVSS score deviation:

$\text{CVSS\_Score} = 1 - \frac{\text{MAD}}{10}$

where MAD is the mean absolute deviation of the predicted CVSS base score.

4. Performance Metrics and Comparative Analysis

Foundation-Sec-8B-Instruct demonstrates superior or highly competitive results compared with both general and cybersecurity-focused LLMs:

• On CTIBench-RCM (root cause mapping), it achieves state-of-the-art performance, at times outperforming larger models such as GPT-4o-mini.
• The model convincingly surpasses Llama 3.1-8B-Instruct on all evaluated cybersecurity tasks, while matching it on general instruction-following metrics (e.g., IFEval, AlpacaEval).
• For conversational fluency and context maintenance, Foundation-Sec-8B-Instruct wins nearly twice as often as baseline models in human preference evaluations.
• On general-purpose tasks (MMLU, HumanEval, GSM8K, MATH, BigBenchHard), the model retains broad competence, evidencing minimal catastrophic forgetting from its domain adaptation.

5. Instruction-Following, Dialogue, and Persona Specialization

A notable aspect of Foundation-Sec-8B-Instruct is its dual ability to:

• Respond accurately to complex, technical security prompts with structured, data-driven outputs (such as vulnerability causal chains or severity assessments).
• Engage in open-ended, context-aware conversations, with adaptation for roles such as SOC analysts or threat intelligence experts (validated via PersonaGym).
• Output both concise answers for triage scenarios and detailed explanations suitable for reporting and education.
• Employ prompt-driven structured outputs using LaTeX and markdown, and interpret input following NIST/SP 800-61, MITRE ATT&CK, or other formal frameworks where applicable.

6. Integration into Cybersecurity Workflows and Usability

Foundation-Sec-8B-Instruct is intended as a “virtual assistant” codenamed Metis by default. The model:

• Supports direct, natural-language querying for rapid triage, investigation, and reporting by analysts.
• Generates remediation actions, code snippets for infrastructure automation, and maps vulnerabilities to MITRE and CAPEC taxonomies.
• Reduces analyst workload by pre-summarizing events, organizing multi-source alerts, and suggesting incident response paths.
• Facilitates comprehensive training, onboarding, and continuous education by providing up-to-date domain-specific knowledge and explanations.

Model deployment: It is released publicly at https://huggingface.co/fdtn-ai/Foundation-Sec-8B-Instruct for research and practical integration.

7. Safety, Alignment, and Future Recommendations

The safety and alignment mechanisms of Foundation-Sec-8B-Instruct are basic relative to top commercial models, relying primarily on standard alignment protocols and the option for pairing with external guardrails (e.g., LlamaGuard). The report notes the implementation of rigorous decontamination as a best practice for regulatory and safety assurance.

Potential future work includes:

• Extension to enhanced safety mechanisms (e.g., prompt injection defense, advanced red teaming).
• Broader coverage across evolving threat intelligence schemas and multi-modal security data (logs, telemetry, etc.).
• Integration with interactive workflows, tool-calling, and real-time reasoning for more autonomous agentic defense systems.
• Research into efficiency, parameter scaling, and robustness against adversarial or out-of-distribution cyber prompts.

---

Foundation-Sec-8B-Instruct exemplifies progress in domain-adapted instruction-tuned models for specialized verticals like cybersecurity. It demonstrates that combining large-scale, domain-specific pretraining with advanced instruction and alignment procedures yields models that are both expert in specialized content and broadly usable as interactive assistants for technical professionals.