ForensicsData: LLM-Powered Forensic QA Resource
- ForensicsData is a digital forensics dataset providing a structured Question-Context-Answer format derived from real malware analysis reports.
- It employs an LLM-driven generation pipeline and a rigorous three-stage validation framework to ensure high JSON validity and forensic terminology alignment.
- The dataset supports forensic question answering and report understanding, serving as a reproducible resource for LLM-based forensic research.
ForensicsData is a digital forensics dataset designed specifically for LLM applications. It is presented as a source-grounded synthetic Question-Context-Answer dataset derived from real malware sandbox reports and intended to address a persistent bottleneck in digital forensics research: the lack of realistic, publicly usable datasets caused by privacy, legal, and ethical constraints around sharing forensic evidence. The authors describe it as “the first publicly available, structured Q-C-A resource specifically designed for digital forensics applications,” built from actual malware analysis reports to support forensic question answering, structured report understanding, and LLM-based forensic tooling (Chakir et al., 31 Aug 2025).
1. Dataset identity and research problem
ForensicsData is centered on a minimal Question-Context-Answer schema:
$0.00059/\$1
The dataset contains “more than 5,000 Q-C-A triplets,” and the paper also refers to “approximately 5,000 initial Q-C-A triples.” The authors frame this as a response to a specific gap: existing digital forensics datasets do not provide a structured, LLM-ready forensic QA resource grounded in malware behavior, and prior cybersecurity or synthetic-data benchmarks do not address malware behaviors in structured Question-Context-Answer format as comprehensively as ForensicsData (Chakir et al., 31 Aug 2025).
The paper repeatedly defines the target knowledge space in forensic rather than generic cybersecurity terms. Each triplet may include malware metadata, behavioral patterns, indicators of compromise, tactics/techniques/procedures, and mitigation information. The prompt design targets five dimensions: malware identification and metadata; technical indicators of compromise; suspicious behavioral patterns and techniques; malicious behavioral patterns and techniques; and impact assessment and mitigation strategies. This makes the dataset less a general QA corpus than a structured supervision layer over malware execution evidence.
A notable structural feature is what the paper does not provide. Beyond the core Q-C-A JSON schema, the released representation does not include a formal ontology, a label taxonomy, or per-triplet provenance fields. Upstream report content contains hashes, verdicts, tags, IOCs, behavior activities, behavior graphs, network indicators, and process trees, and the authors state that these are used to ground generation; however, the final schema shown in the paper remains the three-field Q-C-A structure. The paper also does not print a full example triplet. For reuse, this means semantic intent is clearer than annotation style.
2. Source corpus, malware coverage, and forensic semantics
The source material came from 1,500 execution reports collected from the public ANY.RUN malware analysis platform, described as an interactive sandbox service with over 10 million user-contributed malware and benign traces. The reports were restricted to samples submitted in 2025 so that the resulting dataset would reflect contemporary malware behavior. The paper says the corpus covered 15 malware families plus benign samples, while the enumerated list names qbot, trickbot, sality, formbook, hawkeye, amadey, gandcrab, wannacry, nanocore, xworm, remcos, agenttesla, lumma, and emotet, plus a benign class; this is one of several small internal inconsistencies in the description (Chakir et al., 31 Aug 2025).
The malware families are also grouped by malware type: banker, botnet, infostealer, infostealer/loader, ransomware, RAT, stealer, and trojan. The family distribution table reports the following counts:
| Family | Count |
|---|---|
| agenttesla | 73 |
| amadey | 40 |
| emotet | 33 |
| gandcrab | 100 |
| lumma | 47 |
| nanocore | 76 |
| sality | 83 |
| trickbot | 64 |
| formbook | 86 |
| hawkeye | 54 |
| qbot | 69 |
| remcos | 69 |
| wannacry | 36 |
| xworm | 72 |
| benign | 150 |
The paper states that uniform distribution was a design goal to reduce class imbalance, but the reported table is only moderately balanced rather than perfectly uniform. That matters because downstream evaluation over malware families may inherit variation in question frequency and semantic coverage.
The upstream extraction targets the sections judged most relevant for forensic interpretation: metadata, IOCs, behavior activity, behavior graphs, network indicators, and process trees. As a result, the dataset encodes questions about what a sample is, what it does, what files or processes it touches, how it communicates over the network, what suspicious behaviors it exhibits, and how it may be mitigated. This suggests that ForensicsData is best understood as a natural-language supervision layer over malware execution telemetry rather than a raw malware artifact corpus.
3. Construction workflow and generation architecture
The construction pipeline begins with raw report collection and parsing. The reports existed in HTML and semi-structured XML, and the authors parsed them using a custom extraction pipeline built with BeautifulSoup and lxml. Preprocessing then applied three explicit steps: noise removal, standardization, and de-duplication. Background processes unrelated to malware behavior were removed; files with incomplete or meaningless content were filtered out; extracted content was transformed into a consistent JSON structure with standardized field names and value formats; and duplicate malware reports were removed using SHA-256 hash comparison. The output of this phase is described as a “clean, structured dataset in JSON format” ready for question and answer generation (Chakir et al., 31 Aug 2025).
Q-C-A generation is LLM-driven. Each structured JSON report is provided to an LLM with instructions to generate contextually grounded question-context-answer triplets across the five forensic dimensions. The generation policy is fixed at five Q-C-A triples per report. Five models were used in a stratified allocation scheme, with 20% of the report corpus assigned to each model and malware-family representation balanced across models. Since roughly 300 reports were processed per model, the paper states that this yielded about 1,000–1,200 triples per model and around 5,000 total triples.
The generation models and their reported operational characteristics are as follows:
| Model | Context / latency | JSON validity / token cost |
|---|---|---|
| Qwen-QWQ-32B | 32k, 250 s | 99.3%, \$0.00029/\$0.00039 |
| LLaMA 3–70B | 8k, 364 s | 98.6%, \$0.00059/\$0.00079 |
| Gemini 2.0 Flash | 1M, 425 s | 99.8%, \$0.0001/\$0.0004 |
| Mistral 8B | 32k, 750 s | 97.2%, \$0.0001/\$0.0001 |
| DeepSeek V3 | 128k, 1667 s | 99.5%, \$0.00038/\$0.00089 |
Parameter counts are given for all except Gemini: approximately 7.3B for Mistral 8B, 70B for LLaMA 3–70B, 33B for DeepSeek V3, and approximately 32B for Qwen-QWQ-32B. The paper further states that Qwen-QWQ-32B and LLaMA 3–70B had the highest throughput due to GroqCloud infrastructure, Qwen offered the best cost-performance tradeoff, Mistral and Gemini were the most cost-efficient, LLaMA 3–70B had the highest output cost, and all models exceeded 97% JSON validity. At the same time, the authors note that Mistral 8B, despite often being syntactically correct, “occasionally failed to cover all semantic meaning of prompts,” which they interpret as weaker instruction following.
The paper emphasizes prompt engineering but does not reproduce the exact prompt text. It also mentions error handling and retry mechanisms for generation failures or unexpected output, without pseudocode or low-level implementation detail. That omission limits exact pipeline reimplementation even though the architectural sequence is clear.
4. Validation framework and reported quality
ForensicsData uses a three-stage validation framework combining structural validation, semantic redundancy filtering, and LLM-based semantic review. First, schema validation is performed with Pydantic to ensure that every entry has the expected keys and string values. Second, semantic deduplication uses sentence embeddings from all-MiniLM-L6-v2 and cosine similarity. Third, an LLM-as-Judge stage uses Gemini Advanced 2.0 Flash to review each Q-C-A triple for logical validity and realism, relevance, completeness, and consistency or absence of hallucinations (Chakir et al., 31 Aug 2025).
The semantic deduplication stage is formalized in the paper. For a question , the embedding model produces , where the paper states . Similarity is computed as
with a threshold of $0.90$. Questions above that semantic similarity threshold were to be flagged and removed.
The reported quantitative results are strong. Pydantic schema validation passed 98.68% of all entries without modification; the remaining entries were auto-corrected or discarded. After applying all-MiniLM-L6-v2 embeddings and the $0.00059/\$0 cosine threshold, the paper reports that “no Q-C-A triples were marked as duplicates,” which the authors interpret as evidence of semantic diversity. For LLM-as-Judge validation over 5,000 triples, the paper reports Logical Validity 5,000/5,000 (100.0%), Relevance 5,000/5,000 (100.0%), Completeness 4,850/5,000 (98.0%), and Hallucination-Free 5,000/5,000 (100.0%). The only nontrivial weakness identified in the discussion is completeness: roughly 2% of answers only partially addressed the question, typically by omitting technical details such as file paths, process IDs, or behavior descriptions.
The abstract states that “Among the models evaluated, Gemini 2 Flash demonstrated the best performance in aligning generated content with forensic terminology.” The body supports this only indirectly. The strongest evidence presented is Gemini’s 99.8% JSON validity, its use as the LLM judge, and repeated qualitative descriptions of strong instruction adherence and quality control. The paper does not provide a dedicated terminology-alignment metric or a table scoring terminology fidelity across models. Accordingly, the claim is best read as an empirical conclusion of the authors rather than a result tied to a separate formal benchmark.
The validation framework also has clear omissions. The paper reports no human inter-annotator agreement statistics, no expert panel protocol, no ablation study over prompt variants, no comparison between pre- and post-filter quality, and no downstream task experiment showing improved task performance after training on ForensicsData. The LLM-as-Judge process is central, but its prompt and calibration are not published.
5. Downstream uses, access conditions, and evidentiary cautions
The paper explicitly or implicitly targets several use cases. The most direct is question answering over malware analysis reports, because the Q-C-A structure is already a supervised QA format. The conclusion also identifies retrieval-augmented generation as future work, and the context field naturally supports retrieval-plus-answering setups. Additional intended uses include report understanding, evidence summarization, malware investigation and evidence analysis, and benchmark creation for reproducible experimentation with LLMs (Chakir et al., 31 Aug 2025).
These uses are plausible because the dataset aligns with common analyst questions about metadata, indicators of compromise, behavior, TTP-like patterns, and mitigation. However, the paper does not report any downstream experiment such as fine-tuning a model on ForensicsData and measuring QA accuracy, retrieval performance, or classification improvements. The support for these applications is therefore primarily by design rationale rather than demonstrated task gains.
The access story is incomplete. The paper strongly implies public availability and calls the dataset publicly available, but the text provided does not include a repository URL, DOI, download link, release package description, or explicit license. Provenance is clear—public ANY.RUN reports—but redistribution rights for derived data are not discussed. This is practically important because the dataset is framed as public and reproducible while the legal terms of reuse remain unspecified in the paper text.
On ethics and safety, the authors position Q-C-A generation as a privacy-preserving alternative to sharing raw forensic evidence. They also emphasize “ethical safety” by generating descriptive textual triplets rather than executable code or malware binaries. The dataset is therefore intended to support security research without distributing harmful artifacts. At the same time, the paper itself acknowledges broader challenges around factual consistency and legal admissibility in digital forensics. Since hallucination review is performed by another LLM rather than independent forensic experts, caution remains warranted in investigative or courtroom settings.
Several limitations follow directly from the source design. The corpus is limited to 2025 reports from a single platform, focused on malware sandbox outputs, and restricted to the listed families plus a benign class. It likely reflects sandbox-visible behavior and ANY.RUN reporting conventions. This suggests external validity is constrained: ForensicsData is not a comprehensive representation of all digital forensics scenarios, nor of all malware ecosystems.
6. Position within the forensic data landscape
ForensicsData occupies a distinct position among recent forensic data resources. DFIR-Metric, for example, is a benchmark for evaluating LLMs in Digital Forensics and Incident Response with three components: 700 expert-reviewed multiple-choice questions, 150 CTF-style tasks, and 500 NIST-derived practical analysis cases (Cherif et al., 26 May 2025). NapierOne, by contrast, is a modern mixed-file corpus intended for ransomware detection and forensic analysis research, comprising nearly 500,000 unique files across 100 subsets and 44 file types (Davies et al., 2022). A deterministic preprocessing framework for heterogeneous network evidence addresses schema normalization, temporal normalization, and provenance tracking for IDS, IoT, and enterprise logs rather than natural-language QA (Chaudhary et al., 10 Jun 2026).
Against that background, ForensicsData is neither a raw artifact corpus nor an execution-trace benchmark. It is a natural-language supervision resource grounded in structured malware analysis reports. This suggests a distinct layer in the forensic data stack: file-centric corpora such as NapierOne support byte-level or file-type analysis; preprocessing frameworks address evidentiary normalization and provenance for heterogeneous logs; evaluation benchmarks such as DFIR-Metric stress reasoning and operational competence; ForensicsData contributes report-grounded Q-C-A supervision over malware behavior.
That positioning also clarifies a common misconception. ForensicsData is not a replacement for raw forensic evidence, full incident-response corpora, or human expert validation. Its contribution is the conversion of structured malware reports into validated question-context-answer triples through what the paper calls a “scalable LLM-driven annotation pipeline” and a “comprehensive validation framework.” A plausible implication is that its strongest value lies in research settings where one needs grounded forensic language supervision, reproducible QA experiments, or a common resource for malware-report understanding—provided its malware-centric scope, single-platform provenance, and automated validation regime are kept in view.