- This paper finds that XGBoost achieves 99.98% accuracy in detecting malicious software from volatility and identifying subtypes of malware using memory analysis with real, executables.
- The study identifies gradient-boosted tree ensembles as effective tools for accurately and quickly detecting malware across several features including service names, implementations, and more.
- The research highlights the challenges and practical limits of separating malware into detailed categories.
This paper reports a systematic evaluation of machine learning classifiers for malware detection using memory dump features from the Canadian Institute for Cybersecurity's MalMemAnalysis-2022 dataset (2602.02184). The work addresses two tasks: binary classification of benign versus malicious software, and four-class classification distinguishing benign samples from ransomware, spyware, and Trojan horse malware. The central result is that an extreme gradient boosting (XGBoost) classifier achieves a testing accuracy and F1 score of 99.98% on the binary task, exceeding the 99.00% accuracy reported by the dataset's original authors, while classifying 50 samples in roughly 37 milliseconds—about an order of magnitude faster than the original stacking ensemble.
Motivation and problem setting
The paper grounds its motivation in the substantial economic and societal costs of malware, citing IBM's estimate of a $4.45 million USD average data breach cost in 2023 and Cybersecurity Ventures' projection of$10.5 trillion USD in annual cybercrime costs by 2025. It also draws on the Pegasus spyware investigations by Amnesty International and Citizen Lab to illustrate the severity of modern obfuscated spyware threats. The technical premise is that static and dynamic analysis methods struggle against obfuscated malware, whereas memory forensics—capturing volatile memory snapshots and extracting features from running processes—offers a more robust signal for detection.
Dataset and methodology
The MalMemAnalysis-2022 dataset contains 58,596 records derived from 2,916 malware samples executed in a virtual machine, with memory dumps captured every 15 seconds up to ten times per sample; SMOTE oversampling balances the benign and malicious classes at 29,298 records each. After removing three constant-valued features (pslist.nprocs64bit, handles.nport, svcscan.interactive_process_services), 52 features remain. The data was split 80/20 with stratification, min-max scaling fitted only on the training subset to avoid leakage, and PCA visualization confirmed that benign and malicious samples are highly separable in two dimensions while the three malware sub-types overlap substantially—a finding that foreshadows the multi-class results.
Nine algorithms were compared under both configurations: Gaussian naïve Bayes, logistic regression, k-nearest neighbours, a multi-layer perceptron, decision tree, random forest, gradient boosting, LightGBM, and XGBoost. Evaluation used five-fold stratified cross-validation with per-fold scaling pipelines, followed by held-out testing subset evaluation across eight metrics including balanced accuracy, MCC, ROCAUC, and log loss.
Results
In cross-validation, all models exceed 99% binary accuracy, with random forest, LightGBM, and XGBoost tied at the top (99.99% accuracy, 100% ROCAUC); XGBoost attains the lowest log loss at 0.0006. On the multi-class task, performance drops sharply—the best model, XGBoost, reaches 87.28% cross-validated accuracy and an F1 score of 80.85%, consistent with the poor sub-type separability observed via PCA.
On the testing subset, XGBoost achieves 99.98% accuracy and F1 score for binary classification, improving on the dataset authors' stacking ensemble (99.00% accuracy, 99.02% F1). For multi-class classification, XGBoost achieves 87.54% accuracy and 81.26% F1; notably, the dataset compilers reported no multi-class baselines, so this appears to be among the first evaluations of that task on this dataset. Per-class metrics reveal the difficulty of sub-type discrimination: ransomware recall is 72.57% and Trojan horse recall is 72.34% for XGBoost, with spyware recall highest at 80.19%. Excluding the dominant benign class, the macro-averaged F1 over malware sub-types is only 75.03%, indicating that headline multi-class figures are inflated by benign-sample performance—an important caveat the paper makes explicit.
Feature importance analysis identifies svcscan.nservices (total number of services) as the most informative feature in both configurations, with svcscan.shared_process_services second in the multi-class case. Preliminary feature selection experiments yielded only slight performance degradation, though this direction was not fully explored.
Timing experiments show the binary classifier processes 50 samples sequentially in 37.3 ms and the multi-class classifier in 43.2 ms, versus approximately 400 ms for the dataset authors' model. At scale, 10,000 samples require 3.49 s (binary) and 6.24 s (multi-class). These timings cover classification only, excluding memory dump capture, feature extraction, and preprocessing.
Limitations
The paper is candid about several constraints. First, the Kaggle version of the dataset lacks some features described in the original paper, such as the API-hook counts, limiting exact comparability. Second, the choice of Trojan horse as a sub-type is taxonomically awkward, since it denotes an implementation style that can overlap with ransomware or spyware rather than a distinct goal. Third—and most consequential—the dataset contains multiple dumps per sample (up to ten, spaced 15 seconds apart), creating a risk of train-test leakage at the sample level; records from the same sample may appear in both subsets, inflating generalization estimates. The author notes this could be mitigated using record identifiers, but only for malicious samples, since benign records lack unique sample identifiers. Finally, hardware differences between the timing setup (a Google Colaboratory instance with 12.7 GB RAM) and the original study complicate speed comparisons, and no resource-consumption analysis beyond classification latency is provided.
Open questions left by the paper include whether sample-level splitting would preserve the near-perfect binary performance, whether feature reduction can cut end-to-end pipeline latency without accuracy loss, and how well models generalize to finer-grained family-level classification (e.g., Conti, 180Solutions).
Conclusion
This work demonstrates that gradient-boosted tree ensembles, specifically XGBoost, provide an effective trade-off between detection accuracy and inference speed on memory-analysis-based malware detection, surpassing the original dataset benchmark on the binary task and establishing a first multi-class baseline. The strong binary results coexist with materially weaker sub-type discrimination (75.03% malware-only macro F1) and unresolved concerns about sample-level data leakage, both of which temper claims about real-world generalizability. The paper positions these findings as a step toward accurate, low-latency obfuscated malware detectors built on volatile memory forensics.