Packed Malware Detection Using Grayscale Binary-to-Image Representations

Abstract: Detecting packed executables is a critical step in malware analysis, as packing obscures the original code and complicates static inspection. This study evaluates both classical feature-based methods and deep learning approaches that transform binary executables into visual representations, specifically, grayscale byte plots, and employ convolutional neural networks (CNNs) for automated classification of packed and non-packed binaries. A diverse dataset of benign and malicious Portable Executable (PE) files, packed using various commercial and open-source packers, was curated to capture a broad spectrum of packing transformations and obfuscation techniques. Classical models using handcrafted Gabor jet features achieved intense discrimination at moderate computational cost. In contrast, CNNs based on VGG16 and DenseNet121 significantly outperformed them, achieving high detection performance with well-balanced precision, recall, and F1-scores. DenseNet121 demonstrated slightly higher precision and lower false positive rates, whereas VGG16 achieved marginally higher recall, indicating complementary strengths for practical deployment. Evaluation against unknown packers confirmed robust generalization, demonstrating that grayscale byte-plot representations combined with deep learning provide a useful and reliable approach for early detection of packed malware, enhancing malware analysis pipelines and supporting automated antivirus inspection.