Generic Unpacking of Self-modifying, Aggressive, Packed Binary Programs

Abstract: Nowadays most of the malware applications are either packed or protected. This techniques are applied especially to evade signature based detectors and also to complicate the job of reverse engineers or security analysts. The time one must spend on unpacking or decrypting malware layers is often very long and in fact remains the most complicated task in the overall process of malware analysis. In this report author proposes MmmBop as a relatively new concept of using dynamic binary instrumentation techniques for unpacking and bypassing detection by self-modifying and highly aggressive packed binary code. MmmBop is able to deal with most of the known and unknown packing algorithms and it is also suitable to successfully bypass most of currently used anti-reversing tricks. This framework does not depend on any other 3rd party software and it is developed entirely in user mode (ring3). MmmBop supports the IA-32 architecture and it is targeted for Microsoft Windows XP, some of the further deliberations will be referring directly to this operating system.