Specification-Based Protocol Obfuscation

Abstract: This paper proposes a new obfuscation technique of a communication protocol that is aimed at making the reverse engineering of the protocol more complex. The obfuscation is based on the transformation of protocol message format specification. The obfuscating transformations are applied to the Abstract Syntax Tree (AST) representation of the messages and mainly concern the ordering or aggregation of the AST nodes. The paper also presents the design of a framework that implements the proposed obfuscation technique by automatically generating, from the specification of the message format, a library performing the corresponding transformations. Finally, our framework is applied to two real application protocols (Modbus and HTTP) to illustrate the relevance and efficiency of the proposed approach. Various metrics recorded from the experiments show the significant increase of the complexity of the obfuscated protocol binary compared to the non-obfuscated code. It is also shown that the execution time and memory overheads remain acceptable for a practical deployment of the approach in operation.