Loading...

Software De-Obfuscation and its Applications to Malware Analysis

Momeni, Behnam | 2018

706 Viewed
  1. Type of Document: Ph.D. Dissertation
  2. Language: Farsi
  3. Document No: 51491 (19)
  4. University: Sharif University of Technology
  5. Department: Computer Engineering
  6. Advisor(s): Kharrazi, Mehdi
  7. Abstract:
  8. Obfuscation transformations complicate a software and make it incomprehensible via syntactical changes. This provides high incentive for malware authors to employ different obfuscation techniques. The practical nature of this problem and lack of common definitions in this area have limited the encountering fronts. Whenever a new obfuscation method becomes known in the wild, ad-hoc deobfuscation solutions follow it, trying to recognize it in details and reverse it step by step. As more obfuscation transformations become readily available for malware developers, this approach becomes more costly and impractical. This rises the perceived need for automated deobfuscation. For example, the virtualization obfuscation can be noted as one of the latest obfuscation transformations which replaces the entire program with an interpreter and runs it without decoding the original instruction in the memory.This dissertation recognizes the importance of this problem and faces it by designing a comprehensive software deobfuscation framework. The proposed framework deobfuscates any kind of obfuscation transformation automatically through a Cook reduction to three subproblems: One for concolic execution of the binary program, another for analysis of its possible execution paths, and last one for solving SMT problem instances. The proposed framework is made without any assumption about the specific obfuscation transformations and so is expected to remain equally effective against future obfuscation techniques.Moreover, an architecture is instantiated out of the proposed framework in order to measure its practical usability and performance besides its theoretical aspects.To evaluate the deobfuscation effectiveness and performance, different standard performance testing programs and a set of virtualization-obfuscated programs are used. In this reference implementation, different practical challenges during the software dynamic analysis are recognized and their mitigations are devised for 32/64 bit Intel hardware architectures and Windows and Linux operating systems.Instrumentation technique is used to realize this dynamic analysis and an architecture is designed to maintain instrumentation scalability while minimizing the code duplication and protecting symbolic executions performance, especially when the number of instrumented instruction types increases
  9. Keywords:
  10. Software Security ; Reverse Engineering ; Instrumentation ; Software De-Obfuscation ; Virtualization Obfuscation ; Malwares

 Digital Object List

 Bookmark

No TOC