Malware Detection Through Disassembled Function Analysis

March 20th, 2015
11:15am – 12:05pm
Campbell Hall Room: CH405

Advisor – Robert Hyatt
Supervisory Committee:
Robert Hyatt (Chair; UAB CIS)
Steven Bethard (UAB CIS)
Alan P. Sprague (UAB CIS)

ABSTRACT

With an abundance of new malware being discovered daily, antivirus software vendors have been forced into an industry of reaction. Using a low ratio of malware analysts to malware, plus antiquated techniques, the antivirus industry floods the market with products with regrettably low detection rates, often only marginally outperforming competitors.

This thesis presents a new function-edge-based set detection and identification method which significantly outperforms traditional antivirus techniques. The technique used reduces a malware binary to its fundamental assembly code and performs functional analysis using a newly created function hashing technique. By combining calling and called functions into pairs, function hash edges are created. Each malware sample can be described as the set of function hash edges it generates. Multiple malware samples are then grouped into unique parent-child function trees. Each tree produced using this technique represents the respective malware family the samples belong to. These trees not only help correctly identify which family of malware a sample belongs to, but the tree itself can be used as a detection mechanism for any future versions of these malware samples.

This new detection method provides many consumer level benefits because it can learn based on functional signature sets. The technique can protect from future generations of a previously detected form of malware, therefore giving it an advantage over every other antivirus product on the market. In addition to these, and many other consumer level benefits, this technique can also be used as an investigative or analytical tool in malware research and criminal investigations. It is also useful when questioning author attribution and code sharing. Additional implications of this new technique include uses as a tool for source code theft investigation and unauthorized use of compiled libraries?