基於內核函數呼叫模式之惡意程式種類辨認方法

Abstract

惡意程式種類辨認方法是用來判斷一隻被測試的惡意程式是不是屬於某特定種類的成員。任何一種辨認方法都必須有能力產生代表各種類的共同行為特徵。然而，現有的產生行為特徵的方式仍存在漏洞，例如：核心層次的Rootkit能夠繞過在分析系統內紀錄有哪些應用程式函式庫被使用的監測方法。在本篇論文中，我們設計了一個能夠產生代表整個種類之惡意程式行為特徵的方法。此方法利用將惡意程式置於虛擬機器中執行，以監視惡意程式的行為。為了讓惡意程式無法繞過本系統的分析，我們藉由在虛擬機器外部設置監控機制，記錄核心函式的呼叫情形。此外，也運用對於整個系統的污染資料流分析，可以得知有哪些被呼叫的核心函式之參數與被測試的惡意程式有關聯。再者，這樣的方式也能夠讓我們追蹤到有跨程序行為的惡意程式，這一個特點是之前與我們目標相同的研究都做不到的。最後將產生的核心函式呼叫記錄轉換成HMM的模型，作為表示惡意程式種類的行為特徵。由評鑑結果顯示，利用本系統產生的行為特徵於辨認惡意程式種類時，能夠達到非常低的漏報率。

Malware family recognition is the process of judging whether a malicious binary program belongs to certain family. In this process, a pattern representing a sequence of malicious behaviors shared among malware in the same family shall be automatically generated. Existing mechanisms such as in-system API profiling can be circumvented by some malware such as kernel-level rootkit. In this thesis, a novel scheme is proposed which generates a unique behavior pattern for each family of malware. In our scheme, malware are executed on a virtual machine. By hooking in-kernel functions underlying the VMM, invocation sequences of a malware program cannot be disguised and therefore are accurately profiled. Our scheme covers the whole-system taint analysis to identify the in-kernel function invocations where parameters are contaminated by the malware being tested. Our scheme also tracks cross-process malware, which is not covered by previous work. Profiled invocation sequences are further converted to HMM-based pattern. The evaluation result shows that our behavior patterns give extremely low false negative in the recognition phase.