Abstract
Malware programs, such as viruses, worms, Trojans, etc., are a worldwide epidemic in the digital world. Studies and statistics show that malware volume has increased tremendously year after year in the past decade. Due to the rapid malware growth in recent years, the malware detection approaches have been experiencing a paradigm shift from the laborious manual analysis, signature-based approach to a data-driven, machine learning-based approach. This thesis presents a semi-automated malware detection solution using machine learning. It notifies the user if the application she downloaded behaves differently than what she expected at download time. The hypothesis is that in spite of millions of currently downloadable executables on the Internet, almost all of them provide functionalities from a limited set. Additionally, because of each functionality, e.g., text editor, requires particular system resources, it exhibits a unique system-level activity pattern. During an on-line training process, the system creates a profile dictionary of various functionalities. This profile dictionary is then used to warn the user if she downloads an executable whose observed activity does not match its advertised functionality. The proposed solution is deployed as a cloud service. It includes a multi-model classification module that takes into account the time-variant property of functionality and behavior features from the system level. Since static features are easier to be extracted, but it is less effective compared to dynamic behavioral features; Dynamic behavioral features are much more pricey to collect, but it is very effective. However, the effectiveness of dynamic behavioral features depends on the length of analysis; thus accurate detection requires more time and computing resources. Existing works focused on improving the model accuracy by discovering distinctive features in static analysis or dynamic analysis. Despite these recent advances, to implement an efficient and user interactive malware detection system remains challenging. The uniform length of dynamic analysis adopted by previous research failed to capture the ongoing evolvement of malware behaviors. Extending the duration of dynamic analysis, although advantageous in improving the accuracy, is nevertheless both resource intensive and time-consuming. There exist a need to balance the accuracy and resource consumption in a practical system. We modeled the system using contextual multi-armed bandit framework and presented two on-line learning algorithms that, for each sample to be analyzed ensures the high probability of selecting the best classifier. To that end, we define Quality of Experience (QoE) as a user metric in the framework to balance the accuracy and efficiency trade-off and use static file feature as the context to facilitate the classifier selection. Our experiment results using 2000 real malware samples show that context specification of classifiers can be discovered over time to create a strong detector given K weak detectors.