RMAS

(Run-Time Malware Analysis System) A Framework for Malware Analysis and Malware Detection

Roberto Sponchioni

Sommario
Introduction ....................................................................................................................................................... 3 Architecture ....................................................................................................................................................... 3 Static Analysis .................................................................................................................................................... 4 Static Analysis Conclusion ............................................................................................................................ 5 Dynamic Analysis ............................................................................................................................................... 5 Dynamic Malware Detection Module ........................................................................................................... 9 Dynamic Analysis Conclusion .................................................................................................................... 10 Case Study ....................................................................................................................................................... 11 Scenario 1 .................................................................................................................................................... 11 Scenario 2 .................................................................................................................................................... 11 Scenario 3 .................................................................................................................................................... 11 Future Work..................................................................................................................................................... 12 Conclusion ....................................................................................................................................................... 12 Bibliography ..................................................................................................................................................... 13

Introduction
The malware is a threat for our systems because it can hit and retrieve our confidential information by using different attacking vectors. The security of modern computer systems depends on the ability by the users to keep software, OSes and antivirus products up-to-date. Since new viruses or new viral strains are released every day, the antivirus vendors have to update their software and their signatures and then distribute them. Conventional anti-malware programs rely on static signature to detect malware, but malware writers improve their codes to make them undetectable and stay one step ahead of static-signature-based detection. The Dynamic Malware Detection Systems are not new concepts, but my project is based on the development and the implementation of a system, called RMAS (Run-Time Malware Analysis System) in order to allow analysts to make dynamic analysis of new malware, to understand the malware behavior and produce a graphical dangerousness level of the analyzed program, by using just one modular tool. RMAS has been developed also because sometimes there are malware samples that the company has to analyze within the organization perimeter for privacy or policy reasons, to prevent leakage of confidential data. For example, malware samples could be gathered from sources such as honeypots of a specific network or from investigations and lots of these software cannot be analyzed from external analysis software, this is the main reason why I developed RMAS. The system has also been developed in order to realize an automated analysis thus limiting human intervention.

Architecture
RMAS is a modular framework, still under development even if it worked correctly on lots of samples analyzed. Its design allows technical users to integrate new external components; and this makes RMAS a flexible and an extensible system; the Figure 1, shows the overall design of the project. New malware samples, could be introduced in RMAS easily by the analyst and, after that, the system can extract the information concerning the malware. The framework consists of 3 modules:

Figure 1 RMAS Architecture

1. Static Analysis module, that provides static information, such as file hash, antivirus reports, PE structure, file entropy, Packer Signature, and strings. 2. Dynamic Analysis module, that extracts the program behavior, by using a DLL, that will be injected in every new thread created by the malware, and a kernel driver that intercepts system calls made by the malware. 3. Detection Engine, through a Database of dynamic signature can analyze the malware behavior, and after matching the behavior with the signatures in the database, it can produce an HTML report of the analyzed program. RMAS has been tested on MicrosoftWindows XP, Service Pack 3, and currently new Windows system are not supported. The system is able to trace the behavior of the software both in kernelspace and user-space, in order to obtain a huge amount of high quality information, that will be parsed subsequently with the RMAS Detection Engine.

Static Analysis
Static analysis is the first process that is carried out by RMAS, without actually executing the file, just parsing it and extracting static information. This analysis is used to profile the malware and, by using different modules, RMAS can: Calculate file hashes, such as MD5 and SHA1, used to identify univocally each analyzed executable. Scan through different antivirus engines by which it is possible to determine if the file has already known as a malicious code. Extract PE Information: such as Imported Symbols, Machine Type, Packers/Crypto Signatures Read embedded strings Every task presented before is performed by the following modules, each implemented in RMAS: MD5/SHA1 module, which avoids duplicated malware sample

PE File Structure module, which extracts information such as imported DLL functions, PE Sections and their sizes, etc. Antivirus module, which scans the sample using several antivirus engines; the positive results will be presented in the report. Packer detection module, which can extracts packer information through the PEFILE library, by Ero Carrera, by using a signature database.

 Entropy Analysis module, which measures the uncertainty in a series of bytes so that the system can predict a presence of a packer for example. Strings external tool, provided by Mark Russinovich, which can help the framework by extracting strings from the executable. TrID external tool, provided by Marco Pontello, which can help to identify file types by using static signature matching.

Static Analysis Conclusion

As said before, the RMA System is completely modular and this makes it flexible and extensible allowing to skip the execution of a module. With the preliminary static analysis it is possible to extract valuable information that could shape the picture of the malware.

Figure 2 Static Analysis Modules

Dynamic Analysis
Thanks to the static analysis module, RMAS is able to show substantial information to the analysts, concerning the file, such as dependencies, strings, antivirus signature, packer/crypto signature, etc. At this point the analyst can start a new step of the analysis, also called behavioral analysis, by which it is possible to execute and to monitor the software behavior and to see which side effects are involved in the execution of the code. The behavior analysis is the core of the system and it is the most important module of the RMAS. Currently, the run-time analysis is performed on Windows PE executable, but the idea is to improve the framework in order to provide analysis of PDFs, Microsoft Office Documents and so on. As described before RMAS depends on Windows Systems but it could be deployed also on real systems or on VMs. The Dynamic Analysis has been developed entirely in C Language, and it implements the following modules: 1. DLL, called DMADll that stands for Dynamic Malware Analysis DLL, by which it can trace Registry operations, FileSystem operations, Network, Process, Strings, Services. 2. Kernel Module Driver that intercepts the system calls. The driver can trace Registry operations, FileSystem operations and Processes.

The Microsoft Windows API (Application Programming Interface) provides services used by all programs in a Windows-based OS, to communicate with the Operating System. API calls made by a malicious software can provide significant information regarding the nature and the purpose of the program examined, for example file system, network or registry access. Furthermore, by monitoring API calls, the analyst can observe the work flow of the program, and the interaction with the System. This information is a useful roadmap to understand the malware behavior and it could be subsequently used to detect the anomaly-based behavior. Using API hooking in user-space allows the analysts to trace and to monitor all relevant system calls and generate automatically, a human-readable reports with useful, helpful and high quality information that successively could be analyzed automatically or by hand from a malware analyst. The reports describe: The files the malware sample creates, modifies, copies, moves and so on. The changes the malware sample performs on the Windows Registry. The network connections opened, information sent and retrieved. The process that it creates.

The services that it creates, starts and stops. API Hooking is carried out by the DLL that is injected in every new thread created. By hooking in kernel-space the function NtCreateThread(), RMAS is able to check whenever a new thread is created and successively it can inject the DLL in the new thread. When the DLL is loaded in the malware process context it can do its job by dumping every function and every parameter in a log file. In addition to monitoring the activity on the file system, and on the registry is very interesting also to trace network traffic to and from the system. Capturing network behavior could be useful to identify which connection has been made by the malware, which services are requested, and which data are sent or retrieved from servers. Malware usually interacts with external servers to gather further exploits, or gather other malware from remote sources or could signify an interaction with command and control servers. Monitoring network activity is carried out by API Hooking in user-mode. Examples of traced API are: DNS Query Connect Bind Send Recv

+----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Kernel32.dll] - CopyFileExW Existing File Name: C:\DOCUMENTS AND

SETTINGS\ADMINISTRATOR\DESKTOP\EXPLORE.EXE New File Name: C:\WINDOWS\SYSTEM32\testing.exe +----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Kernel32.dll] - SetFileAttributesW File Name: C:\WINDOWS\SYSTEM32\testing.exe Attributes: 0x2 +----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Advapi32.dll] - RegSetValueExA HKEY: OTHER - (0x90) Value: Winsock32 driver Type REG_SZ Data: testing.exe Handle Length: 108 Handle Type: Key Handle Name: \REGISTRY\USER\S-1-5-21-790525478-854245398-1177238915500\Software\Microsoft\Windows\CurrentVersion\RunOnce +----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Advapi32.dll] - RegSetValueExA HKEY: OTHER - (0x90) Value: Winsock32 driver Type REG_SZ Data: testing.exe Handle Length: 63 Handle Type: Key Handle Name: \REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run +----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Kernel32.dll] - CopyFileExW Existing File Name: C:\DOCUMENTS AND SETTINGS\ADMINISTRATOR\DESKTOP\EXPLORE.EXE New File Name: C:\WINDOWS\SYSTEM32\kazaabackupfiles\\zoneallarm_pro_crack.exe +----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Kernel32.dll] - CopyFileExW Existing File Name: C:\DOCUMENTS AND SETTINGS\ADMINISTRATOR\DESKTOP\EXPLORE.EXE New File Name: C:\WINDOWS\SYSTEM32\kazaabackupfiles\\Battlefield1942_bloodpatch.exe +----------------------------------------------------+ [Proc 1652] - [Thread 1452] [Kernel32.dll] - CopyFileExW Existing File Name: C:\DOCUMENTS AND SETTINGS\ADMINISTRATOR\DESKTOP\EXPLORE.EXE New File Name: C:\WINDOWS\SYSTEM32\kazaabackupfiles\\AquaNox2 Crack.exe +----------------------------------------------------+
Table 1 - DLL: dynamic behavior

In the above table a little section of the report generated by DMADll is presented.

The process explore.exe (process ID 1652) creates a copy of itself, in the SYSTEM32 directory, called testing.exe and sets the attributes of this file to HIDDEN; to conceal its presence in the system. It also creates a startup key in \RunOnce and \Run registry subkey to allow the auto-execution of the malware after the system reboots. It is also quite interesting that the software creates a directory named \kazaabackupfiles and copies itself with different names about 50 times. This behavior could be linked to a P2P worm that spreads via peer-to-peer file sharing networks. This is a sample of report thanks to which it is possible to detect an anomaly behavior that could be flagged as malicious. Fortunately the system deploys also a driver that traces the malware behavior with the aim to get a new level of data and information that could be analyzed to reach further results. By using SSDT hooking technique the Kernel Module Driver traces the system calls to obtain information concerning: Registry changes Filesystem changes Process creation, termination Driver loading The purpose of developing also a Kernel Module Driver is to obtain more information regarding the state of the system and the processes during the execution of an application. This driver offers all the privileges of executing in Ring 0 and the ability to filter, to monitor and to log system calls. (Func) ZwSetValueKey: Pid: 1080 - svchost.exe KeyHandle - Exe-name: \REGISTRY\MACHINE\SYSTEM\ControlSet001\Services\SharedAccess\Parameters\FirewallPoli cy\StandardProfile\AuthorizedApplications\List Value: C:\Documents and Settings\Administrator\Desktop\Samples\Samples\Boh\yihahago.exe REG_SZ -> C:\Documents and Settings\Administrator\Desktop\Samples\Samples\Boh\yihahago.exe:*:Disabled:yihahago NTSTATUS: STATUS_SUCCESS

(Func) ZwSetValueKey: Pid: 720 - cam2cam.exe KeyHandle - Exe-name: \REGISTRY\USER\S-1-5-21-790525478-854245398-1177238915500\Software\Microsoft\Windows\CurrentVersion\RunOnce Value: Winsock2 driver REG_SZ -> wuaumqr.exe NTSTATUS: STATUS_SUCCESS

(Func) ZwSetValueKey: Pid: 720 - cam2cam.exe KeyHandle - Exe-name: \REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run Value: Winsock2 driver REG_SZ -> wuaumqr.exeNTSTATUS: STATUS_SUCCESS

(Func) ZwSetInformationFile: Pid: 1336 - cmd.exe Delete Operation: \Device\HarddiskVolume1\DOCUME~1\ADMINI~1\Desktop\malware1\ijkl.exe

Table 2 - Kernel Driver: Dynamic Behavior

In the kernel report just described , which brings together many different behaviors of different malware, what catches the eye is that every action logged is potentially malicious. The first action performed by the software svchost.exe (1080), is to modify the authorized applications list to use the network without being blocked by the firewall. The program whose name is cam2cam.exe, identified by the PID 720, updates the Registry Key \Run and \RunOnce in order to keep the malware alive also after the system reboots. The third action, recorded here, identifies a process named cmd.exe - seemingly a safe process which deletes a file; the malware that has just been executed, in order to hide the presence of a malicious software in the system.

Dynamic Malware Detection Module

Another interesting module implemented in the framework is DMD, that stands for Dynamic Malware Detection module, completely written in Python. DMD is able to create a Database containing the malware behavior, previously analyzed, and then it is able to compare run-time behavior with a set of dynamic signatures. This module also allows the analyst to implement new signatures that will be included in the cycle of malware detection. This is interesting because every time an analyst finds a new malicious behavior, he can include it in the dynamic signature database.

Figure 3 RMAS: malware detection cycle

Dynamic Analysis Conclusion

By using behavior analysis, it is possible to understand the nature and the purpose of the malicious software and this could reveal which files are read or accessed and which operations the malware has carried out. Using both the kernel driver and the DLL to monitor which actions are executed by the program, is useful for the analyst in order to gain a better view concerning the picture of the system.

Figure 4 Dynamic Analysis Modules

Case Study
RMAS is useful in order to analyze software behavior, especially malware behavior, and it also could be used for a rapid incident response report. The framework could be used to detect the dangerousness level of an executable that has not been detected by any antivirus product. RMAS could be used to aid a Malware Analyst in order to provide dynamic information to determine if a next step for malware analysis is required or not. In the following paragraphs are presented typical scenarios where RMAS has been deployed and tested.

Scenario 1
The Marketing Manager (MM) has received via e-mail as attachment a suspicious file from an unknown source. Before executing it, in order to avoid infection, MM has tried to run an antivirus scan against that file, but without any interesting result; it turned out to be a safe one. After that, he sent to the IT Staff the malware in order to be analyzed with RMAS. With static analysis, just a low percentage of antivirus product used by RMAS had detected the suspicious file (7/40 17.5%), but the dynamic analysis detected lots of malicious behavior and flagged it as a malware. This prevented the system infection.

Scenario 2
During network traffic analysis, an analyst has noticed a process that was downloading external software on the local machine. He obtained the malware and he sent it to RMAS. During the analysis the major part of the antivirus software detected the threat and classified it as FakeAV. RMAS reported lots of behavioral information, such as file, registry and network access. The results showed which DNS Query had been made, which IP address had been contacted and this helped to deploy new IDS signature.
Figure 5 Malware Customization

Scenario 3
A lot of compromised PCs have been attacked through customized malware. The customization helps the malware to be undetectable from the anti-malware software. During a forensics analysis, an analyst has detected a strange behavior on the system analyzed but

the antivirus had found no malware presence. For this reason he chose to analyze the system with RMAS and after a little period of a time the RMAS found the malicious files and which network requests were made. The report generated helped to identify the threat and to classify it as a bot.

Future Work
RMAS was developed to be a modular system, and this makes it very extensible, and when a new tool or module will be developed it could be plugged into the framework easily. In future my research aims to improve the detection engine, in order to permit analysis of sequences of actions rather than individual actions. In the roadmap there is also the idea to port the code also for Windows Vista and Windows 7 OS. Implementing a web interface could be interesting just to help analysts to share knowledge, samples, reports and so on. In order to have a general overview about future work on RMA System, a list of points that need to be developed is described below: 1. Porting on Windows Vista/Seven OSes 2. Detection Engine: analysis of a sequences of actions 3. Detection Engine: analysis of Kernel reports 4. Extracting Static Metadata extraction from the malicious software 5. Improving dynamic signature 6. Developing a Web interface

Conclusion
In this paper I presented the design and the implementation of a framework useful to carry out the hard work of malware analysis and malware detection. Thanks to its modular structure, RMAS is flexible, extensible and portable. Furthermore, it is also possible to detect unknown malwares on the basis at the low average similarity compared with the existing and already known ones. This possibility is due to the fully extensible detection engine that has been developed and to the new dynamic signature that could be added by the analyst, every time he detects a possible malicious or anomalous behavior. As described in the section Future Works, I plan to extend the system and to implement new features in order to improve malware analysis and detection.

Bibliography
Adam, R. (2009). API Hooking (LoadLibrary). Tratto da CodeProject: http://www.codeproject.com Adrian, S. (2007). Malware Analysis: Environment Design and Artitecture. SANS Institute Reading Room . Alex, A. (2008). API Hooking with MS Detours. Tratto da CodeProject: http://www.codeproject.com Bill, B. (2009). The Rootkit Arsenal. Escape and Evasion in the Dark Corners of the System. Wordware Publishing, Inc. Carrera, E. (s.d.). Win32 Static Analysis in Python. Carsten, W., Holz, T., & Felix, F. (2007). Toward Automated Dynamic Malware Analysis Using CWSandbox. IEEE Security&Privacy , p. 32-39. Christian, S., Ramon, S., Ian, W., Peter, K., & Barbara, E.-P. (2007). Capture A behavioral analysis tool for applications and documents. CyberSource. (2010). Online payment fraud trends, merchant practices and benchmarks. Tratto da CyberSource: http://www.cybersource.com David, H. (2010, Gennaio). 2010: Cybercrime Coming of Age. Tratto da ESET: http://www.eset.com Dhruv, M., & Shrikant, P. (s.d.). The Malware Market: Demystified. Tratto da http://www1.webng.com/dhruv/ Dmitri, A., Toralv, D., Paula, G., Rahul, K., David, M., Sam, M., et al. (s.d.). 2010 Threat Predictions. Tratto da McAfee Labs: http://www.mcafee.com Economics, C. (2007, Giugno). Annual Worldwide Economic Damages from Malware Exceed $13 Billion. Tratto da Computer Economics: http://www.computereconomics.com Elia, F., & Mircea, C. (2007). Peerbot: Catch me if you can. Symantec Security Response . Erdlyi, G. (2008). Introduction to Reverse Engineering. Tratto da Malware Analysis and Antivirus Technologies: http://www.tml.tkk.fi/Opinnot/T-110.6220/2008/ Erdlyi, G. (2008). Reverse Engineering II: Basics. Tratto da Malware Analysis and Antivirus Technologies: http://www.tml.tkk.fi/Opinnot/T-110.6220/2008/ Erdlyi, G. (2008). Reverse Engineering III: PE Format. Tratto da Malware Analysis and Antivirus Technologies: http://www.tml.tkk.fi/Opinnot/T-110.6220/2008/ Eric, K. (s.d.). APPLYING USER-MODE MEMORY SCANNING ON WINDOWS NT. USA. Eric, K. (2006). Battle with the Unseen Understanding Rootkits on Windows. USA. Federal Trade Commission. (2010, Febbraio). Fraud Report 2010. Tratto da Federal Trade Commission: http://www.ftc.gov

Franois, P. (s.d.). Frodi finanziarie e Internet Banking: minacce e contromisure. Tratto da http://www.mcafee.com G., B., T., R., D., M., & T., T. (s.d.). WYSINWYX: What You See Is Not What You eXecute. Galen, H., & Doug, B. (1999, Luglio). Microsoft Research: Detours. Greg, H., & James, B. (2005). Rootkits: Subverting the Windows Kernel. In H. Greg, & B. James, Rootkit Detection (p. 295-312). Grgoire, J., Herv, D., & Eric, F. (2008). Behavioral detection of malware: from a survey towards an established taxonomy. Computer Virology 2008 . Internet Crime Complaint Center. (2008). 2008 - Internet Crime Report. Tratto da Internet Crime Complaint Center: http://www.ic3.gov Jamie, B., & Kris, K. (2007). Blackout: What really happened. Black Hat USA. Jamie, V. R., Ken, C., Levi, L., & Vanderveen, K. (2008). FARM: An Automated Malware Analysis Environment. IEEE International Carnahan Conference on Security Technology (ICCST) 2008 , p. 321-325. Jason, R., & Eric, L. (s.d.). Deobfuscator: an Automated Approach to the Identification and Removal of Code Obfuscation. Jonell, B. (2010, Maggio). Web 2.0 Botnet Evolution. KOOBFACE Revisited. Josh, J. (2008). An Anti-Reverse Engineering Guide. Tratto da CodeProject: http://www.codeproject.com Justin, S. (2009). Gray Hat Python. San Francisco: No Starch Press. Kimmo, K., Stahlberg, M., Samuli, L., & Antti, T. (2008). HIDE 'N SEEK REVISITED - FULL STEALTH IS BACK. Virus Bulletin Conference. Konrad, R., Thorsten, H., Carsten, W., Patrick, D., & Pave, L. (2008). Learning and Classification of Malware Behavior. Kris, K. (2007). Practical Malware Analysis. Li, S., Tim, E., & Serdan, B. (2008). An automatic anti-anti-VMware technique applicable for multi-stage packed malware. Malicious and Unwanted Software (MALWARE), 2008 3rd International Conference. , p. 17-23. Maksym, S. (2007). The Online Shadow Economy: A Billion Dollar Market For Malware Authors. Tratto da MessageLabs: http://www.messagelabs.com Malware Intelligence. (2010, Febbraio). SpyEye Bot: Analysis of a new alternative scenario crimeware. Tratto da Malware Intelligence: http://www.malwareint.com Malware Intelligence. (2010, Febbraio). SpyEye Bot: Conversations with the creator of crimeware. Tratto da Malware Intelligence: http://www.malwareint.com Mark E., R., Solomon, D. A., & Ionescu, A. (2009). Windows Internals, Fifth Edition. In R. Mark E., D. A. Solomon, & A. Ionescu, Crash Dump Analysis (p. 1119-1152). Microsoft Press.

Mark E., R., Solomon, D. A., & Ionescu, A. (2009). Windows Internals, Fifth Edition. In R. Mark E., D. A. Solomon, & A. Ionescu, Concepts and Tools, System Architecture, System Mechanisms, Management Mechanisms, Processes, Threads, and Jobs (p. 1-450). Microsoft Press. Nick, H., Tom, R., Casey, S., & Suan, N. (s.d.). PIMP MY PE: PARSING MALICIOUS AND MALFORMED EXECUTABLES. Organisation for Economic Co-operation and Development. (2007). Malicious Software (Malware): A Security Threat to the Internet Economy. Tratto da OECD: http://www.oecd.org Petter, W. (2008, Giugno). Malware Analysis. A Systematic Approach. Philipp, T., Thorsten, H., Jan, G., & Felix, F. (2009). Visual Analysis of Malware Behavior Using Treemaps and Thread Graphs. Pontiello, M. (2003, Maggio 07). TrID - File Identifier. Tratto da Sito Web di Marco Pontiello: http://mark0.net/soft-trid.html Rahul, M. (s.d.). Hacking the Malware - A reverse-engineers analysis. Rich, C. (2009, Ottobre). Seven Things Malware Authors, Botnet Controllers and Security Companies Dont Want You to Know. Tratto da HBGary: http://www.hbgary.com Roberto, L., & James, H. (2007). Using Entropy Analysis to Find Encrypted and Packed Malware. IEEE Security&Privacy , p. 40-45. Russ, M. (2007, Dicembre). Mandiant Red Curtain: Malware identification for incident responders. ISSA Journal . Rutkowska, J. (2006, November). Introducing Stealth Malware Taxonomy. Tratto da Invisiblethings Labs: http://invisiblethings.org Shelton, T. (2006). Windows/Linux/Mac/BSD Rootkit Basics. Tratto da Black Security: http://blacksecurity.org Symantec Security Response. (2009, Maggio). The State of Phishing. Tratto da Symantec: http://www.symantec.com TrendMicro. (2009). 2009s Most Persistent Malware Threats. Tratto da TrendMicro: http://it.trendmicro.com Ulrich, B. (2005, Dicembre 12). TTAnalyze: A Tool for Analyzing Malware. Vienna, Austria. Ulrich, B., Andreas, M., Christopher, K., & Engin, K. (2006). Dynamic analysis of malicious code. Computer Virology . Ulrich, B., Imam, H., Davide, B., Engin, K., & Christopher, K. (2009, Aprile). A View on Current Malware Behaviors. Viviane, Z. (2010, Marzo). An Introduction To Writing TDI Filter Drivers.

William, A., William, F., & John, M. (2000, Dicembre). Windows of Vulnerability: A Case Study Analysis. Tratto da University of Maryland: http://www.cs.umd.edu