CY 411 Reverse Software Engineering

Basic Dynamic Malware Analysis

Dr. Qasem Abu Al-Haija

Department of Cybersecurity
Faculty of Computer & Information Technology
Jordan University of Science and Technology

1 Qasem Abu Al-Haija Reverse Engineering

 Why Perform Dynamic Analysis?
• Static analysis can reach a dead-end due to
✓ Obfuscation
✓ Packing
✓ Examiner has exhausted the available static analysis techniques

• Dynamic analysis is efficient and will show

you exactly what the malware does

Reverse Engineering Qasem Abu Al-Haija 2

 Dynamic analysis

• It can involve
✓ Monitoring malware as it runs or
✓ Examining the system after the malware has been executed.

• It observes the malware’s true functionality,

✓ e.g., locate the keylogger’s log file on the system.

Reverse Engineering Qasem Abu Al-Haija 3

Advantages of Dynamic Analysis
• Observe the malware’s true functionality
– Existence of an action string in a binary does not mean
the action will actually execute

• Identify malware functionality

– Example: For Keyloggers, you can:
• Locate the keylogger’s log file on the system.

• Discover the kinds of records it keeps.

• Decipher where it sends its information, and so on.

Reverse Engineering Qasem Abu Al-Haija 4

 Disadvantages of Dynamic Analysis
• Dynamic analysis can put your network and system at risk.
– Malware can leak to your host (if no proper protection at the host).

• Not all code paths may execute when malware is run.

– E.g., in case of command-line malware that requires arguments
– Each argument could execute different program functionality.
– Without knowing the options, you wouldn’t be able to examine all of
the program’s functionality dynamically.
– Your best bet will be to use advanced dynamic or static techniques to
figure out how to force the malware to execute all its functionality.

Reverse Engineering Qasem Abu Al-Haija 5

 Sandboxes:
The Quick-and-Dirty
Approach
Reverse Engineering Qasem Abu Al-Haija 6
Sandboxes: The Quick-and-Dirty Approach
• sandbox is a security mechanism for running
untrusted programs in a safe environment without
fear of harming “real” systems.

• Sandboxes comprise virtualized environments that

– Simulate network services to ensure the software or
malware being tested will function normally.

Reverse Engineering Qasem Abu Al-Haija 7

 Using a Malware Sandbox
• Many malware sandboxes— will analyze malware for free.
– Such as Norman SandBox, GFI Sandbox, Anubis, Joe Sandbox,
ThreatExpert, BitBlaze, and Comodo Instant Malware Analysis.

– These provide easy-to-understand output and are great for

initial triage if you are willing to submit your malware to the
sandbox websites.

• Even though the sandboxes are automated:

– you might choose not to submit malware that contains
company information to a public website.
Reverse Engineering Qasem Abu Al-Haija 8
 GFI Sandbox

Reverse Engineering Qasem Abu Al-Haija 9

 Free Online Automated Malware Analysis
• Hybrid Analysis. Note: good
• sandbox.pikker.ee. Note: good
• Akana (Android files)
• Binary Guard True Bare Metal
• Intezer Analyze (Community Edition)
• Comodo Valkyrie
• Detux Sandbox (Linux binaries)
• Joe Sandbox Cloud (Community Edition)
• Malwr (also see MalwareViz). Note: down
• SecondWrite (free version)
• ThreatExpert
• ThreatTrack
• ViCheck
Reverse Engineering Qasem Abu Al-Haija 11
 Sandbox Drawbacks (1)
• Sandbox simply runs the executable, without command-line options.
– If the malware executable requires command-line options, it will not execute
any code that runs only when an option is provided.

– If your subject malware is waiting for a command-and-control packet to be returned

before launching a backdoor, the backdoor will not be launched in the sandbox.

• The sandbox may not record all events, because neither you
nor the sandbox may wait long enough.
– For example, if the malware is set to sleep for a day before it performs
malicious activity, you may miss that event.

– Most sandboxes hook the Sleep function and set it to sleep only briefly, but there is more
than one way to sleep, and the sandboxes cannot account for all of these.
Reverse Engineering Qasem Abu Al-Haija 12
 Sandbox Drawbacks (2)
• Malware often detects when it is running in a VM.
– If a VM is detected, the malware might stop running or
behave differently.
– Not all sandboxes take this issue into account.

• Some malware requires the presence of certain

registry keys or files on the system that might not
be found in the sandbox.
– legitimate data, such as commands or encryption keys.

Reverse Engineering Qasem Abu Al-Haija 13

 Sandbox Drawbacks (3)
• If the malware is a DLL, certain exported functions
will not be invoked properly.
– Because a DLL will not run as easily as an executable.

• Sandbox environment OS may not be correct for

malware.
– For example, the malware might crash on Win XP but run
correctly in Win 7.

• A sandbox cannot tell you what the malware does.

– It may report basic functionality, but it cannot tell you, for example, if the
malware is a custom Security Accounts Manager (SAM) hash dump utility.
– Those are conclusions that you must draw on your own.
Reverse Engineering Qasem Abu Al-Haija 14
SAM

Lsass.exe
Local Security Authority
Subsystem Service

15
 Running Malware

Reverse Engineering Qasem Abu Al-Haija 16

 Running Malware
• EXE files can be run directly, but DLLs can’t

• Use Rundll32.exe (included in Windows)

➔ rundll32.exe DLLname, Export arguments

• The Export value is one of the exported functions you

found in Dependency Walker, PEview, or PE Explorer.

Reverse Engineering Qasem Abu Al-Haija 17

 Running Malware
• Example: rip.dll has these exports: Install and
Uninstall: rundll32.exe rip.dll, Install

• Some functions use ordinal values instead of

names, like: rundll32.exe xyzzy.dll, #5

• It's also possible to modify the PE header and

convert a DLL into an EXE

Reverse Engineering Qasem Abu Al-Haija 18

 Monitoring with
Process Monitor

Reverse Engineering Qasem Abu Al-Haija 22

 Monitoring with Process Monitor (procmon)
• Advanced monitoring tool for Windows
– Monitor certain registry, file system, network, process, and thread activity.

– Combines two legacy tools: FileMon and RegMon.

• All recorded events are kept, but you can filter the display to
make it easier to find items of interest

• Don't run it too long, or it will fill up all RAM and crash the machine
– Procmon monitors all system calls it can gather as soon as it is run.

– Procmon uses RAM to log events!

Reverse Engineering Qasem Abu Al-Haija 23
 Monitoring with Process Monitor (procmon)
Procmon captures much data but doesn’t capture everything.

– For example, It can miss:

• Device driver activity of a user-mode component

– talking to a rootkit via device I/O controls,

• Certain GUI calls, such as SetWindowsHookEx.

Reverse Engineering Qasem Abu Al-Haija 24

 Procmon Display
• Procmon displays configurable columns containing information
about individual events, including the event’s
– sequence number,
– timestamp,
– name of the process causing the event,
– event operation,
– path used by the event, and
– result of the event.

Reverse Engineering Qasem Abu Al-Haija 26

 Launching Calc.exe
• Many, many events recorded

Reverse Engineering Qasem Abu Al-Haija 27

 Process Monitor Toolbar
• Many, many events recorded

Reverse Engineering Qasem Abu Al-Haija 28

 Procmon Filtering
• You can set procmon to filter on one executable running
on the system.

• This feature is particularly useful for malware analysis

because you can set a filter on the piece of malware you
are running.

• You can filter on individual system calls such as RegSetValue,

CreateFile, WriteFile, or other suspicious or destructive calls.

• The most important filters for malware analysis are

Process Name, Operation, and Detail

Reverse Engineering Qasem Abu Al-Haija 29

Reverse Engineering Qasem Abu Al-Haija 30
 Automatic filters on its toolbar
• Registry By examining registry operations, you can tell
how a piece of malware installs itself in the registry.

• File system Exploring file system interaction can show all

files that the malware creates or configuration files it uses.

• Process activity Investigating process activity can tell

whether the malware spawned additional processes.

• Network Identifying network connections can show you

any ports on which the malware is listening.

Reverse Engineering Qasem Abu Al-Haija 32

 Procmon - notes
• Analysis of procmon’s recorded events takes
practice and patience since many events are simply
part of the standard way executables startup.

• The more you use procmon, the easier you will

find it to review the event listing quickly.

Reverse Engineering Qasem Abu Al-Haija 33

 Viewing Processes
with Process Explorer

Reverse Engineering Qasem Abu Al-Haija 34

 Viewing Processes with Process Explorer
• The Process Explorer, free from Microsoft, is an extremely
powerful task manager that should be running when you
are performing dynamic analysis.

• It can provide valuable insight into the processes currently

running on a system.

• You can use Process Explorer to list

– active processes,
– DLLs loaded by a process,
– various process properties, and
– overall system information.

• You can also use it to kill a process, log out users, and
launch and validate processes.
Reverse Engineering Qasem Abu Al-Haija 35
 Process Explorer Interface

Options ->
Configure colors …

Reverse Engineering Qasem Abu Al-Haija 36

 Process Explorer Coloring
• Monitors the processes running on a system
and shows them in a tree structure that
displays child and parent relationships.
– Services are highlighted in pink,

– Processes in blue,

– New processes in green.

– Terminated processes in red.

Reverse Engineering Qasem Abu Al-Haija 37
 Process Explorer (PExp) Coloring
• PExp view five main columns:
– Process (the process name).
– PID (the process identifier).
– CPU (CPU usage).
– Description.
– Company Name.

• When analyzing malware, watch the Process

Explorer window for changes or new processes,
and be sure to investigate them thoroughly.
Reverse Engineering Qasem Abu Al-Haija 38
 Using the Verify Option on the Image tab
• This verifies that the image on disk is Microsoft signed.
– Microsoft digitally signs most of its core executables.
– PExp verifies that a signature is valid, and you can be
sure that the file is executable from Microsoft.
– This is useful to verify that the Windows file on disk
has not been corrupted;
– Since Malware often replaces authentic Windows
files with its own in an attempt to hide.

Reverse Engineering Qasem Abu Al-Haija 39

 Using the Verify Option on the Image tab
• This verifies the image on disk rather than in memory

• It is useless if an attacker uses process replacement,

– Running a process on the system and overwriting its
memory space with a malicious executable.

– This provides malware with the same privileges as the

process it is replacing ( to appear as a legitimate process).

– But it leaves a fingerprint: The image in memory will

differ from the image on disk.
Reverse Engineering Qasem Abu Al-Haija 40
Comparing Strings within Process Explorer
• One way to recognize process
replacement is to use the
Strings tab in the Process
Properties window to
compare the strings
contained in the disk
executable (image) against
the strings in memory for that
same executable running in
memory.

• If the two string listings are

drastically different, process
replacement may have
occurred
Reverse Engineering Qasem Abu Al-Haija 41
 Using “Find DLL” within Process Explorer
• It also lets you search for a handle or DLL by
choosing Find | Find Handle or DLL

• The Find DLL option is particularly useful when

you find a malicious DLL on disk and want to
know if any running processes use that DLL.

• To determine whether a DLL is loaded into a

process after load time, you can compare the
DLL list in Process Explorer to the imports shown
in Dependency Walker.
Reverse Engineering Qasem Abu Al-Haija 42
 Analyzing Malicious Documents within Process Explorer
• You can also use Process Explorer to analyze
malicious documents, such as PDFs and Word
documents.

• A quick way to determine whether a document is

malicious is to open Process Explorer and open the
suspected malicious document.

• If the document launches any processes, you

should see them in Process Explorer and be able
to locate the malware on disk via the Image tab
of the Properties window.
Reverse Engineering Qasem Abu Al-Haija 43
 Comparing Registry
Snapshots with Regshot

Reverse Engineering Qasem Abu Al-Haija 44

Comparing Registry Snapshots with Regshot
• Regshot is an open-source registry
comparison tool that allows you to take
and compare two registry snapshots.

• To use Regshot for malware analysis,

simply take the first shot by clicking the
1st Shot button, and then run the
malware and wait for it to finish
making any system changes. Next, take
the second shot by clicking the 2nd Shot
button. Finally, click the Compare
button to compare the two snapshots.

• As with procmon, your analysis of

these results requires patient scanning
to find nuggets of interest

Reverse Engineering Qasem Abu Al-Haija 45

 Faking a Network

Reverse Engineering Qasem Abu Al-Haija 46

 Faking a Network
• Malware often beacons out and eventually communicates
with a command-and-control server

• You can create a fake network and quickly obtain network

indicators without actually connecting to the Internet.

• These indicators can include

– DNS names,
– IP addresses, and
– packet signatures.

• To fake a network successfully, you must prevent the

malware from realizing that it is executing in a virtualized
environment
Reverse Engineering Qasem Abu Al-Haija 47
 Using ApateDNS
• ApateDNS, a free tool from Mandiant
(www.mandiant.com/products/research/mandiant_apatedns/d
ownload), is the quickest way to see DNS requests made by
malware.
– Needs .Net Framework 4.0/3.5?

• ApateDNS spoofs DNS responses to a user-specified IP address by

listening on UDP port 53 on the local machine.

• It responds to DNS requests with the DNS response set to an IP

address you specify.

• ApateDNS can display the hexadecimal and ASCII results of all

requests it receives.

Reverse Engineering Qasem Abu Al-Haija 48

 ApateDNS Capturing Malware DNS Requests
• Set the IP address you want sent in DNS responses and select the interface.
• Press the Start Server button
• This will automatically start the DNS server and change the DNS settings to
localhost.

• Run your malware and watch as DNS requests appear in the ApateDNS window.

• You can catch additional domains used by a malware sample through the use of
the nonexistent domain (NXDOMAIN) option.
– Malware will often loop through the different domains it has stored if the first or second
domains are not found.
– Using this NXDOMAIN option can trick malware into giving you additional domains it has in
its configuration.

49
 Monitoring with Netcat
• Netcat, the “TCP/IP Swiss Army knife,” can be used over
both inbound and outbound connections for port
scanning, tunneling, proxying, port forwarding, and much
more.

• In listen mode, Netcat acts as a server, while in connect

mode it acts as a client.

• Netcat takes data from standard input for transmission

over the network.

• All the data it receives is output to the screen via standard

output.
Reverse Engineering Qasem Abu Al-Haija 50
 NetCat Capturing Malware Packets
• Using ApateDNS, we redirect the DNS request for
evil.malwar3.com to our local host.

• Assuming that the malware is going out over port 80 (a

common choice), we can use Netcat to listen for
connections before executing the malware.

• Malware frequently uses port 80 or 443 (HTTP or HTTPS

traffic, respectively), because these ports are typically not
blocked or monitored as outbound connections.

• The malware connects to our Netcat listener because we’re

using ApateDNS for redirection.

Reverse Engineering Qasem Abu Al-Haija 51

Packet Sniffing with Wireshark
• Wireshark is an open-source sniffer, a
packet capture tool that intercepts and logs
network traffic.

• Wireshark provides visualization, packet-

stream analysis, and in-depth analysis of
individual packets.

Reverse Engineering Qasem Abu Al-Haija 52

 Using INetSim
• INetSim is a free, Linux-based software suite for simulating common
Internet services.

• The easiest way to run INetSim if your base operating system is Microsoft
Windows is to install it on a Linux virtual machine and set it up on the
same virtual network as your malware analysis virtual machine.

• INetSim is the best free tool for providing fake services, allowing you to
analyze the network behavior of unknown malware samples by emulating
services such as HTTP, HTTPS, FTP, IRC, DNS, SMTP, and others.

• INetSim does its best to look like a real server

– INetSim can serve almost any file requested in the case of http, https

• INetSim can also record all inbound requests and connections

Reverse Engineering Qasem Abu Al-Haija 53

 Notes: Install inetsim over Ubuntu
Desktop 16.04
• Install VirtualBox guest additions
• Refer to INetSim installation using apt

• sudo touch /etc/apt/sources.list.d/inetsim.list

• sudo chmod 755 /etc/apt/sources.list.d/inetsim.list
• echo "deb http://www.inetsim.org/debian/ binary/" >
/etc/apt/sources.list.d/inetsim.list
• wget -O - http://www.inetsim.org/inetsim-archive-signing-key.asc | sudo apt-
key add -
• sudo find / -name inetsim

• Refer INetSim
• Log files are stored in the /var/log/inetsim/ directory:
– debug.log: debug information in case inetsim is run in debug mode
– main.log: information logs (services started, stopped, ...)
– service.log: when connections are made against the services, logs are added to this file

Reverse Engineering Qasem Abu Al-Haija 54

Example Malware Analysis Setup
• This virtual network contains two hosts: the malware analysis Windows virtual machine and
the Linux virtual machine running INetSim.
• The Linux virtual machine is listening on many ports.
• The Windows virtual machine is listening on port 53 for DNS requests through ApateDNS.
• The DNS server for the Windows virtual machine has been configured to localhost (127.0.0.1).
• ApateDNS is configured to redirect you to the Linux virtual machine (192.168.117.169).

Reverse Engineering Qasem Abu Al-Haija 55

Main Sources for these slides
• Michael Sikorski and Andrew Honig, "Practical Malware
Analysis: The Hands-On Guide to Dissecting Malicious Software";
ISBN-10: 1593272901.

• Xinwen Fu, “Introduction to Malware Analysis,” University of

Central Florida

• Sam Bowne, “Practical Malware Analysis,” City College San

Francisco

• Abhijit Mohanta and Anoop Saldanha, “Malware Analysis and

Detection Engineering: A Comprehensive Approach to Detect and
Analyze Modern Malware,” ISBN: 1484261925.
Reverse Engineering Qasem Abu Al-Haija 56
 Thank you

Reverse Engineering Qasem Abu Al-Haija 57