Analyzing Windows Programs
Malware Analysis
CSCI 4976 - Fall 2015
Branden Clark
Malware - 09/15/2015 Analyzing Windows Programs 1
Side note
• If you’re ever bored...check out http://
security.cs.rpi.edu/courses/malware-
spring2013/
• Lot’s of links, articles, forums, CrackMe’s,
and tutorials are posted there
Malware - 09/15/2015 Analyzing Windows Programs 2
Windows
• You’ve probably heard us throw around these
terms
– Handle
– Process
– Thread
– Registry
– DLL
– Service
–…
• And you’re probably wondering what they all
mean
Malware - 09/15/2015 Analyzing Windows Programs 3
Windows API
• The standard way of interacting with Windows
• Implemented through a broad set of DLLs
(Dynamic Link Library)
– kernel32.dll & ntdll.dll
∘ Low level, interact with kernel (think syscalls)
∘ kernel32.dll and ntdll.dll are always loaded
– wininet.dll / ws2_32.dll
∘ Networking
– advapi32.dll
∘ “Advanced” API
∘ Services, processes, permissions, crypto, etc
– Lots more
Malware - 09/15/2015 Analyzing Windows Programs 4
Processes
• A resource container
• Each process has its own
– virtual address space
– threads
– Windows bookkeeping information
∘ Usually stored in structs and linked lists
∘ Diagram by Jeremy Blackthorne
• Some API: CreateProcess,
CreateProcessAsUser, EnumProcesses
Malware - 09/15/2015 Analyzing Windows Programs 5
Threads
• Scheduled and executed by the OS
• Belong to a single process, and share its
address space
• Have their own thread context and stack
• Some API: CreateThread,
CreateRemoteThread
Malware - 09/15/2015 Analyzing Windows Programs 6
Thread Context
• Keeps track of the state of a thread
– necessary when there are multiple threads on a
system
– State is defined by register values
• MSDN: “See CONTEXT in WinNT.h”
• Some API: GetThreadContext,
SetThreadContext
Malware - 09/15/2015 Analyzing Windows Programs 7
DLLs
• All processes share loaded DLLs
– Saves space, only loaded into memory once
∘ Must map into your own virtual address space
(LoadLibrary)
– Changes only affect your processes address space
• Exports functions for use by processes
• DllMain is automatically called when loaded
Some API: LoadLibrary, GetProcAddress
Malware - 09/15/2015 Analyzing Windows Programs 8
Registry
• Big bookkeeping mess, “Organized” in a
hierarchical tree
• Used to store configuration information for
everything
– Key: like a folder
– Value entry: like a file
• Use regedit to navigate it
• Buzzwords: Autorun, Load Order Hijacking,
Trusted DLLs
• Some API: RegOpenKey, RegSetValue,
RegEnumKey
Malware - 09/15/2015 Analyzing Windows Programs 9
Service
• Similar to a process
• Gets “Installed”
– onto disk/registry
• Can interact with them via the service
manager (services.exe)
– Start, stop, suspend, schedule, autostart
• Some API: CreateService,
OpenSCServiceManager,
EnumDependentServices
Malware - 09/15/2015 Analyzing Windows Programs 10
Handle
• Abstract pointer to something
– A specific process, file, registry key, service, etc.
– Is sometimes an actual (memory) pointer
Some things that return Handles Some things that use open handles
OpenFile, CreateFile WriteFile, ReadFile
OpenService, OpenSCManager OpenService, ControlService
RegCreateKey RegCreateKey, RegReplaceKey
Malware - 09/15/2015 Analyzing Windows Programs 11
Lecture sample
• http://ark.rpis.ec/Malware/Lectures/
03_Analyzing_Windows_Programs/
Malware - 09/15/2015 Analyzing Windows Programs 12
COM (Component Object Model)
• Code sharing / interoperability giant by MS
• COM servers offer up implementation
– Identified by GUIDs, CLSIDs, and IIDs
• COM Clients use the interface to exec the
implementation
• Similar to the #include/DLL method but…
– Anyone can offer up new interfaces
– Client doesn’t need to know who is implementing it
or where
Malware - 09/15/2015 Analyzing Windows Programs 13
Lecture sample
• PMA Lab 07-02
Malware - 09/15/2015 Analyzing Windows Programs 14
Lab this Friday
• Jeremy Blackthorne from MIT Lincoln
Laboratory is coming to speak
• No lab if he’s still coming
Malware - 09/15/2015 Analyzing Windows Programs 15
Fairgame & CSAW
• Fairgame solutions this Friday!!
– fairgame.rpis.ec
• CSAW CTF this weekend!
Malware - 09/15/2015 Analyzing Windows Programs 16
References
1. Sikorski, Michael, and Andrew Honig. Practical Malware Analysis the
Hands-on Guide to Dissecting Malicious Software. San Francisco: No
Starch, 2012. Print.
Malware - 09/08/2015 Advanced Static Analysis 17
