Calculate-TCPflow-Entropy.
py User Guide
By Ken Hartman
web: www.KennethGHartman.com
email: kgh@kennethghartman.com
General Description
The Calculate-TCPflow-Entropy.py program will compute the Shannon Entropy of every file
generated by tcpflow and will create a histogram to visualize the entropy. The tcpflow program is
a well-known tool that outputs a file for the stream of data between each socket pair in a packet
capture file.
Set-up Instructions
The Calculate-TCPflow-Entropy.py is designed to run on Windows and requires
Python(x,y)-2.7.6.0.exe. It has all of the required modules built in and is available at
http://code.google.com/p/pythonxy/wiki/Downloads.
1. Download and install Python(x,y)
2. Once Python is installed, extract the ####### zip file to the desired location on your computer.
This file is available at #########.
3. Lastly download tcpflow-win-1.3.0.zip from https://github.com/downloads/simsong
/tcpflow/tcpflow-win-1.3.0.zip and extract the payload to the Entropy Analysis folder, which the
extraction in Step 2 created.
Program Components
To work properly, the Calculate-TCPflow-Entropy.py program calls a batch file named Call-
TCPFlow.bat to purge the contents of the output directory and then pass the appropriate
parameters to tcpflow.
The figure below shows the contents of the Entropy Analysis folder.
As mentioned above in Step 3, the Calculate-TCPflow-Entropy.py program expects the
tcpflow executables to be in the Entropy Analysis folder.
© Kenneth G. Hartman, 2014 Page 1
The default version of the Call-TCPFlow.bat assumes that the packet capture file is named
capture.pcap. Change this, if desired, by editing the batch file.
It also assumes that the operating system supports a 64-bit processor. If not, edit the batch file to use
tcpflow32.exe instead of tcpflow64.exe.
Program Operation
The easiest way to run the Calculate-TCPflow-Entropy.py program is to right-click on it and
select “run in interactive mode.”
The diagram below shows typical output messages displayed by the program after execution. You may
close this window once the “Processing is complete” message is displayed.
© Kenneth G. Hartman, 2014 Page 2
While executing the program performs the following steps:
1. Run the Call-TCPFlow.bat batch file to generate the stream output files
2. Read in the report.xml file to get the list of filenames created by tcpflow
3. Create the top part of the EntropyTable.html web page file.
4. For each file that was generated, perform the following:
a. Rename each file with a TXT file extension. This makes it viewable with a web browser.
b. Calculate the frequency of each byte value in the file and the file’s Shannon Entropy.
c. Append the file name, the file size, and the entropy to the EntropyTable.html web
page file. The file name is a hyperlink to the tcpflow output TXT file and the Entropy
value is a hyperlink to the corresponding Frequency Histogram (which is generated in
the next step.)
d. Generate the Frequency Histogram and save it as a PDF file.
5. Close up the EntropyTable.html web page file.
When the python program has finished its execution, Double-click the EntropyTable.html web
page file to view the results in the default web browser. It should look something like the following:
The web page shows files that have entropy below 7.0 with a highlight in yellow.
© Kenneth G. Hartman, 2014 Page 3
Clicking on a file name hyperlink displays the raw stream content as a text file in the browser.
Alternatively, clicking on the associated entropy value, will display the frequency histogram.
Further Reading
For more discussion on File Entropy, visit the following pages on my blog:
http://www.kennethghartman.com/calculate-file-entropy/
http://www.kennethghartman.com/shannon-entropy-of-file-formats/
© Kenneth G. Hartman, 2014 Page 4
