Data
Preprocessing
Contents
What & Why preprocess the Data
Data Cleaning
Data integration
Data Transformation
Data reduction
Data Preprocessing
It is a data mining technique that involves
transformation of raw data into an
understandable format.
Why to Preprocess Data
• Data in the real world is:
Incomplete: lacking values, certain attributes of
interest etc.
Noisy: containing errors or outliers
Inconsistent: lack of compatibility or similarity
between two or more facts.
Why to Preprocess Data
• No Quality Data, no Quality mining:
Quality decision must be based on quality data.
Measure of Data Quality
Accuracy
Completeness
Consistency
Timeliness
Value added
Interpretability
Accessibility etc
Data Preprocessing
Techniques
Data Cleaning
Data Integration
Data Transformation
Data Reduction
What is Data?
Attributes Class
attribute
Tid Refund Marital Taxable
Collection of data objects and Income Cheat
Status
their attributes 1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
Objects
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10
10 No Single 90K Yes
Types of Attributes
There are different types of attributes
Nominal
Examples: ID numbers, eye color, zip codes
Ordinal
Examples: rankings (e.g., taste of potato chips on a scale
from 1-10), grades, height in {tall, medium, short}
Interval
Examples: calendar dates, temperatures in Celsius or
Fahrenheit.
Ratio
Examples: temperature in Kelvin, length, time, counts
Properties of Attribute
Values
The type of an attribute depends on which of the
following properties it possesses:
Distinctness: =
Order: < >
Addition: + -
Multiplication: */
Nominal attribute: distinctness
Ordinal attribute: distinctness & order
Interval attribute: distinctness, order & addition
Ratio attribute: all 4 properties
Data Types and Forms
A1 A2 … An C
Attribute-value data:
Data types
numeric, categorical
(see the hierarchy for
its relationship)
static, dynamic
(temporal)
7
Record Data
Data that consists of a collection of records, each of which consists of
a fixed set of attributes
Tid Refund Marital Taxable
Income Cheat
Status
1 Yes Single 125K No
2 No Married 100K No
Record
3 No Single 70K No
Data Matrix
4 Yes Married 120K No Document Data
5 No Divorced 95K Yes Transaction Data
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10
10 No Single 90K Yes
Data Matrix
Data objects with the fixed set of numeric attributes
Consider them as points in a multi-dimensional space, where each
dimension represents a distinct attribute
Represent by an m by n matrix,
where there are m rows, one for each object, and n columns, one for
each attribute
Projection Projection Distance Load Thickness
of x Load of y load
10.23 5.27 15.22 2.7 1.2
12.65 6.25 16.22 2.2 1.1
Document Data
Each document becomes a `term' vector,
each term is a component (attribute) of the vector,
the value of each component is the number of times
the corresponding term occurs in the document.
timeout
season
coach
game
score
pla y
team
wi n
ball
lost
Document 1 3 0 5 0 2 6 0 2 0 2
Document 2 0 7 0 2 1 0 0 3 0 0
Document 3 0 1 0 0 1 2 2 0 3 0
Transaction Data
A special type of record data, where
each record (transaction) involves a set of items.
For example, a grocery store transactions.
TID Items
1 Bread, Coke, Milk
2 Beer, Bread
3 Beer, Coke, Diaper, Milk
4 Beer, Bread, Diaper, Milk
5 Coke, Diaper, Milk
Graph Data
World Wide Web
Molecular Structures
Generic graph and HTML Links
2
5 1
2
5
Benzene Molecule: C6H6
Ordered data Sequences of transactions
Spatial Data
Temporal Data
Sequential Data
Genetic Sequence Data
Genomic sequence data
GGTTCCGCCTTCAGCCCCGCGCC
Spatio-Temporal Data CGCAGGGCCCGCCCCGCGCCGTC
GAGAAGGGCCCGCCTGGCGGGCG
GGGGGAGGCGGGGCCGCCCGAGC
CCAACCGAGTCCGACCAGGTGCC
Average Monthly Temperature of land and ocean
CCCTCTGCTCGGCCTAGACCTGA
The data analysis pipeline
Mining is not the only step in the analysis process
Data Result
Preprocessing
Data Mining Post-processing
Preprocessing: real data is noisy, incomplete and inconsistent
Data cleaning is required to make sense of the data
Techniques: Sampling, Dimensionality Reduction, Feature selection
Post-Processing: Make the data actionable and useful to the user
Statistical analysis of importance
Visualization
Data Preprocessing
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Data Quality
Examples of data quality problems:
Noise and outliers Tid Refund Marital
Status
Taxable
Income Cheat
Missing values 1 Yes Single 125K No
Duplicate data 2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
A mistake or a millionaire? 5 No Divorced 10000K Yes
6 No NULL 60K No
Missing values 7 Yes Divorced 220K NULL
8 No Single 85K Yes
9 No Married 90K No
Inconsistent duplicate entries 9 No Single 90K No
10
Noise
Noise refers to modification of original values
Examples: distortion of a person’s voice when talking on a poor phone
Two Sine Two Sine Waves
Waves + Noise
Outliers
Outliers are data objects with characteristics that are considerably
different than most of the other data objects in the data set
Can help to
detect new phenomenon or
discover unusual behavior in data
detect problems
Sample applications of outlier detection
Fraud detection
Abnormal buying patterns can characterize credit card abuse
Medicine
Unusual symptoms or test results may indicate potential health problems of a
patient
Public health
The occurrence of a particular disease
Sports statistics
Outstanding (in a positive as well as a negative sense) players
may be identified as having abnormal parameter values
Detecting measurement errors
• Data derived from sensors (e.g. in a given scientific experiment) may contain
measurement errors
• Abnormal values could provide an indication of a measurement error
“One person‘s noise could be another person‘s signal.”
Missing Values
Reasons for missing values
Information is not collected
(e.g., people decline to give their age and weight)
Attributes may not be applicable to all cases
(e.g., annual income is not applicable to children)
Handling missing values
Eliminate Data Objects
Estimate Missing Values
Ignore the Missing Value During Analysis
Replace with all possible values (weighted by their
probabilities)
Duplicate Data
Data set may include data objects that are duplicates, or
almost duplicates of one another
Major issue when merging data from heterogeous sources
Examples:
Same person with multiple email addresses
Forms of data
preprocessing
• Fill in missing values
• Smooth noisy data
• Remove outliers
• Resolve inconsistencies
Normalization and aggregation
How to Handle Noisy Data?
Binning method:
first sort data and partition into (equi-depth) bins
then smooth by bin means, smooth by bin median, smooth by bin
boundaries, etc.
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human
Regression
smooth by fitting the data
into regression functions
Simple Discretization Methods:
Binning
Equal-width (distance) partitioning:
Divides the range into N intervals of equal size: uniform grid
if A and B are the lowest and highest values of the attribute,
the width of intervals will be: W = (B-A)/N.
The most straightforward but outliers may dominate presentation
Skewed data is not handled well.
Simple Discretization Methods:
Binning
Equal-depth (frequency) partitioning:
Divides the range into N intervals, each containing
approximately same number of samples
Good data scaling
Managing categorical attributes can be tricky.
Binning Example
Attribute values (for one attribute e.g., age):
0, 4, 12, 16, 16, 18, 24, 26, 28
Equi-width binning – for bin width of e.g.,
10: Bin 1: 0, 4 [-,10) bin
Bin 2: 12, 16, 16, 18 [10,20) bin
Bin 3: 24, 26, 28 [20,+) bin
– denote negative infinity, + positive infinity
Equi-frequency binning – for bin density of e.g.,
3: Bin 1: 0, 4, 12 [-, 14) bin
Bin 2: 16, 16, 18 [14, 21) bin
Bin 3: 24, 26, 28 [21,+] bin
Binning Methods for Data
Smoothing
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
* Partition into (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Data Integration
Data integration:
combines data from multiple sources into a coherent
store
Detecting and resolving data value conflicts
for the same real world entity, attribute values from
different sources are different
possible reasons:
different representations,
different scales, e.g., metric vs. British units
Handling Redundant Data
Redundant data occur often when integration of multiple
databases is done
The same attribute may have different names in different
databases
Careful integration of the data from multiple sources may help
reduce/avoid
redundancies
inconsistencies and
improve mining speed and quality
Data Transformation
Transform or consolidate data into forms appropriate for
mining
Smoothing: remove noise from data
Aggregation: summarization, data cube construction
Daily sales data aggregated to compute monthly or annual amount
Normalization: scaled to fall within a small, specified range
min-max normalization
z-score normalization
normalization by decimal scaling
Data Transformation
Aggregation: summarization, data cube construction
Daily sales data aggregated to compute monthly or annual amount
A data cube for sales
Data Transformation: Normalization
An attribute values are scaled to fall within a small, specified
range , such as 0.0 to 1.0
Min-Max normalization
performs a linear transformation on the original data.
v minA
v' (new _ maxA new _ minA) new _
maxA minA
minA Let min and max values for the attribute income are
Example:
$12,000 and $98,000, respectively.
Map income to the range [0.0;1.0].
Data Transformation: Normalization
z-score normalization(or zero-mean normalization)
An attribute A, values are normalized based on the mean and
standard deviation of A.
v m e a nA
v'
s ta n d _ d e vA
Example: Let mean and standard deviation of the values for the
attribute income are $54,000 and $16,000, respectively.
With z-score normalization, a value of $73,600 for income is
transformed to
Data Transformation: Normalization
Decimal scaling
normalizes by moving the decimal point of values of attribute A.
The number of decimal points moved depends on the
maximum absolute value of A.
v
v'
Where j is the smallest integer such that Max(| v'
10
j
|)<1
Example: Suppose that the recorded values of A range from -986 to 917.
The maximum absolute value of A is 986.
To normalize by decimal scaling, we therefore divide each value by
1,000 (i.e., j = 3)
-986 normalizes to -0.986 and 917 normalizes to 0.917.
Data Reduction
Warehouse may store terabytes of data: Complex data
analysis/mining may take a very long time to run on the
complete data set
Data reduction
Obtains a reduced representation of the data set that is much
smaller in volume
but produces the same (or almost the same) analytical results
Data Reduction Strategies
Dimensionality reduction
Data compression
use encoding schemes to reduce the data set size
Numerosity reduction
data is replaced or estimated by alternative smaller data
representations
Sampling
Histograms
Clustering
Discretization and concept hierarchy
generation
replace raw data values for attributes by ranges or higher
conceptual
levels
Histograms
A popular data reduction 4
technique 0
Divide data into buckets and
3
store average (sum) for each 3
50
bucket
Can be constructed optimally 25
in one dimension using
2
dynamic programming
0
1
5
1
0
100002000030000400005000060000700008000090000 100000
5
Cluster Analysis
Partition data into
clusters, and store
cluster representation
only
Sampling
Statisticians sample because obtaining the entire set of data
of interest is too expensive or time consuming.
Example: What is the average height of a person in Pakistan?
We cannot measure the height of everybody
Sampling is used in data mining because processing the entire
set of
data of interest is too expensive or time consuming.
Example: We have 1M documents. What fraction has at least 100 words in
common?
Computing number of common words for all pairs requires 1012 comparisons
Example: What fraction of tweets in a year contain the word “Lahore”?
300M tweets per day, if 100 characters on average, 86.5TB to store all tweets
Sampling …
The key principle for effective sampling is the following:
using a sample will work almost as well as using the entire data
sets, if the sample is representative
A sample is representative if it has approximately the same
property (of interest) as the original set of data
Otherwise we say that the sample introduces some bias
Types of Sampling
Simple Random Sampling
There is an equal probability of selecting any particular item
Sampling without replacement
As each item is selected, it is removed from the population
Sampling with replacement
Objects are not removed from the population as they are
selected for the sample.
In sampling with replacement, the same object can be picked
up more than once.
This makes analytical computation of probabilities easier
Sampling
Raw Data
Sample Size
8000 points 2000 Points 500
Points
Discretization
Discretization:
Divide the range of a continuous attribute into intervals
Reduce data size by discretization
Interval labels can be used to replace actual data values.
Discretization for numeric data
Binning
sensitive to the user-specified number of bins and
outliers
Histogram
Clustering analysis
Segmentation by natural Partitioning
Summar
y
Data preparation is a big issue for both warehousing and mining
Data preparation includes
Data cleaning and data integration
Data reduction and feature selection
Discretization
A lot a methods have been developed but still an active area of
research
